Array

Director of Data Engineering

Description

We seek an experienced and dynamic Data Engineering Manager to join our innovative technology team. 

The Data platform of Datarails serves as the multi-tenant infrastructure for the Datarails application, running tens of thousands custom Data pipelines each day, serving thousands of different organizations across the globe. 

You will have the opportunity to design, build, and manage the next generation of the Datarails Data Platform, build a data engineering team from the ground up, and manage all data-related aspects of the Datarails application.

What You'll Do

  • Take a major role in designing, architecting, and selecting technology for the next generation of the Data platform. 
  • You will recruit, establish, and manage a data engineering team responsible for all Datarails data (Customer and business data), including Data Integrity, Data integrations, Pipelines, Data transformation, Data aggregation, Reports, Insights, Events, Monitoring, etc. 
  • You will define working methodology, KPIs for measuring success, and quality control measures.
  • Design data profiling and monitoring to identify anomalies and maintain data integrity.
  • Work closely with our AI and data scientists team, responsible for developing algorithms, analytical rules, and LLM/ML data models.


Requirements

  • BSc in Computer Science or equivalent.
  • 7+ years of experience as a Data Engineer 
  • 3+ years experience as a Data Engineering Manager
  • Experience in building Data infrastructure.
  • Experience and familiarity with the following tools:
  • Big-data query engines (Spark, Databricks, Presto)
  • Data lakes and data Lakehouse architecture
  • Familiarity with Azure data stack - advantage.
  • Experience creating and optimizing big data processes, pipelines, and architectures on cloud-native environments (Spark/Databricks).
  • Proficiency with SQL, Python, and one of Java/Scala/CPP.
  • Understanding of how to build robust and reliable solutions
  • Familiarity with cloud Databases (Snowflake-Advantage)
  • Have a strong analytical mind with proven problem-solving abilities


Apply for this position

Apply for this position

Privacy Notice

We want to make sure everyone has an equal chance to participate and make a difference. Datarails is an equal opportunity employer and prioritizes building a diverse and inclusive workplace.

We provide equal employment opportunities to all employees and applicants of any type and do not discriminate based on race, color, religion, national origin, gender, age, sexual orientation, physical or mental disability, genetic information or characteristic, gender identity and expression, veteran status, or other non-job related characteristics or other prohibited grounds specified in applicable federal, state, and local laws.

Datarails’ policy is to comply with all applicable laws related to nondiscrimination and equal opportunity and will not tolerate discrimination or harassment based on any of these characteristics. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.