Director of Data Engineering
Description
We seek an experienced and dynamic Data Engineering Manager to join our innovative technology team.
The Data platform of Datarails serves as the multi-tenant infrastructure for the Datarails application, running tens of thousands custom Data pipelines each day, serving thousands of different organizations across the globe.
You will have the opportunity to design, build, and manage the next generation of the Datarails Data Platform, build a data engineering team from the ground up, and manage all data-related aspects of the Datarails application.
What You'll Do
- Take a major role in designing, architecting, and selecting technology for the next generation of the Data platform.
- You will recruit, establish, and manage a data engineering team responsible for all Datarails data (Customer and business data), including Data Integrity, Data integrations, Pipelines, Data transformation, Data aggregation, Reports, Insights, Events, Monitoring, etc.
- You will define working methodology, KPIs for measuring success, and quality control measures.
- Design data profiling and monitoring to identify anomalies and maintain data integrity.
- Work closely with our AI and data scientists team, responsible for developing algorithms, analytical rules, and LLM/ML data models.
Requirements
- BSc in Computer Science or equivalent.
- 7+ years of experience as a Data Engineer
- 3+ years experience as a Data Engineering Manager
- Experience in building Data infrastructure.
- Experience and familiarity with the following tools:
- Big-data query engines (Spark, Databricks, Presto)
- Data lakes and data Lakehouse architecture
- Familiarity with Azure data stack - advantage.
- Experience creating and optimizing big data processes, pipelines, and architectures on cloud-native environments (Spark/Databricks).
- Proficiency with SQL, Python, and one of Java/Scala/CPP.
- Understanding of how to build robust and reliable solutions
- Familiarity with cloud Databases (Snowflake-Advantage)
- Have a strong analytical mind with proven problem-solving abilities
Apply for this position
We want to make sure everyone has an equal
chance to participate and make a difference. Datarails is an equal opportunity employer and
prioritizes
building a diverse and inclusive workplace.
We provide equal employment opportunities to all employees and applicants of any type and do not
discriminate based on race, color, religion, national origin, gender, age, sexual orientation,
physical
or mental disability, genetic information or characteristic, gender identity and expression, veteran
status, or other non-job related characteristics or other prohibited grounds specified in applicable
federal, state, and local laws.
Datarails’ policy is to comply with all applicable laws related to nondiscrimination and equal
opportunity and will not tolerate discrimination or harassment based on any of these
characteristics.
This policy applies to all terms and conditions of employment, including recruiting, hiring,
placement,
promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.