JOB SUMMARY
We are seeking an experienced Data Engineer with strong hands-on Databricks expertise to support the design, development, and optimization of enterprise data solutions. The successful candidate will have deep experience building and maintaining ELT pipelines using native Databricks orchestration, working with large-scale data platforms, and applying modern data architecture principles.
JOB RESPONSIBILITIES
-
Design, develop, and maintain ELT pipelines using Databricks and related technologies.
- Build and optimize large-scale data processing workflows using PySpark and SQL.
- Orchestrate multi-stage data pipelines using Databricks Workflows and Jobs, and develop Delta Live Tables (DLT) pipelines with built-in data quality enforcement.
- Develop and manage data models to support analytics, reporting, and business intelligence requirements.
- Implement and support data Lakehouse architectures and data warehousing solutions.
- Collaborate with data analysts, architects, and business stakeholders to understand and deliver on data requirements.
- Ensure data quality, integrity, and governance across all data platforms.
- Monitor and troubleshoot data workflows, pipeline performance, and platform issues.
- Contribute to continuous improvement of data engineering standards and practices.
REQUIRED TECHNICAL SKILLSET
-
Strong hands-on experience with Databricks (Delta Lake, Databricks Workflows, Unity Catalog)
- Strong hands-on experience with Databricks-native orchestration (Databricks Workflows, Jobs) as the primary mechanism for multi-stage pipeline orchestration and scheduling
- Experience with Delta Live Tables (DLT) for pipeline development and expectation-based data quality enforcement
- Proven experience building and supporting ELT pipelines at scale
- Strong proficiency in PySpark and SQL
- Experience with data modeling techniques and best practices
- Good understanding of data Lakehouse architecture and data warehousing concepts
- Experience with version control tools (e.g., Git)
- Familiarity with cloud platforms (Azure preferred; AWS or GCP considered)
PREFERRED EXPERIENCE
-
Experience using Azure Data Factory for data ingestion from source systems, with Databricks Workflows/Jobs as the primary orchestration layer for transformation logic
- Experience with Apache Airflow
- Exposure to data governance frameworks and tools (e.g., Unity Catalog, Microsoft Purview)
- Experience in enterprise consulting or regulated industry environments
- Experience working with globally distributed teams across different time zones
CERTIFICATIONS PREFERRED
-
Databricks Certified Associate Developer for Apache Spark
- Databricks Certified Data Engineer Associate or Professional
- Microsoft Certified: Azure Data Engineer Associate (DP-203) or equivalent
- Additional data or cloud certifications are a plus
