Senior Data Engineer – Databricks / Azure
Experience: 5+ Years
Employment Type: Full-Time
Work Mode: Remote
We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks, PySpark, Delta Lake, and Azure Data Engineering to design and develop scalable data pipelines and cloud-based data solutions.
Primary Skills
- Databricks
- PySpark
- Delta Lake
- Azure Data Factory (ADF)
- Logic Apps
- Apache Airflow
Secondary Skills
- Python
- SQL
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- CI/CD
- MLflow
- Data Observability
- Azure Cloud
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
- Build and optimize data processing solutions using Delta Lake.
- Develop and manage ETL/ELT workflows using Azure Data Factory and Airflow.
- Integrate Azure services including ADLS Gen2 and Logic Apps into data solutions.
- Write efficient Python and SQL code for data transformation and processing.
- Implement data quality, monitoring, and data observability practices.
- Optimize Spark jobs, Delta tables, and data pipelines for performance and scalability.
- Implement CI/CD pipelines for data engineering workflows and deployments.
- Collaborate with data scientists and engineering teams on MLflow and machine learning workflows.
- Troubleshoot production data pipelines and ensure reliability and availability.
Required Qualifications
- 5+ years of experience in Data Engineering.
- Strong hands-on experience with Databricks, PySpark, and Delta Lake.
- Good experience with Azure Data Factory, ADLS Gen2, and Azure Cloud.
- Strong programming skills in Python and SQL.
- Experience with workflow orchestration using Airflow.
- Understanding of CI/CD, data quality, monitoring, and cloud-based data platforms.
Preferred
Experience with MLflow, Logic Apps, Data Observability, and enterprise-scale Azure data platforms will be an added advantage.
Work Location: Remote
