Job Title: Senior Data Engineer
Experience: 7+ Years
Employment Type: Full-Time
About the Role
We are looking for an experienced Senior Data Engineer to lead cloud data modernization initiatives and build scalable, AI-ready data platforms. The ideal candidate should have strong expertise in Databricks, PySpark, Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow, with hands-on experience in designing, orchestrating, and optimizing enterprise-grade data pipelines.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks (PySpark), Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow.
- Build and manage end-to-end orchestration frameworks by integrating Airflow DAGs with ADF and Logic Apps.
- Implement advanced workflow orchestration patterns including event-driven, micro-batch, and hybrid scheduling.
- Ensure pipeline dependency management, execution reliability, and operational excellence.
- Develop high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture.
- Implement data observability, monitoring, logging, and alerting mechanisms across workflows.
- Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
- Integrate pipelines with Azure services such as Azure Data Lake Storage (ADLS Gen2), Azure Blob Storage, and Event Triggers.
- Implement CI/CD pipelines using GitHub Actions or similar DevOps tools.
- Ensure data quality, governance, security, and compliance across the data platform.
- Collaborate with Data Science and AI/ML teams to enable AI/ML and GenAI data readiness.
- Mentor junior engineers and establish best practices for orchestration, pipeline development, and data engineering.
Required Skills & Qualifications
- 7+ years of experience in Data Engineering.
- Strong expertise in:
- Databricks (PySpark, Delta Lake)
- Azure Data Factory (ADF)
- Azure Logic Apps
- Apache Airflow
- Strong programming skills in Python and Advanced SQL.
- Experience building enterprise-scale orchestrated data platforms with multiple tool integrations.
- Solid understanding of ETL/ELT design patterns and workflow orchestration frameworks.
- Experience with cloud-native data architectures, preferably on Microsoft Azure.
- Hands-on experience with monitoring, logging, alerting, and pipeline reliability.
- Experience working with Azure Data Lake Storage (ADLS Gen2), Azure Blob Storage, and event-driven architectures.
- Knowledge of CI/CD practices and tools such as GitHub Actions.
- Strong understanding of data governance, security, and compliance best practices.
Preferred Qualifications
- Experience with event-driven architectures and API integrations.
- Exposure to AI/ML data pipelines and MLflow.
- Knowledge of Data Lakehouse architecture and governance frameworks.
- Azure and Databricks certifications are an added advantage.
- Experience working in Agile/Scrum development environments.
Preferred Technology Stack
- Databricks (PySpark, Delta Lake)
- Azure Data Factory (ADF)
- Azure Logic Apps
- Apache Airflow
- Python
- SQL
- Azure Data Lake Storage (ADLS Gen2)
- Azure Blob Storage
- GitHub Actions
- MLflow
- Azure Cloud
Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)
