Job Title: Senior Data Engineer
Experience: 7+ Years
Employment Type: Full-Time
About the Role
We are looking for an experienced Senior Data Engineer to lead cloud data modernization initiatives and build scalable, AI-ready data platforms. The ideal candidate should have strong expertise in Databricks, PySpark, Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow, with hands-on experience in designing, orchestrating, and optimizing enterprise-grade data pipelines.
Key Responsibilities
- * Design, develop, and maintain scalable data pipelines using Databricks (PySpark), Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow.
- * Build and manage end-to-end orchestration frameworks by integrating Airflow DAGs with ADF and Logic Apps.
- * Implement advanced workflow orchestration patterns including event-driven, micro-batch, and hybrid scheduling.
- * Ensure pipeline dependency management, execution reliability, and operational excellence.
- * Develop high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture.
- * Implement data observability, monitoring, logging, and alerting mechanisms across workflows.
- * Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
- * Integrate pipelines with Azure services such as Azure Data Lake Storage (ADLS Gen2), Azure Blob Storage, and Event Triggers.
- * Implement CI/CD pipelines using GitHub Actions or similar DevOps tools.
- * Ensure data quality, governance, security, and compliance across the data platform.
- * Collaborate with Data Science and AI/ML teams to enable AI/ML and GenAI data readiness.
- * Mentor junior engineers and establish best practices for orchestration, pipeline development, and data engineering.
Required Skills & Qualifications
- * 7+ years of experience in Data Engineering.
- * Strong expertise in:
- * Databricks (PySpark, Delta Lake)
- * Azure Data Factory (ADF)
- * Azure Logic Apps
- * Apache Airflow
- * Strong programming skills in Python and Advanced SQL.
- * Experience building enterprise-scale orchestrated data platforms with multiple tool integrations.
- * Solid understanding of ETL/ELT design patterns and workflow orchestration frameworks.
- * Experience with cloud-native data architectures, preferably on Microsoft Azure.
- * Hands-on experience with monitoring, logging, alerting, and pipeline reliability.
- * Experience working with Azure Data Lake Storage (ADLS Gen2), Azure Blob Storage, and event-driven architectures.
- * Knowledge of CI/CD practices and tools such as GitHub Actions.
- * Strong understanding of data governance, security, and compliance best practices.
- Preferred Qualifications
- * Experience with event-driven architectures and API integrations.
- * Exposure to AI/ML data pipelines and MLflow.
- * Knowledge of Data Lakehouse architecture and governance frameworks.
- * Azure and Databricks certifications are an added advantage.
- * Experience working in Agile/Scrum development environments.
Preferred Technology Stack
- * Databricks (PySpark, Delta Lake)
- * Azure Data Factory (ADF)
- * Azure Logic Apps
- * Apache Airflow
- * Python
- * SQL
- * Azure Data Lake Storage (ADLS Gen2)
- * Azure Blob Storage
- * GitHub Actions
- * MLflow
- * Azure Cloud
Could you Please go through this jd and let me know
Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)
