Senior Data Engineer - Job Description
remote job
Exp:-7 yrs to 10 yrs
Salary:-15lpa to 24lpa
Contact via WhatsApp:-9899061997
Job Overview
We are seeking a Senior Data Engineer to drive cloud data modernization and build scalable, AI-ready data platforms. This role emphasizes expertise in *Databricks, PySpark, Azure Data Factory, Logic Apps, and Airflow*, with a strong focus on orchestration, pipeline reliability, and end-to-end data workflow management.
Key Responsibilities
● Design, build, and maintain scalable data pipelines using *Databricks (PySpark), ADF, Logic Apps, and Airflow.● Develop and manage end-to-end orchestration frameworks integrating *Airflow DAGs with ADF and Logic Apps.● Implement advanced workflow orchestration patterns, including *event-driven, micro-batch, and hybrid scheduling.● Ensure pipeline dependency management, execution reliability, and operational excellence.● Build high-performance *ETL/ELT pipelines* using Databricks and *Delta Lake architecture.● Implement data observability, monitoring, and alerting mechanisms across workflows.● Optimize pipelines and workflows for performance, scalability, and cost efficiency.● Integrate pipelines with Azure services such as *ADLS Gen2, Blob Storage, and event triggers.● Implement *CI/CD* for pipelines and workflows using GitHub Actions or equivalent.● Ensure data quality, governance, and security compliance.● Collaborate with data science teams for *ML/GenAI data readiness.● Mentor junior engineers and drive best practices for orchestration and pipeline design.
Required Qualifications
● 7+ years of experience* in data engineering.● Highly proficient in *Databricks (PySpark, Delta Lake), Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow.● Strong programming skills in *Python* and advanced *SQL.● Experience building orchestrated data platforms with *multi-tool integration.● Strong understanding of *ETL/ELT patterns and orchestration frameworks.● Experience in *cloud-native data architectures*, preferably Azure.● Hands-on experience with *monitoring, logging, and pipeline reliability.
Preferred Qualifications
● Experience with *event-driven architecture and API integrations*.● Exposure to *AI/ML data pipelines and MLflow*.● Knowledge of *Data Lakehouse architectures and governance frameworks*.● Azure / Databricks certifications.
Must-Have Tech Stack
Databricks + PySpark | Azure Data Factory (ADF) | Logic Apps | Airflow
Core requirement: Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow, along with building high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture.
Regards
Neha Gupta
9899061997
Pay: ₹1,500,000.00 - ₹2,400,000.00 per year
Work Location: Remote
