We are looking for a Senior ETL Data Engineer with 8+ years of experience to design, build, and maintain robust data pipelines and scalable data architectures. In this role, you will work closely with cross-functional teams to transform complex raw data into structured, reliable datasets for analytics and business intelligence.
Key Responsibilities
- Design, build, and optimize end-to-end data pipelines and ETL workflows using Azure Data Factory (ADF).
- Develop efficient, scalable data processing scripts and data transformations using PySpark.
- Build and manage enterprise data warehouse solutions in Azure Synapse Analytics.
- Implement high-performance data storage and querying patterns using Delta Lake architecture.
- Write complex, optimized SQL queries, stored procedures, and transformations for data modeling and analysis.
- Monitor, troubleshoot, and optimize data pipeline performance, ensure data quality, and resolve data bottlenecks.
Mandatory Qualifications & Requirements
- PySpark: Core proficiency in writing PySpark code for large-scale distributed data processing.
- Azure Data Factory (ADF): Hands-on experience creating, orchestrating, and monitoring complex ETL/ELT pipelines.
- Azure Synapse Analytics: Proven experience in designing data warehouses and running distributed analytical queries.
- SQL: Advanced proficiency in SQL for data manipulation, schema design, and query optimization.
- Delta Lake: Practical experience implementing ACID transactions, schema enforcement, and time travel features on lakehouse storage.
Preferred Skills
- Strong understanding of Data Warehousing principles (Dimensional Modeling, Star/Snowflake Schemas).
- Experience with Git, CI/CD pipelines, and DevOps principles for data workflows.
- Familiarity with data governance, security, and access control in Azure environments.
Pay: ₹700,000.00 - ₹1,500,000.00 per year
Experience:
- Pyspark: 6 years (Required)
- Synapse: 6 years (Required)
- ADF: 6 years (Required)
- Delta Lake: 6 years (Required)
Work Location: Hybrid remote in Chennai, Tamil Nadu
