Design, develop, and maintain scalable data engineering solutions on the Azure platform using Azure Databricks, Azure Data Factory, ADLS Gen2, and related Azure data services. Build enterprise-grade data pipelines, transformation frameworks, and Lakehouse architectures. Key Responsibilities • Design and implement end-to-end data pipelines using Azure Databricks and Azure Data Factory. • Develop scalable ETL/ELT solutions using PySpark and Spark SQL. • Build and maintain Lakehouse architectures using Bronze, Silver and Gold layers. • Develop Delta Lake-based data models and optimize data processing performance. • Integrate data from ERP, CRM, databases, APIs, IoT, and external sources. • Implement data quality checks, monitoring, and error-handling frameworks. • Create and maintain CI/CD pipelines using Azure DevOps. • Configure and manage Unity Catalog, data governance, and access controls. • Work with Power BI and analytics teams to deliver curated data marts and semantic models. • Collaborate with business stakeholders, architects, and data scientists to support reporting and AI initiatives. • Troubleshoot production issues and optimize pipeline performance. • Prepare technical documentation, deployment guides, and operational runbooks. Required Technical Skills • Azure Databricks • PySpark and Spark SQL • Azure Data Factory (ADF) • Azure Data Lake Storage Gen2 (ADLS) • Delta Lake • SQL Server and SQL scripting • Azure DevOps and CI/CD • Git/Version Control • Data Warehousing and ETL Concepts • Performance Tuning and Optimization Qualifications • B.E./B.Tech/MCA or equivalent.
