Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Azure Databricks and the Azure data ecosystem.
- Build and optimize ETL/ELT workflows to support data integration, transformation, and analytics requirements.
- Develop high-performance data processing solutions using PySpark, Python, and SQL.
- Integrate data from multiple structured and unstructured data sources into enterprise data platforms.
- Optimize Spark jobs, data pipelines, and SQL queries to improve performance and scalability.
- Collaborate with data architects, analysts, and business stakeholders to understand data requirements and deliver robust solutions.
- Ensure data quality, integrity, security, and governance across all data engineering processes.
- Participate in code reviews, testing, deployment, and production support activities.
- Troubleshoot and resolve data pipeline issues while ensuring high system availability and reliability.
- Follow industry best practices for coding standards, documentation, and CI/CD implementation.
Mandatory Skills
- Strong experience in Azure Databricks for enterprise data engineering solutions.
- Hands-on expertise in PySpark and Python for distributed data processing.
- Strong proficiency in SQL, query optimization, and relational databases.
- Experience in developing and maintaining ETL/ELT pipelines.
- Hands-on experience with Azure Data Factory (ADF) for workflow orchestration.
- Strong knowledge of Azure Data Lake Storage (ADLS) and Azure cloud services.
- Good understanding of Spark architecture, Delta Lake, and data optimization techniques.
- Experience with Git and CI/CD pipelines.
- Excellent analytical, problem-solving, and communication skills
Pay: From ₹2,000,000.00 per year
Benefits:
Work Location: In person
