Senior Data Engineer
Location: Bangalore, Chennai, Hyderabad
Experience: 6 - 10 Years
About the Role
We are seeking an experienced Senior Data Engineer to join our data engineering team. In this role, you will design, develop, and optimize large-scale data pipelines and architectures using modern cloud technologies. You will work with cross-functional teams to build robust ETL/ELT solutions, manage data lakes and lakehouses, and ensure high-performance data processing systems that support critical business intelligence and analytics initiatives.
Key Responsibilities
- Design and implement scalable data pipelines using Azure Data Factory and Azure Databricks
- Develop and optimize ETL/ELT processes for large-scale distributed data processing
- Build and maintain data lake and lakehouse architectures on Azure cloud platform
- Write efficient SQL and PySpark code for data transformation and analysis
- Implement Lakeflow Declarative Pipelines for streamlined data workflows
- Perform performance tuning and optimization of data processing systems
- Manage version control using Git and implement CI/CD pipelines for data solutions
- Collaborate with data scientists, analysts, and stakeholders to understand data requirements
- Ensure data quality, security, and governance standards are met across all pipelines
- Mentor junior team members and contribute to technical best practices
Required Skills & Experience
Mandatory Requirements
- Azure Data Factory (ADF): Strong hands-on experience in designing and managing data pipelines
- Azure Databricks: Proficiency in building and optimizing distributed data processing solutions
- Apache Spark: In-depth knowledge of Spark architecture and distributed computing concepts
- Lakeflow Declarative Pipelines: Practical experience with declarative pipeline development
- SQL: Expert-level proficiency in writing complex queries and data manipulation
- PySpark: Strong programming skills in PySpark for data transformation
- Azure Cloud Platform: Hands-on experience with Azure data services and cloud infrastructure
- ETL/ELT Concepts: Strong understanding of data integration patterns and methodologies
- Data Warehousing Principles: Knowledge of dimensional modeling, schema design, and best practices
- Distributed Data Processing: Experience working with large-scale data systems and handling big data challenges
- Performance Tuning & Optimization: Ability to identify bottlenecks and optimize data pipeline performance
- Data Lake & Lakehouse Architectures: Understanding of modern data architecture patterns and implementation
- Git & CI/CD: Experience with version control systems and continuous integration/deployment practices
Preferred Skills & Experience
- Experience with Delta Lake and Unity Catalog for data governance
- Knowledge of streaming frameworks and real-time data processing
- Exposure to Azure Synapse Analytics or Microsoft Fabric
- Working experience in Agile/Scrum development environments
- Familiarity with data governance frameworks and security best practices
Qualifications
- Bachelor's degree in Computer Science, Engineering, Mathematics, or related field (or equivalent professional experience)
- Minimum 5+ years of experience as a Data Engineer or similar role
- Proven track record of delivering production-grade data solutions
- Strong problem-solving and analytical skills
- Excellent communication and collaboration abilities
- Ability to work independently and as part of a team
Pay: ₹2,300,000.00 - ₹3,500,000.00 per year
Application Question(s):
- Are you ready to join within a week after selection?
- What is your current & preferred location?
- Do you have minimum 5+ years of experience as a Data Engineer or similar role ?
- What is your current & expected salary?
- Are you skilled in PySpark for data transformation?
- Do you have practical experience with Lakeflow declarative pipeline development?
Work Location: In person
