Lead Data Engineer
About the Role
We are looking for an experienced Lead Data Engineer to lead the design, development, and optimization of scalable data platforms and pipelines. The ideal candidate will have strong hands-on experience in data engineering, cloud technologies, data warehousing, ETL/ELT, and distributed data processing.
As a Lead Data Engineer, you will work closely with engineering, product, analytics, and business teams to build reliable data solutions that support business intelligence, analytics, and AI/ML initiatives.
Key Responsibilities
- Lead the design and development of scalable, high-performance data pipelines and data platforms.
- Design and implement robust ETL/ELT workflows for batch and real-time data processing.
- Develop and optimize data pipelines using Python, SQL, and distributed data processing frameworks.
- Design and maintain data warehouses, data lakes, and lakehouse architectures.
- Work with large and complex datasets while ensuring data quality, accuracy, and reliability.
- Optimize data processing jobs, queries, storage, and infrastructure for performance and cost efficiency.
- Implement data validation, monitoring, logging, and error-handling mechanisms.
- Collaborate with Data Scientists, ML Engineers, Analysts, Software Engineers, and Product teams to understand data requirements.
- Lead technical discussions, architecture decisions, code reviews, and engineering best practices.
- Mentor and guide junior and mid-level data engineers.
- Establish and maintain standards for data governance, security, quality, and documentation.
- Troubleshoot production data issues and ensure timely resolution.
- Evaluate new data technologies and recommend solutions aligned with business and technical requirements.
Technical Skills
Must Have
- 6+ years of experience in Data Engineering or a related field.
- Strong proficiency in Python and SQL.
- Strong experience building ETL/ELT pipelines and data processing workflows.
- Hands-on experience with Apache Spark / PySpark.
- Strong understanding of data warehousing and dimensional modeling.
- Experience with relational and NoSQL databases.
- Strong understanding of distributed systems and data engineering architecture.
- Experience with Git, CI/CD, testing, and deployment practices.
- Strong understanding of data quality, data governance, and security principles.
Cloud & Modern Data Stack
Experience with one or more cloud platforms:
- AWS / Azure / GCP
- Cloud storage and data lake technologies
- Cloud-based data warehouses
- Containerization using Docker/Kubernetes
- Workflow orchestration tools such as Apache Airflow
- Streaming technologies such as Kafka
Experience with modern data platforms such as Databricks, Snowflake, BigQuery, Redshift, or equivalent will be an advantage.
Good to Have
- Experience supporting AI/ML and Generative AI data pipelines.
- Experience with real-time data processing and streaming architectures.
- Knowledge of DataOps, MLOps, or DevOps practices.
- Experience with lakehouse architecture.
- Experience working with large-scale datasets and high-volume data systems.
- Relevant cloud or data engineering certifications.
Apply Now
If you are a passionate Data Engineer with strong technical expertise and leadership experience, we'd love to hear from you!
Pay: ₹1,500,000.00 - ₹2,500,000.00 per year
Benefits:
- Food provided
- Health insurance
- Provident Fund
Work Location: In person
