Senior Data Engineer – Databricks, Python & Azure
Location: Chennai
Client: US-based IT services and consulting firm
Work Mode: Work from Office (5days/week)
Experience: 10+ Years
Employment Type: Full-time
About the Role
We are looking for an experienced Senior Data Engineer with deep expertise in Databricks, Python, Apache Spark, SQL, and Azure to design and deliver enterprise-scale Lakehouse solutions. This role requires hands-on experience in building scalable data platforms, modern ETL/ELT pipelines, and cloud-native data engineering solutions.
The ideal candidate will have strong architectural experience, excellent stakeholder management skills, and a passion for building high-performance data platforms that support analytics and AI initiatives.
Key Responsibilities
- Design and implement enterprise-grade Lakehouse solutions using Databricks.
- Build scalable batch and real-time data pipelines using Apache Spark (PySpark) and SQL.
- Develop ETL/ELT pipelines, metadata-driven ingestion frameworks, and incremental data processing.
- Implement and optimize Databricks features including Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, Unity Catalog, and Workflows.
- Design and maintain enterprise data warehouse solutions using 3NF and dimensional data modeling.
- Establish data governance, security, data quality, and metadata management frameworks.
- Lead ML lifecycle management using MLflow and support AI initiatives such as RAG and AI/BI.
- Deploy and manage cloud-native data solutions on Microsoft Azure.
- Build CI/CD pipelines and automate deployments using Azure DevOps.
- Optimize Spark workloads for scalability, reliability, and performance.
- Integrate enterprise applications such as ServiceNow with data platforms.
- Collaborate with business stakeholders to translate requirements into scalable technical solutions.
- Mentor engineering teams and provide technical leadership throughout project delivery.
Required Skills
- 10+ years of software engineering experience with at least 5+ years in Data Engineering.
- Strong hands-on experience with Databricks and end-to-end Lakehouse implementation.
- Expertise in Python, SQL, and Apache Spark (PySpark).
- Strong experience with Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, Unity Catalog, and Databricks Workflows.
- Experience designing enterprise-scale data warehouses using 3NF and dimensional modeling.
- Experience working on both batch and real-time data processing solutions.
- Architecture-level experience owning solution design through deployment.
- Good understanding of AI services, machine learning concepts, or Generative AI.
- Strong stakeholder management and requirement gathering experience with US or UK clients.
- Experience working in an IT Services or IT Consulting environment.
Preferred Skills
- Azure Databricks and Azure Data Services
- Azure DevOps and CI/CD implementation
- MLflow and MLOps
- AI use cases including RAG and AI/BI
- Terraform or Infrastructure as Code (IaC)
- Databricks Asset Bundles (DABs)
- ServiceNow or enterprise application integrations
- Databricks, Azure, or AWS certifications
What We're Looking For
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management abilities.
- Passion for modern data engineering and cloud technologies.
- Experience mentoring teams and leading technical initiatives.
- Ability to deliver scalable, secure, and high-performance data solutions.
Benefits
- Free AWS and Azure certification training.
- Group Medical Insurance coverage of ₹5 Lakhs.
- Meal allowance.
- Opportunity to work on enterprise-scale cloud and AI-driven data platforms.
- Excellent learning and career growth opportunities.
Note: Compensation package includes a 5% variable pay component.
Pay: Up to ₹3,000,000.00 per year
Benefits:
Application Question(s):
- Are you comfortable appearing for a face-to-face interview in the office? Final round will be face-face (mandatory)
- Are you available to join immediately or within 15 days in office?
- If you are serving NP, when is your last working day?
Experience:
- total: 10 years (Required)
Location:
- Chennai, Tamil Nadu (Chennai District) (Required)
Work Location: In person
