Senior Databricks Data Engineer
Experience: 7+ Years
Location: Remote (Preferred)
Employment Type: Contract (6–12 Months)
Working Hours: 10:00 AM – 7:00 PM IST, with flexibility to overlap with EST/PST hours when required.
Job Summary
We are looking for an experienced Senior Databricks Data Engineer to design, build, and optimize scalable data engineering solutions on the Databricks platform. The ideal candidate should have strong expertise in Apache Spark, cloud platforms, and modern data engineering practices. This role requires hands-on development experience, technical leadership, and the ability to deliver enterprise-grade data solutions across multiple projects.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Lead the implementation of enterprise data engineering solutions from design to production.
- Deliver multiple Databricks projects with hands-on development experience.
- Collaborate with architects, business stakeholders, and cross-functional teams to define data requirements and solution architecture.
- Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.
- Build and manage ETL/ELT workflows using cloud-native data services.
- Implement CI/CD pipelines for Databricks deployments.
- Monitor, troubleshoot, and optimize production data workloads.
- Apply data governance, security, and best practices within cloud environments.
- Mentor junior engineers through code reviews, technical guidance, and best practices.
- Stay updated with the latest Databricks features, Spark enhancements, and MLOps technologies.
Required Skills
- 7+ years of experience in Data Engineering.
- 10+ years of overall IT/Consulting experience is preferred.
- Hands-on experience with Databricks in at least 6–8 production implementations.
- Databricks Data Engineer Professional Certification.
- Strong expertise in Apache Spark and distributed computing.
- Proficiency in Python and SQL.
- Strong understanding of Delta Lake, Spark SQL, and Databricks Workflows.
- Experience with Azure Data Engineering services.
- Experience working with at least two cloud platforms (Azure, AWS, or GCP).
- Deep expertise in one cloud ecosystem.
- Experience implementing CI/CD using Azure DevOps, GitHub Actions, or Jenkins.
- Strong knowledge of data modeling, ETL/ELT processes, and data warehousing concepts.
- Excellent problem-solving, communication, and stakeholder management skills.
Nice to Have
- Experience with MLflow and MLOps.
- Knowledge of Spark runtime internals.
- Experience with Unity Catalog.
- Performance tuning and Spark optimization expertise.
- Experience with Full Stack Development.
- Familiarity with Infrastructure as Code (Terraform).
- Exposure to real-time streaming using Kafka or Structured Streaming.
Preferred Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- Databricks Data Engineer Professional Certification (Mandatory).
- Azure, AWS, or GCP Cloud Certifications are a plus.
Apply Now
If you have strong expertise in Databricks, Apache Spark, Azure Data Engineering, and Cloud Data Platforms, we'd love to hear from you.
Interested candidates can share their updated resume to:
[email protected]
Pay: ₹80,000.00 - ₹100,000.00 per month
Work Location: Remote
