Pune, Maharashtra
Job Summary
Data Platform Engineer with a strong focus on Databricks and Azure to design, build, and operate a scalable and secure data platform. You will work at the intersection of infrastructure, data engineering, and cloud automation to establish an enterprise platform for analytics and AI workloads.
Key Responsibilities
Automate the provisioning of Databricks and Azure platform resources using Terraform, adhering to Infrastructure-as-Code (IaC) best practices
Design and develop reusable, modular Terraform components to enable scalable and maintainable deployments within the Databricks platform
Implement and maintain CI/CD pipelines using Azure DevOps for automated infrastructure and platform deployments
Provision new infrastructure using customized Databricks Terraform modules to support requests for new catalogs, job policies, and clusters in a standardized, automated, and governed manner
Enable self-service infrastructure provisioning by integrating an existing ServiceNow interface with automated workflows that generate pull requests for new Databricks resources, ensuring standardized and governed deployments
Troubleshoot and resolve platform-related issues to ensure high availability and uninterrupted service, including proactive monitoring and management of Azure quota limits, cluster performance, resource utilization, and platform constraints
Drive FinOps integration by onboarding Databricks service cost data into the existing FinOps (Umbrella) system, including configuring required data formats, enabling cost monitoring and reporting, and ensuring transparency and optimized resource usage across Azure and Databricks
Stay up to date with industry best practices and emerging technologies, especially Terraform (Azure Provider, Databricks Provider)
Skill Requirements
Degree in Computer Science, Information Technology, or a related field.
Proven experience as a Platform Engineer/Cloud Engineer or similar role.
Strong understanding of cloud platforms (preferably Microsoft Azure) and their services.
Good knowledge of infrastructure as code (IaC) using Terraform. Familiarity with Databricks Provider is an advantage.
Experience with Azure Data Lake Gen2 setup using Terraform
Knowledge in data management (e,g. archiving, backup) and the setup of interfaces between systems along with authentication mechanisms
Excellent problem-solving skills and the ability to work in a fast-paced environment.
Strong communication and collaboration skills.
Excellent English skills. German skills are a bonus.
Certfications in Microsoft Azure or Databricks are welcome.
Other Requirements
Skill Level (SFIA):
3 = Apply: Performs varied tasks, sometimes complex and non-routine, using standard methods and procedures. Works under general direction, exercises discretion, and manages own work within deadlines. Proactively enhances skills and impacts in the workplace.
#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-
