Senior Cloud Engineer – Azure, Terraform & Databricks
Experience: 6+ Years
Azure Experience: Minimum 4 Years
Work Mode: Remote / Hybrid
Shift: 5:00 AM – 1:00 PM EST
Primary Skills: Azure, Terraform, Databricks, ADLS Gen2, Azure Data Factory, Python
Job Summary
We are looking for an experienced Senior Cloud Engineer to design, implement, and manage the Azure infrastructure supporting an enterprise data and AI platform.
The role will focus on Azure Databricks, ADLS Gen2, Azure Data Factory, Terraform, Azure networking, security, observability, identity management, CI/CD, and cost optimization. The ideal candidate should be capable of taking ownership of platform architecture and working closely with data engineering, data science, and Databricks administration teams.
Key Responsibilities
- Design and manage Azure cloud infrastructure using Terraform.
- Build and maintain reusable Terraform modules for Azure services.
- Manage Terraform remote state, module versioning, drift detection, and CI/CD-based infrastructure deployment.
- Provision and manage Azure Databricks workspaces, ADLS Gen2, Azure Data Factory, Key Vault, networking, and Azure Machine Learning.
- Design secure Azure network architecture using VNets, hub-and-spoke topology, private endpoints, NSGs, firewall rules, and VNet injection.
- Implement identity and access management using Entra ID, managed identities, RBAC, service principals, SCIM, and Key Vault.
- Design and manage enterprise ADLS Gen2 architecture, ACLs, lifecycle policies, encryption, and access controls.
- Administer Databricks account and workspace configuration.
- Manage Unity Catalog, permissions, storage credentials, lineage, audit, and Delta Sharing.
- Define cluster policies, autoscaling, instance pools, and SQL warehouse sizing.
- Build monitoring and observability using Azure Monitor and Log Analytics.
- Manage CI/CD pipelines across development, test, and production environments.
- Monitor and optimize Databricks and storage costs.
- Troubleshoot infrastructure, connectivity, permissions, clusters, and platform-related issues.
- Publish platform standards, reference architectures, and reusable configurations.
Required Skills
- 6+ years of overall experience
- Minimum 4 years of hands-on Microsoft Azure experience
- Expert-level Terraform
- Strong Azure Databricks administration
- Hands-on experience with Unity Catalog
- Azure Databricks workspace and account administration
- Cluster policies and compute governance
- Azure networking and security
- VNets, Private Endpoints, NSGs, hub-and-spoke architecture
- Entra ID / Azure AD
- Managed Identities and RBAC
- Azure Key Vault
- ADLS Gen2
- Azure Data Factory
- CI/CD pipelines
- Strong Python scripting
- PowerShell and/or Bash
- Working knowledge of Spark and SQL
- Strong troubleshooting and root-cause analysis skills
- Ability to work independently as a senior technical resource.
Preferred Skills
Experience with Databricks Asset Bundles, Databricks Terraform Provider, Unity Catalog migrations, Azure Machine Learning, MLflow, AKS/Kubernetes, Azure Policy, OPA, Sentinel, Checkov, Event Hubs, Kafka, Stream Analytics, FinOps, and multi-region Databricks environments will be an advantage.
Preferred Certifications
- Databricks Certified Data Engineer Professional
- Databricks Platform Administrator certification
- Microsoft Azure Solutions Architect Expert – AZ-305
- Microsoft Azure Administrator Associate – AZ-104
- Microsoft DevOps Engineer Expert – AZ-400
- HashiCorp Certified Terraform Associate
Candidate Requirements
We are looking for a senior engineer who can:
- Take ownership of Azure platform architecture
- Make infrastructure and security decisions
- Work independently with minimal supervision
- Support multiple engineering teams
- Troubleshoot complex cloud and Databricks issues
- Define reusable platform standards
- Communicate clearly with technical and business stakeholders
Indeed Screening Questions
- How many years of total cloud/infrastructure experience do you have?
- How many years of hands-on Microsoft Azure experience do you have?
- How many years of Terraform experience do you have?
- Have you authored production Terraform modules?
- Do you have hands-on Azure Databricks administration experience?
- Have you worked with Unity Catalog?
- Do you have hands-on experience with ADLS Gen2?
- Do you have experience with Azure Data Factory?
- Have you worked with VNets, Private Endpoints, NSGs, and hub-and-spoke networking?
- Do you have experience with Entra ID, Managed Identity, RBAC, and Key Vault?
- Do you have Python and PowerShell/Bash scripting experience?
- Are you comfortable working 5:00 AM – 1:00 PM EST?
Pay: Up to ₹60,000.00 per month
Work Location: In person
