We are looking for an experienced Databricks Platform & Infrastructure Engineer with strong expertise in Azure, Databricks, Unity Catalog, Terraform, networking, identity, security, and CI/CD.
The role is implementation-focused and will involve deploying, configuring, and supporting the Azure and Databricks platform capabilities required to establish a secure, scalable, and governed data platform using Unity Catalog.
The candidate will work from architect-defined designs and collaborate closely with Infrastructure, Security, Data Engineering, and Architecture teams.
Key ResponsibilitiesAzure Platform & Infrastructure
- Configure and manage Azure subscriptions, resource groups, and environment structures.
- Deploy and manage ADLS Gen2 storage accounts and supporting Azure services.
- Implement Infrastructure-as-Code using Terraform.
- Configure Azure DevOps projects, repositories, and deployment pipelines.
- Implement Development, Test, and Production environment separation.
- Configure platform monitoring, diagnostic logging, alerting, and operational dashboards.
Network Security & Identity
- Configure Microsoft Entra ID integration with Databricks and Unity Catalog.
- Implement RBAC and ABAC access control models.
- Configure Managed Identities, Service Principals, and authentication mechanisms.
- Configure Databricks networking including:
- VNet Injection
- Secure Cluster Connectivity
- Private connectivity
- Access Connectors
- Support Hub-and-Spoke networking patterns and enterprise security standards.
- Enable secure connectivity between Databricks, ADLS, source systems, and enterprise platforms.
- Collaborate with Infrastructure, Security, and Architecture teams for approvals and security compliance.
Databricks & Unity Catalog
- Deploy and configure Databricks accounts and workspaces.
- Configure Unity Catalog metastore, catalogs, schemas, and storage locations.
- Configure storage credentials and external locations.
- Implement Unity Catalog governance and access controls.
- Establish workspace administration and environment isolation standards.
- Support Hive Metastore to Unity Catalog migration activities.
- Implement Databricks security, governance, and access management best practices.
CI/CD & Automation
- Develop and maintain Terraform infrastructure templates and modules.
- Implement automated deployment pipelines using Azure DevOps.
- Implement branching, pull-request, release-management, and deployment governance standards.
- Use Databricks Asset Bundles (DABs) for Databricks object deployment and environment promotion.
- Implement secure credential management using Azure Key Vault, Managed Identities, and Databricks secrets.
- Maintain source-control and release-management standards.
- Prepare deployment runbooks and operational documentation.
Governance & Operating Model
- Implement Unity Catalog RBAC and ABAC models based on security requirements.
- Configure ownership for catalogs, schemas, storage credentials, and external locations.
- Define and implement UC admin groups, data-owner groups, and platform-support roles.
- Configure governed access using approved Microsoft Entra ID groups.
- Support access-request, approval, audit, and recertification processes.
- Configure monitoring and audit logging for BAU operations and security reviews.
- Enable Unity Catalog audit, lineage, and access-governance capabilities.
- Implement cost monitoring, tagging standards, and usage-optimization controls across Azure and Databricks.
- Create UC administration runbooks covering onboarding, ownership, access changes, and support processes.
Must-Have SkillsAzure & Infrastructure
- Azure Subscriptions and Resource Groups
- Azure DevOps
- ADLS Gen2
- Azure Networking and Security
- Microsoft Entra ID
- Terraform
- Databricks Asset Bundles (DABs)
Databricks
- Databricks Workspace Administration
- Databricks Account Administration
- Unity Catalog
- Unity Catalog Metastore
- Storage Credentials
- External Locations
- Hive Metastore to Unity Catalog Migration
- Databricks Security & Governance
Security & Governance
- RBAC and ABAC
- Managed Identities
- Service Principals
- Private Networking
- Private Endpoints
- Data Governance
- Access Management
Nice-to-Have Skills
- Azure Key Vault
- Azure Private DNS
- Microsoft Purview
- Delta Live Tables (DLT)
- FinOps / Cloud Cost Optimization
- Data Governance Frameworks
- Data Vault
- Lakehouse Architecture
- Enterprise Data Platform Modernization
- Unity Catalog implementation/support experience
Soft Skills
- Strong collaboration across Infrastructure, Security, Data Engineering, and Architecture teams.
- Ability to conduct technical workshops and governance discussions with stakeholders.
- Strong verbal and written communication skills.
- Ability to explain complex technical concepts to technical and non-technical audiences.
- Proactive approach to identifying risks, dependencies, and operational challenges.
- Comfortable working with distributed onshore/offshore teams.
- Strong documentation and knowledge-sharing skills.
- Outcome-focused approach with emphasis on secure, scalable, and operationally sustainable solutions.
Ideal Candidate
The ideal candidate should have hands-on experience in Azure Databricks platform administration and Unity Catalog implementation, combined with strong Azure infrastructure, Terraform, networking, identity, security, and CI/CD knowledge.
Candidates with only Databricks Data Engineering experience but without hands-on Unity Catalog, Azure infrastructure, and platform administration experience may not be suitable for this position.
Apply with your updated resume highlighting relevant Azure, Databricks, Unity Catalog, Terraform, and infrastructure experience.
Pay: ₹60,000.00 - ₹70,000.00 per month
Benefits:
Work Location: Remote
