Job Description – Databricks Platform Manager / Engineer (AWS)Job Details
Job Title: Databricks Platform Manager / Engineer
Experience: 5–10 Years
Employment Type: Contract (6–12 Months)
Work Mode: Remote
Job Summary
We are seeking an experienced Databricks Platform Manager / Engineer with 5–10 years of overall IT experience, including 3–5+ years of hands-on expertise in administering enterprise Databricks platforms on AWS.
The ideal candidate will be responsible for designing, managing, securing, and optimizing the Databricks platform across multiple environments while ensuring scalability, governance, operational excellence, and cost efficiency. This role is focused on platform engineering rather than data engineering or analytics development.
Key ResponsibilitiesPlatform Administration
- Design, deploy, and manage Databricks workspaces across Development, Test, UAT, and Production environments.
- Establish workspace standards, networking configurations, and environment isolation.
- Plan, test, and implement Databricks Runtime (DBR) upgrades.
- Ensure platform availability, reliability, scalability, and performance.
Compute & Cluster Management
- Configure and manage cluster policies, instance pools, and job clusters.
- Standardize cluster configurations across teams.
- Optimize cluster startup times and resource utilization.
- Prevent oversized or non-compliant cluster deployments.
Security & Identity Management
- Integrate Databricks with AWS IAM.
- Configure authentication, authorization, SSO, and identity management.
- Manage Service Principals, Personal Access Tokens (PATs), and Secret Scopes.
- Implement least-privilege access controls and enterprise security policies.
Governance & Data Access
- Administer Unity Catalog.
- Manage catalogs, schemas, tables, permissions, and RBAC.
- Enforce governance policies and secure data access.
- Support compliance, audit, and regulatory requirements.
DevOps & CI/CD
- Configure Git integration using Databricks Repos.
- Support CI/CD pipelines for notebooks, workflows, and infrastructure.
- Collaborate with DevOps teams on deployment automation.
- Implement Infrastructure as Code (IaC) using Terraform.
Monitoring & Operations
- Monitor platform health, clusters, jobs, and overall performance.
- Troubleshoot incidents and perform root cause analysis.
- Maintain operational dashboards, alerts, and monitoring frameworks.
- Support production releases and ongoing platform maintenance.
Cost Optimization
- Monitor Databricks consumption and AWS cloud costs.
- Optimize cluster sizing, auto-scaling, and scheduling strategies.
- Recommend cost-saving initiatives and improve compute utilization.
Compliance & Audit
- Configure audit logging.
- Ensure adherence to enterprise security and governance standards.
- Support internal and external audit activities.
- Maintain platform documentation and operational standards.
Required SkillsDatabricks
- Workspace Administration
- Cluster Management
- Cluster Policies
- Instance Pools
- Job Clusters
- Databricks Runtime (DBR) Management
- Workflows
- Databricks Repos
- Unity Catalog
- Secret Scopes
- Service Principals
- Personal Access Tokens (PATs)
AWS
- IAM
- VPC Fundamentals
- EC2
- Amazon S3
- CloudWatch
- AWS KMS
- Security Groups
- Networking Fundamentals
DevOps & Automation
- Git
- CI/CD
- Terraform
- Infrastructure as Code (IaC)
- Azure DevOps / GitHub Actions / Jenkins
Monitoring & Operations
- Platform Monitoring
- Incident Management
- Log Analysis
- Performance Tuning
- Cost Optimization
Required Experience
Candidates should have hands-on experience with:
- Managing enterprise Databricks platforms in production environments
- Setting up and administering multiple Databricks workspaces
- Configuring and managing Unity Catalog
- Integrating Databricks with AWS IAM
- Creating and enforcing Cluster Policies
- Performing Databricks Runtime upgrades
- Implementing CI/CD pipelines
- Git integration using Databricks Repos
- Platform monitoring and troubleshooting
- Secret management and security controls
- Audit logging and compliance
- Production support
- Cloud cost optimization initiatives
Preferred Qualifications
- Experience managing enterprise Databricks environments with 100+ users.
- Experience supporting multiple business units or enterprise data platforms.
- Hands-on experience provisioning Databricks infrastructure using Terraform.
- Knowledge of Lakehouse Architecture and Delta Lake.
- Experience implementing FinOps best practices for cloud cost optimization.
- Familiarity with enterprise data governance and security frameworks.
Nice to Have
- AWS Organizations
- AWS Control Tower
- PrivateLink & VPC Endpoints
- CloudWatch, Datadog, or Splunk
- Apache Spark Internals
- AI/ML workloads on Databricks
Education
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.
Preferred Certifications
- Databricks Certified Data Engineer Professional
- Databricks Certified Platform Administrator
- AWS Certified Solutions Architect – Associate/Professional
- AWS Certified SysOps Administrator
- HashiCorp Terraform Associate
Pay: ₹100,000.00 - ₹150,000.00 per month
Work Location: Remote
