Job Title
Infrastructure Data Engineer
Job Description Summary
Job Description
We are seeking a highly skilled Azure Cloud Engineer with deep expertise in designing, implementing, and optimizing Azure Databricks environments. This role will also support a broad range of Azure services—including Web Applications, Azure Data Factory, Application Gateway, Storage Accounts, Lakebase, Azure SQL/Databases, and Azure Monitor—to ensure our cloud ecosystem is secure, scalable, cost-efficient, and aligned with the Five Pillars of the Azure Well-Architected Framework: Operational Excellence, Security, Reliability, Performance Efficiency, and Cost Optimization.
The ideal candidate combines strong hands-on engineering skills with a strategic mindset for cloud governance, automation, and continuous improvement.
Key Responsibilities
Primary Focus: Azure Databricks Engineering
- Design, deploy, and optimize Azure Databricks workspaces, clusters, jobs, and pipelines.
-
Implement best practices for cluster sizing, autoscaling, job orchestration, and cost governance.
-
Collaborate with data engineering teams to build scalable ETL/ELT pipelines using Databricks notebooks, Delta Lake, and Spark.
-
Integrate Databricks with Azure Data Lake, Azure SQL, Azure Data Factory, and other platform services.
-
Implement security controls including Unity Catalog, RBAC, service principals, secret scopes, and network isolation.
-
Monitor Databricks performance and reliability using Azure Monitor, Log Analytics, and Databricks native tools.
Azure Platform Engineering Responsibilities
Azure Application & Integration Services
-
Deploy, configure, and maintain Azure Web Apps, Function Apps, and API Management.
-
Build and maintain Azure Data Factory pipelines for data ingestion, transformation, and orchestration.
-
Configure and optimize Azure Application Gateway, WAF policies, routing rules, and SSL/TLS certificates.
Azure Storage & Database Services
-
Manage Azure Storage Accounts, lifecycle policies, replication, encryption, and access controls.
-
Support Azure SQL, Cosmos DB, and other database services including performance tuning and security hardening.
Monitoring, Governance & Optimization
-
Use Azure Monitor, Log Analytics, Application Insights, and Azure Advisor to proactively identify and resolve issues.
-
Implement alerting, dashboards, and automated remediation workflows.
-
Continuously optimize cloud resources for cost, performance, and reliability.
-
Contribute to cloud governance standards including tagging, resource organization, RBAC, and policy enforcement.
Automation & Infrastructure as Code
-
Build and maintain IaC templates using Terraform, Bicep, or ARM.
-
Automate deployments, configuration, and operational tasks using Azure DevOps, GitHub Actions, or PowerShell.
-
Support CI/CD pipelines for application and data workloads.
Responsibilities Aligned to the 5 Pillars of the Cloud Operating Model
Operational Excellence
-
Develop repeatable deployment patterns and IaC modules.
-
Implement monitoring, logging, and alerting standards across all Azure services.
-
Document architectures, runbooks, and operational procedures.
Security
-
Enforce Zero Trust principles across identity, network, and data layers.
-
Implement RBAC, managed identities, Key Vault, private endpoints, and network security groups.
-
Support compliance initiatives and security assessments.
Reliability
-
Design resilient architectures using availability zones, redundancy, and failover strategies.
-
Implement backup, disaster recovery, and business continuity plans.
-
Monitor system health and proactively address reliability risks.
Performance Efficiency
-
Optimize Databricks clusters, SQL databases, and application services for throughput and latency.
-
Conduct performance testing and capacity planning.
-
Use Azure Monitor and Advisor insights to drive continuous improvement.
Cost Optimization
-
Implement cost-efficient Databricks cluster policies and autoscaling.
-
Monitor and optimize Azure consumption across compute, storage, and networking.
-
Provide recommendations for reserved instances, rightsizing, and lifecycle management.
Required Qualifications
-
3–7+ years of hands-on experience with Microsoft Azure.
-
Strong expertise in Azure Databricks, Spark, Delta Lake, and data engineering concepts.
-
Experience with Azure services such as Web Apps, Data Factory, Application Gateway, Storage Accounts, and Azure SQL.
-
Proficiency with Terraform, Bicep, ARM, or similar IaC tools.
-
Strong scripting skills (PowerShell, Python, Bash).
-
Experience with Azure Monitor, Log Analytics, and Azure Advisor.
-
Solid understanding of cloud networking (VNETs, private endpoints, firewalls, routing).
-
Familiarity with the Azure Well-Architected Framework.
Preferred Qualifications
-
Azure certifications (AZ-104, AZ-204, DP-203, AZ-305, or Databricks certifications).
-
Experience with CI/CD pipelines in Azure DevOps or GitHub Actions.
-
Experience with Unity Catalog, Databricks SQL, or MLflow.
-
Background in data engineering, DevOps, or cloud architecture.
Soft Skills
-
Strong communication and documentation abilities.
-
Ability to collaborate across data, application, and infrastructure teams.
-
Problem-solving mindset with a focus on automation and continuous improvement.
-
Comfortable working in fast-paced, cloud-first environments
INCO: “Cushman & Wakefield”
