Job Description: Key Role & Responsibilities
- Define lakehouse architecture: medallion (bronze/silver/gold) patterns, batch/streaming designs, and multi-workspace strategies.
- Design and implement data pipelines using Spark , Delta Lake , and Databricks workflows (Jobs/Workflows, DLT where applicable).
- Establish governance and security using Unity Catalog , access controls, lineage, and data quality gates.
- Optimise performance: cluster policies, autoscaling, partitioning, file sizing, caching, Spark tuning, and job orchestration.
- Build CI/CD and release governance for notebooks, repos, jobs, and infrastructure-as-code.
- Integrate Databricks with enterprise ecosystem (cloud storage, event streaming, data warehouse, BI tools).
- Conduct solution workshops with customers; provide options and trade-offs; create phased implementation roadmaps aligned to business value.
Mentor teams, enforce engineering standards, and ensure operational excellence (monitoring, incident response, SRE practices).
-
Must Have
- 10+ years experience with a strong Data Engineering background (ETL/ELT, distributed compute, production-grade pipelines).
- 4+ years hands-on Databricks experience in architecture/technical leadership roles.
- Strong experience in Apache Spark (PySpark/Scala), Delta Lake , pipeline design, and performance tuning.
- Experience with data orchestration and DevOps practices (Git, CI/CD, testing frameworks).
- Experience designing secure data platforms (RBAC, secrets, network/security integration, compliance considerations).
Strong customer-facing skills: requirements discovery, solution design, and stakeholder management.
-
Good to Have
- Streaming experience (Kafka/Event Hubs, Structured Streaming, CDC patterns).
- ML/AI enablement experience (MLflow, feature engineering, model lifecycle) as it relates to platform design.
Cloud certifications or platform-specific certifications.
-
Education
Bachelor’s/Master’s in Computer Science, Engineering, or related fields.
Key Skills: Databricks, Python, Spark, Data Architecture, Data Pipelines
Responsibilities: Key Role & Responsibilities
- Define lakehouse architecture: medallion (bronze/silver/gold) patterns, batch/streaming designs, and multi-workspace strategies.
- Design and implement data pipelines using Spark , Delta Lake , and Databricks workflows (Jobs/Workflows, DLT where applicable).
- Establish governance and security using Unity Catalog , access controls, lineage, and data quality gates.
- Optimise performance: cluster policies, autoscaling, partitioning, file sizing, caching, Spark tuning, and job orchestration.
- Build CI/CD and release governance for notebooks, repos, jobs, and infrastructure-as-code.
- Integrate Databricks with enterprise ecosystem (cloud storage, event streaming, data warehouse, BI tools).
- Conduct solution workshops with customers; provide options and trade-offs; create phased implementation roadmaps aligned to business value.
Mentor teams, enforce engineering standards, and ensure operational excellence (monitoring, incident response, SRE practices).
-
Must Have
- 10+ years experience with a strong Data Engineering background (ETL/ELT, distributed compute, production-grade pipelines).
- 4+ years hands-on Databricks experience in architecture/technical leadership roles.
- Strong experience in Apache Spark (PySpark/Scala), Delta Lake , pipeline design, and performance tuning.
- Experience with data orchestration and DevOps practices (Git, CI/CD, testing frameworks).
- Experience designing secure data platforms (RBAC, secrets, network/security integration, compliance considerations).
Strong customer-facing skills: requirements discovery, solution design, and stakeholder management.
-
Good to Have
- Streaming experience (Kafka/Event Hubs, Structured Streaming, CDC patterns).
- ML/AI enablement experience (MLflow, feature engineering, model lifecycle) as it relates to platform design.
Cloud certifications or platform-specific certifications.
-
Education
Bachelor’s/Master’s in Computer Science, Engineering, or related fields.
Key Skills: Databricks, Python, Spark, Data Architecture, Data Pipelines
Qualifications: Key Role & Responsibilities
- Define lakehouse architecture: medallion (bronze/silver/gold) patterns, batch/streaming designs, and multi-workspace strategies.
- Design and implement data pipelines using Spark , Delta Lake , and Databricks workflows (Jobs/Workflows, DLT where applicable).
- Establish governance and security using Unity Catalog , access controls, lineage, and data quality gates.
- Optimise performance: cluster policies, autoscaling, partitioning, file sizing, caching, Spark tuning, and job orchestration.
- Build CI/CD and release governance for notebooks, repos, jobs, and infrastructure-as-code.
- Integrate Databricks with enterprise ecosystem (cloud storage, event streaming, data warehouse, BI tools).
- Conduct solution workshops with customers; provide options and trade-offs; create phased implementation roadmaps aligned to business value.
Mentor teams, enforce engineering standards, and ensure operational excellence (monitoring, incident response, SRE practices).
-
Must Have
- 10+ years experience with a strong Data Engineering background (ETL/ELT, distributed compute, production-grade pipelines).
- 4+ years hands-on Databricks experience in architecture/technical leadership roles.
- Strong experience in Apache Spark (PySpark/Scala), Delta Lake , pipeline design, and performance tuning.
- Experience with data orchestration and DevOps practices (Git, CI/CD, testing frameworks).
- Experience designing secure data platforms (RBAC, secrets, network/security integration, compliance considerations).
Strong customer-facing skills: requirements discovery, solution design, and stakeholder management.
-
Good to Have
- Streaming experience (Kafka/Event Hubs, Structured Streaming, CDC patterns).
- ML/AI enablement experience (MLflow, feature engineering, model lifecycle) as it relates to platform design.
Cloud certifications or platform-specific certifications.
-
Education
Bachelor’s/Master’s in Computer Science, Engineering, or related fields.
Key Skills: Databricks, Python, Spark, Data Architecture, Data Pipelines
