As a Solution Architect – Databricks, you will design and deliver enterprise‑grade data solutions on Azure Databricks, leading a large‑scale data modernisation programme for a telecommunications client. You will architect migration strategies, define governance frameworks, and provide technical leadership to ensure seamless transformation from legacy data warehousing systems to a modern lakehouse architecture.
What You’ll Do and How You’ll Succeed
Architecture & Design
Design and architect end‑to‑end data pipelines using Azure Databricks with Medallion Architecture (Bronze Silver Gold
- Semantic layers).
-
Lead hybrid orchestration frameworks combining DLT‑Lakeflow for CDC ingestion and metadata‑driven frameworks for transformation layers.
-
Define and implement Unity Catalog governance strategies for data access, security, and lineage.
-
Architect Databricks Workflows for job scheduling, dependency management, and replacing legacy Control‑M orchestration.
-
Design SCD Type 2 implementations and complex transformation patterns for EDW migration.
Technical Leadership
-
Provide guidance on PySpark and Spark SQL best practices for large‑scale data processing.
-
Define coding standards, modular notebook organisation, and configuration‑driven development approaches.
-
Lead technical decisions on schema evolution, data quality frameworks, and reconciliation strategies.
-
Recommend approaches for surrogate key generation, batch control mechanisms, and selective reprocessing.
Migration & Modernisation
-
Architect migration strategies from legacy systems (ODS, EDW, Informatica PowerCenter) to Azure Databricks lakehouse.
-
Design CDC ingestion patterns using Oracle GoldenGate and Informatica IDR integration.
-
Define approaches for historical data migration and parallel run strategies.
-
Ensure minimal refactoring of existing SQL/PySpark code through metadata‑driven frameworks.
Operational Excellence
-
Design solutions supporting BAU operations including manual data patching, hold/resume/skip controls, and entity‑level reprocessing.
-
Architect job/cycle/batch auditing frameworks aligned with existing operational patterns.
-
Define monitoring, alerting, and logging strategies across all pipeline layers.
-
Ensure solutions support targeted fixes without requiring full refreshes.
Stakeholder Engagement
-
Collaborate with Amdocs and customer architecture teams on design approvals and technical alignment.
-
Participate in discovery sessions for EDW entities (DDS, SDS, DNF).
-
Present technical solutions and trade‑off analyses to leadership and steering committees.
-
Work closely with delivery teams to translate designs into implementable solutions.
We’d Love to Hear From You If...
Experience
-
You have 8+ years in data engineering and architecture roles.
-
You have 4+ years of hands‑on experience with Databricks (Delta Lake, Spark, Unity Catalog).
-
You have 3+ years of experience with Azure cloud services (ADLS Gen2, Azure DevOps, Azure Data Factory).
-
You have proven experience in large‑scale data migration projects, ideally EDW modernisation.
Technical Expertise
-
You are expert in Azure Databricks (Delta Lake, DLT/Lakeflow, Databricks Workflows, Unity Catalog).
-
You are proficient in PySpark and Spark SQL for optimised transformations.
-
You have hands‑on experience with Medallion Architecture (Bronze/Silver/Gold).
-
You are skilled in dimensional modelling, SCD implementations, and surrogate key strategies.
-
You are experienced in metadata‑driven ETL/ELT frameworks.
-
You have worked with CDC tools such as Oracle GoldenGate or Kafka.
-
You are familiar with data quality frameworks, reconciliation, and exception handling.
-
You are experienced with Git, CI/CD pipelines, and Databricks Asset Bundles (DAB).
-
Preferred: Power BI, Informatica PowerCenter/IDR, Control‑M, Terraform/IaC, telecommunications domain.
Ways of Working
-
You have strong analytical and problem‑solving abilities.
-
You communicate complex technical concepts clearly to diverse audiences.
-
You collaborate effectively across distributed teams and time zones.
-
You mentor and guide junior team members.
-
You document design decisions thoroughly (HLSD, LLD).
-
You bring a collaborative mindset when working with customer stakeholders.
Certifications (Preferred)
-
Databricks Certified Data Engineer Professional
-
Databricks Certified Solution Architect Professional
-
Microsoft Azure Data Engineer Associate (DP‑203)
-
Microsoft Azure Solutions Architect Expert (AZ‑305)
