Role description
Senior Azure Data Engineer – Databricks Job Summary
We are looking for a Senior Azure Data Engineer with strong hands-on experience in Azure Databricks, Azure Data Factory, Unity Catalog, Azure DevOps, Python, and PySpark.
The candidate will be responsible for data acquisition and ingestion, ETL development, data modeling, data governance, metadata and lineage management, data quality, testing, CI/CD, deployment, and production support.
Banking domain experience is strongly desired.
Key Responsibilities
- Maintain foundational knowledge of upstream data, including knowledge provided through data profiling, data quality reporting, and production of metadata.
- Support the acquisition and ingestion of data.
- Articulate technical design and development details to non-technical business partners.
- Elicit, analyze, and understand business and data requirements to develop complete business solutions under the guidance of senior peers and managers.
- Develop and maintain data models, including Entity Relationship Diagrams (ERDs) and Dimensional Data Models.
- Develop and maintain ETL jobs using the Bank's standard tools.
- Define and implement business rules, data life-cycle management, governance, lineage, and metadata requirements.
- Ensure data is maintained in compliance with enterprise data standards, policies, and guidelines.
- Develop and maintain data models using industry-standard modeling tools.
- Provide support to development and testing teams to resolve data issues.
- Support partners and stakeholders in interpreting and analyzing data.
- Build effective working relationships and collaborate with peers and partners on deliverables.
- Provide data governance and data availability support.
- Support the QA team with data loads and data analysis/investigations as part of SIT/UAT/PAT testing.
- Provide post-implementation support to the production support team during the warranty period.
- Execute code check-in/check-out into the source code repository as part of source code management.
- Work closely with ITS/ARE teams to support code packaging and deployment (CI/CD) into higher environments.
- Actively participate in design and architecture reviews of the application.
- Raise ServiceNow requests and work with the change management team to support release management activities.
- Understand data engineering initiatives and capabilities, data governance principles, and how they apply across the organization.
- Ensure metadata and data lineage are captured and compatible with enterprise metadata and data management tools and processes.
Technology Stack
- Azure Databricks
- Unity Catalog
- Delta Live Tables
- Azure Data Factory
- Azure Data
- Azure Data Connectors
- Python
- PySpark
- TIBCO
- OnPrem SQL
- SharePoint
- SharedFolders
- Azure DevOps
- Strong Infrastructure skills
- Strong Azure Release Engineer skills
- CI/CD
- Source Code Management
- Code Packaging & Deployment
- Release Management
- Data Modeling
- ETL
- Data Profiling
- Data Quality
- Data Governance
- Metadata
- Data Lineage
- Data Life-Cycle Management
Must-Have Skills
- Azure Databricks – 5+ Years
- Azure Data Factory – 5+ Years
- Azure DevOps – 5+ Years
- Unity Catalog – 2+ Years
- Strong hands-on experience with Python
- Strong hands-on experience with PySpark
- Experience with ETL
- Experience with data acquisition and ingestion
- Experience with data modeling
- Experience with data profiling and data quality reporting
- Experience with data governance
- Experience with metadata and data lineage
- Experience with data life-cycle management
- Experience supporting SIT/UAT/PAT testing
- Strong Infrastructure and Azure Release Engineer skills
- Experience with CI/CD
- Experience with source code management
- Experience with code packaging and deployment into higher environments
- Experience participating in design and architecture reviews
- Experience with production and post-implementation support
- Experience with ServiceNow requests and release/change management activities
- Ability to communicate technical design and development details to non-technical business partners
- Experience with OnPrem SQL
- Experience with Azure Data Connectors
Good-to-Have Skills
- Banking domain experience – strongly desired
- Experience with Delta Live Tables
- Experience with TIBCO
- Experience with SharePoint
- Experience with SharedFolders
- Experience using industry-standard data modeling tools
- Experience with enterprise metadata and data management tools
- Experience working with ITS/ARE teams
- Experience supporting enterprise-level data governance and data availability
- Experience working with large-scale banking/financial data environments
Skills
Azure Data Factory, Python, Azure DevOps
About UST
UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.
