Role description
We are seeking an experienced Azure Data Engineer with strong expertise in Azure Databricks and Azure Data Factory to design, develop, and maintain scalable data pipelines and analytics solutions. The ideal candidate will work closely with business stakeholders, data scientists, and engineering teams to deliver high-quality data integration and transformation solutions on the Azure cloud platform.
Key Responsibilities
- Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory.
- Build scalable data processing solutions using Azure Databricks and Apache Spark.
- Ingest data from multiple sources including databases, APIs, flat files, and cloud storage.
- Develop and maintain data transformation workflows using PySpark, Spark SQL, and SQL.
- Implement data quality checks, monitoring, and error-handling mechanisms.
- Integrate data solutions with Azure services such as:
- Azure Data Lake Storage
- Azure Synapse Analytics
- Azure SQL Database
- Optimize Spark jobs for performance and cost efficiency.
- Collaborate with data architects and business teams to understand requirements and deliver solutions.
- Implement CI/CD pipelines and deployment automation.
- Ensure security, governance, and compliance standards are followed.
Required Skills
- Strong experience with:
- Azure Databricks
- Azure Data Factory
- Apache Spark
- PySpark
- SQL
- Experience with Azure cloud services.
- Knowledge of ETL/ELT concepts and data warehousing.
- Experience working with large-scale structured and unstructured datasets.
- Understanding of Delta Lake architecture.
- Experience with Git-based version control.
- Familiarity with REST APIs and data integration patterns.
Skills
azure data factory,sql,databricks,python,performance tuning,
About UST
UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.
