Job Description
UST is looking for a highly skilled Azure Data Lead with 8 to 10 years of experience to contribute to the design and development of robust data solutions on the Azure cloud platform. The ideal candidate should possess a strong background in data engineering, demonstrate expertise in Azure services, and have proficiency in data processing technologies, particularly PySpark.
Responsibilities:
Data Pipeline Development:
Design, implement, and optimize end-to-end data pipelines on Azure, emphasizing efficiency and scalability.
Develop and maintain ETL workflows for seamless data processing.
Azure Cloud Expertise:
Utilize Azure services, including Azure Data Factory, Azure SQL Database, and Azure Databricks, for effective data engineering.
Implement and manage data storage solutions on Azure.
Data Transformation with PySpark:
Leverage PySpark for advanced data transformations, ensuring high-quality and well-structured output.
Implement data cleansing, enrichment, and validation processes using PySpark.
Requirements:
Proven experience as a Data Lead, with a strong emphasis on Azure.
Proficiency in Azure Data Factory and other relevant Azure services.
Expertise in PySpark for data processing and analytics.
Proficiency in programming languages such as Python, SQL, or Scala for data processing.
data engineering,azure data factory,sql,etl,azure databricks,
