Role description
Key Responsibilities
Lead the design and architecture of scalable big data solutions using Apache Spark and Hadoop ecosystem technologies.
Own end-to-end development of batch and streaming data pipelines.
Design, optimize, and tune Spark applications for performance, scalability, and reliability.
Provide technical leadership for Hadoop components including HDFS, Hive, HBase, and YARN.
Ensure data quality, data governance, security, and compliance standards are met.
Lead real-time streaming solutions using Spark Structured Streaming and Kafka.
Review code, conduct technical design reviews, and enforce engineering best practices.
Collaborate with architects, product owners, and business stakeholders to translate requirements into technical solutions.
Mentor and guide data engineers and developers.
Support production deployments, monitoring, and incident resolution.
Drive continuous improvement, automation, and adoption of new big data technologies.
Required Technical Skills
Strong experience with Apache Spark (Spark SQL, DataFrames, Datasets, Structured Streaming).
Hands-on experience with Hadoop ecosystem tools (HDFS, Hive, HBase, YARN).
Advanced SQL and data modeling knowledge.
Experience with data formats such as Parquet, ORC, and Avro.
Knowledge of distributed systems and performance tuning.
Experience with Kafka or similar messaging platforms.
Strong programming skills in Python, Scala, or Java.
Experience with CI/CD pipelines and DevOps practices.
Cloud & Platform Experience
Experience with cloud-based big data platforms such as AWS EMR/Glue, Azure Databricks/Synapse, or GCP Dataproc.
Experience migrating Hadoop workloads to cloud-based platforms.
Understanding of cloud security, cost optimization, and scalability best practices.
Skills
spark,cloud computing,data quality management,databricks,
About UST
UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.
