Role description
Data Engineer – Spark & Streaming (GCP)
The Data Engineer – Spark & Streaming is responsible for designing, developing, and maintaining scalable batch and real-time data pipelines using Apache Spark, Kafka, and Google Cloud Platform (GCP). The role focuses on building high-performance data platforms, optimizing data processing, and delivering reliable data solutions for analytics, reporting, and machine learning workloads.
- Design, develop, and maintain ETL/ELT pipelines for batch and real-time data processing.
- Build scalable data pipelines using Apache Spark (PySpark/Scala) and streaming technologies such as Kafka and Flink.
- Develop and optimize cloud-native data architectures on Google Cloud Platform (GCP).
- Build and manage data lakes, data warehouses (BigQuery), and streaming platforms.
- Optimize Spark jobs, SQL queries, and data processing workflows for performance and cost efficiency.
- Implement data quality checks, monitoring, validation, and error-handling mechanisms.
- Integrate data from multiple sources including databases, APIs, messaging systems, and cloud storage.
- Collaborate with data architects, data scientists, and business stakeholders to deliver scalable data solutions.
- Support CI/CD, deployment automation, and production troubleshooting.
- Ensure data security, governance, and compliance with enterprise standards.
- Minimum 8 years of IT experience.
- Minimum 4+ years of recent hands-on experience with Google Cloud Platform (GCP).
- Strong programming skills in Python and SQL.
- Experience with Scala and/or Java is preferred.
- Strong expertise in Apache Spark (Spark SQL, DataFrames, PySpark, Spark Streaming).
- Hands-on experience with Apache Kafka, Google Pub/Sub, or other streaming technologies.
- Experience designing and developing real-time streaming data pipelines.
- Strong knowledge of BigQuery and GCP data services.
- Experience with ETL/ELT frameworks and large-scale data processing.
- Strong performance tuning skills for Spark jobs and SQL queries.
- Experience implementing data quality, monitoring, and validation frameworks.
- Excellent analytical, troubleshooting, and problem-solving skills.
- Experience with Apache Flink.
- Knowledge of Snowflake, Amazon Redshift, or other cloud data warehouses.
- Experience with NoSQL databases such as MongoDB, Cassandra, or Bigtable.
- Hands-on experience with Apache Airflow for workflow orchestration.
- Experience with Databricks.
- Knowledge of Docker and Kubernetes.
- Experience with CI/CD pipelines and DevOps practices.
- Familiarity with machine learning data pipelines and feature engineering.
- Google Cloud Professional Data Engineer certification is an added advantage.
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or equivalent experience.
Skills
big data,spark,bigquery,sql,apache kafka,scala,
About UST
UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.
