Role description
· Design, build and optimise robust data ingestion pipelines to acquire, transform and load data. · Develop scalable data engineering solutions using Databricks, PySpark, SparkSQL and associated cloud technologies. · Build and maintain secure, reliable and automated data ingestion processes from external vendor systems, including SFTP-based file transfers and other integration methods. · Ensure data is landed, structured, governed and accessible to support reporting, analytics and business use cases. · Work with Business Analysts, Product Owners, Architects and delivery squads to translate business requirements into technical data solutions. · Support data discovery, profiling and validation activities to understand source data structures, data quality issues and data gaps. · Develop and maintain data transformations, curated datasets and data models required to support reporting and analytical use cases. · Monitor, troubleshoot and resolve data ingestion issues, defects and enhancements identified during development, testing, UAT and production support. · Ensure solutions comply with data architecture standards, engineering best practices, security requirements and governance frameworks. · Produce clear technical documentation for data ingestion processes, data flows and operational support requirements. · Provide comprehensive handover documentation and knowledge transfer to the Data Support team following delivery of data ingestion pipelines. · Collaborate within Agile delivery teams, actively contributing to sprint planning, stand-ups, retrospectives and continuous improvement activities. · Identify opportunities to improve pipeline performance, automation, scalability and maintainability through process and technology enhancements. · Support knowledge sharing and contribute to Engineering and Data Communities of Practice. Skills & Experience Essential · Proven experience designing, building and supporting enterprise-scale data ingestion pipelines and ETL/ELT solutions. · Strong hands-on experience with Databricks, PySpark and SparkSQL. · Experience developing and supporting secure data integrations using SFTP and other file-based or API-driven ingestion mechanisms. · Experience ingesting and processing structured, semi-structured and unstructured data from internal and third-party source systems. · Strong understanding of data modelling, transformation techniques and data warehousing principles. · Experience working with cloud-based data lake and analytics platforms. · Strong understanding of batch and near real-time data processing patterns. · Experience conducting data profiling, discovery and validation activities to assess data quality, completeness and suitability for business requirements. · Experience implementing data quality checks, reconciliations and monitoring processes. · Ability to investigate and resolve ingestion, transformation and data quality issues identified during testing, UAT or production support. · Understanding of data governance, security, data lineage and documentation standards. · Experience producing technical documentation and operational handover materials. · Strong stakeholder engagement skills with the ability to work effectively across business, architecture, engineering and analytics teams. · Experience working within Agile delivery environments. · Knowledge of source control, CI/CD practices and release management processes. · Ability to work independently while collaborating effectively within cross-functional squads. Desirable · Experience integrating data from retail technology platforms, IoT devices or third-party vendor systems. · Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (eSEL) technologies. · Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services. · Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets. · Experience working within large-scale retail or data transformation programmes.
Skills
Agile, ETL, PySpark, Azure Data Factory
About UST
UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.
