Job purpose
Looking for a Data QA Engineer to lead data validation and regression testing across capital-markets data pipelines spanning legacy and modern platforms. The role will focus on ensuring data accuracy, completeness, consistency, integrity, and adherence to business rules through source-to-target validation, schema checks, reconciliation, lineage review, and automation-led regression testing.
The candidate will build reusable Python/SQL-based validation frameworks, support batch and cloud execution, validate post-trade data flows, identify data anomalies, and partner with engineering and business teams to embed strong data-quality controls. The ideal candidate combines data QA expertise, capital-markets domain understanding, automation-first thinking, and strong ownership of data quality across environments.
Key responsibilities
- Perform source-to-target data validation across legacy and modern systems, including SQL Server and cloud-based data environments.
- Validate data accuracy, completeness, consistency, timeliness, duplication handling, referential integrity, schema structures, metadata, lineage, and business-rule conformance.
- Build and maintain automation-first regression testing frameworks using Python, PySpark, SQL, and validation utilities.
- Execute field-level, dataset-level, transformation-level, and aggregate-level data checks across source, staging warehouse, data lake, and downstream systems.
- Validate ETL/ELT pipelines, post-trade data flows, stored procedures, batch jobs, and cloud-based execution workflows.
- Integrate regression tests, data validation checks, SQL deployables, and quality gates into CI/CD pipelines using DevOps, Jenkins, GitLab, or equivalent tools.
- Design reusable test cases, validation scenarios, reconciliation checks, automated scripts, and regression packs for repeatable data-quality assurance.
- Write, optimize, and troubleshoot complex SQL queries and stored procedures across SQL Server, Oracle, PostgreSQL, or similar platforms.
- Translate business rules, stored procedure logic, and post-trade process flows into automated validation scripts and assertions.
- Build Python-based utilities, APIs, CLI tools, and reusable frameworks to support data validation, orchestration, API integrations, and reporting workflows.
- Automate execution of AutoSys jobs, batch processes, stored procedures, and validation scripts across development, UAT, and production environments.
- Work with engineering teams to embed data quality controls into pipelines, releases, and operational workflows.
- Identify data anomalies, schema mismatches, duplicate records, null-handling issues, transformation failures, reconciliation breaks, and integrity violations.
- Log defects with clear evidence, impact analysis, root-cause observations, and business context.
- Support capital markets and post-trade validation across allocations, clearing, settlement, confirmations, reconciliations, market data, reference data, and downstream reporting.
Key competencies
- 5–7 years of hands-on experience in Data QA, Data Engineering QA, or Python-based automation roles.
- Strong Python skills for data validation, automation framework development, regression testing, and pipeline verification.
- Working knowledge of PySpark for validating large-scale data pipelines and distributed datasets.
- Solid SQL skills for source-to-target validation, reconciliation, complex queries, stored procedures, schema checks, and defect analysis.
- Experience in data QA, ETL/ELT testing, data pipeline testing, and regression testing across legacy and modern platforms.
- Proven experience building automation-first regression frameworks, reusable validation scripts, test cases, and data verification utilities.
- Strong understanding of data quality dimensions including accuracy, completeness, consistency, timeliness, uniqueness, referential integrity, and business-rule conformance.
- Experience validating metadata, schema structures, data types, constraints, lineage, transformation logic, and downstream outputs.
- Exposure to CI/CD tools such as Azure DevOps, Jenkins, or GitLab, with the ability to integrate automated tests and quality gates into deployment pipelines.
- Familiarity with batch orchestration tools such as AutoSys, Control-M, or Airflow.
- Good analytical skills to understand post-trade, financial, market data, or reference data and translate them into repeatable validation scenarios.
- Experience with JIRA, XRay, agile delivery practices, defect management, and QA process governance.
- Proficiency in leveraging AI-assisted tools such as Claude or GitHub Copilot to accelerate test case generation, SQL
development, data validation, defect analysis, documentation, and test data creation.
- Strong communication, problem-solving, attention to detail, and cross-functional collaboration skills.
Nice to Have
- Exposure to Azure, AWS, ADF, containers, Kubernetes, or cloud-based data platforms.
- Experience with relational and NoSQL databases such as SQL Server, MongoDB, Cassandra, DynamoDB, or similar platforms.
- Exposure to Hadoop, Spark, Databricks, cloud data lakes, and Big Data validation.
- Experience validating batch, streaming, real-time, and Kafka-based ingestion pipelines.
- Capital markets domain exposure, especially post-trade, market data, reference data, trade data, or ION/blotter platform integrations.
- Exposure to DevSecOps, infrastructure as code, and code quality tools such as Terraform, Bicep, Sonar, Mend, or ZAP.
