Required Qualifications:
- 3+ years of experience in QA, SDET, or Data Quality Engineering roles.
- Strong hands-on experience with SQL for complex data validation and analysis.
- Proficiency in Python for test automation and data validation.
- Experience testing data pipelines and ETL/ELT workflows.
- Hands-on experience with Kafka or other streaming platforms.
- Solid understanding of AWS data services (S3, Glue, Redshift, Lambda, Athena, etc.).
- Experience working with large datasets and distributed systems.
- Strong debugging, analytical, and problem-solving skills.
Key Responsibilities:
- Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness.
- Create and maintain data quality checks (nulls, duplicates, schema drift, referential integrity).
- Verify business rules and transformations using SQL-based validations.
- Design, develop, and execute test strategies for Databricks-based data pipelines and analytics workflows
- Establish data reconciliation and end-to-end traceability between source and downstream systems.
- Test ETL/ELT pipelines built using AWS services (Glue, Lambda, EMR, Step Functions).
- Validate transformations written in SQL and Python.
- Ensure correctness across data ingestion, enrichment, aggregation, and publishing layers.
- Test reprocessing, backfills, and historical data loads.
- Validate ETL/ELT processes built using Apache Spark (PySpark/Scala) in Databricks.
- Validate Kafka-based streaming pipelines for data integrity, ordering, and exactly-once/at-least-once
semantics.
- Test producer and consumer logic, serialization formats (Avro, JSON, Protobuf).
- Validate topic configurations, partitions, offsets, retention policies, and schema changes.
- Simulate and test late arrivals, duplicate events, and consumer failures.
- Test data workflows using AWS S3, Glue, Lambda, Redshift, Athena, Kinesis, DynamoDB, or similar services.
- Validate IAM roles, permissions, and secure data access.
- Verify data lifecycle policies, encryption, and storage optimizations.
- Build and maintain automated data testing frameworks using Python.
- Develop reusable test utilities, fixtures, and synthetic datasets. Integrate data tests into CI/CD pipelines for pre-merge, scheduled, and post-deployment validation.
- Enable automated alerts for data quality failures.
- Validate pipeline performance for large-scale datasets.
- Test throughput, latency, and concurrency under peak workloads.
- Validate retry logic, error handling, idempotency, and recovery mechanisms.
- Perform soak, regression, and failover testing Validate data pipeline metrics, logs, and alerts using CloudWatch, Prometheus, Grafana, or equivalent tools.
- Partner with teams to define data SLAs and SLOs. Participate in incident response, root-cause analysis, and postmortems related to data quality issues.
Pay: ₹10,680.55 - ₹40,056.22 per month
Work Location: In person
