Role Summary:
We are seeking a Data Validation Engineer with Testing with strong hands-on experience building and optimizing data pipelines on Databricks. This role focuses on implementation, performance, and reliability while working within a well-defined architectural framework.
Key Responsibilities:
- Develop and maintain data pipelines on Databricks using medallion architecture principles
- Build, optimize, and maintain Databricks notebooks for ingestion, transformation, and analytics
- Implement data flows across boundary systems ensuring reliability, scalability, and data quality
- Apply performance tuning techniques including query optimization, cluster sizing, and Delta Lake optimization
- Work with Unity Catalog to manage schemas, permissions, and data access controls
- Collaborate closely with architects to implement approved data designs and standards
- Troubleshoot pipeline failures, data issues, and performance bottlenecks
- Ensure code quality, documentation, and operational readiness of data solutions
Required Qualifications:
- Strong hands-on experience with Databricks and Delta Lake
- Strong hands-on experience with AWS services such as S3, EC2, Lambda, Glue, and Redshift.
- Working experience with IBM DataStage for ETL development, data integration, and data transformation.
- Proven ability to design metadata-driven test harnesses that scale — parameterized test configs that can run against 500 jobs today and 80K tomorrow without rearchitecting
- Experience with data validation tooling (Great Expectations, Deequ, or custom PySpark validation libraries) for row count, schema, transformation logic, and output comparison checks
- Performance and throughput benchmarking — rows/sec, latency comparisons between legacy (DataStage) and target (Databricks) pipelines
- Experience with Iceberg Tables
- Proven experience implementing medallion architecture in real world use cases
- Experience building notebooks using Python, SQL, or Scala
- Solid understanding of data integration across multiple systems
- Strong performance tuning and debugging skills in Databricks environments
- Experience working in enterprise scale data platforms
- Hands-on experience with any of the Databricks AI/ML ecosystem components, including Mosaic AI, MLflow, Unity Catalog, Feature Engineering, Vector Search, Databricks Model Serving, and Delta Lake.
- Knowledge of Databricks AI Functions and SQL AI capabilities.
- Experience implementing RAG pipelines using Delta Lake, Unity Catalog, and Vector Search.
- Familiarity with LLMOps, model monitoring, evaluation, and governance using MLflow.
Pay: ₹60.00 - ₹65.00 per hour
Experience:
- Databricks: 5 years (Required)
- DataStage: 4 years (Required)
- AI: 3 years (Required)
- AWS: 3 years (Preferred)
- ETL: 4 years (Preferred)
- Data Validation: 5 years (Preferred)
- Iceberg Tables: 3 years (Preferred)
- Pyspark: 5 years (Preferred)
- Data lake: 4 years (Preferred)
- RAG Pipelines: 3 years (Preferred)
- LLM: 3 years (Preferred)
Work Location: Remote
