Designation: Assistant Manager - Data Science
Experience: 5 to 8 Years
Location: Remote-tn, Tamil Nadu , India (REM_TN)
Job Description:
We are looking for an AI Product Quality Assurance Engineer to lead the end-to-end quality assurance process for internally built AI products, with a primary focus on Generative AI applications such as LLM-powered tools, chatbots, and AI agents. The ideal candidate will be responsible for ensuring the reliability, accuracy, safety, and overall quality of AI-driven solutions across the product lifecycle.
The role involves defining and implementing comprehensive QA frameworks tailored for AI systems, covering areas such as functional accuracy, output quality, hallucination detection, fairness, and response consistency. The candidate will design and execute detailed test plans to validate AI model outputs across diverse inputs, edge cases, and adversarial scenarios while developing evaluation rubrics and ground-truth datasets to benchmark performance against business and quality standards.
The candidate should possess strong expertise in Generative AI testing, including validating RAG pipelines for retrieval accuracy, context relevance, and response faithfulness to ensure outputs are grounded and trustworthy. They will identify and document Gen AI-specific failure modes such as prompt injection vulnerabilities, reasoning inconsistencies, context window limitations, and unstable outputs. Collaboration with AI engineers and data scientists will be critical to establish feedback loops that improve model fine-tuning, prompt optimization, and overall system performance.
Additionally, the role requires establishing and maintaining QA testing infrastructure for AI products, including automated regression suites, evaluation pipelines, and test repositories. The candidate will define quality gates and release acceptance criteria to ensure AI products meet defined safety and accuracy thresholds before deployment. Strong communication skills are essential to clearly document and present QA findings, risks, and recommendations to both technical and non-technical stakeholders.
The ideal candidate should have experience in AI/ML testing, strong analytical and problem-solving skills, familiarity with LLMs and Gen AI evaluation methodologies, and the ability to work effectively in a fast-paced, innovation-driven environment.
Responsibilities:
Redesign data architecture to enable AI/ML and Gen AI use cases (RAG, embeddings, vector search)
Build and maintain scalable, high-quality data pipelines for AI systems
Develop semantic layers and reusable data frameworks for AI consumption
Implement AI observability systems to monitor data quality, drift, and pipeline health
Define and enforce data quality standards, SLAs, and proactive issue resolution
Evaluate and integrate modern data tools to optimize the AI data stack
Collaborate with AI/ML teams to translate model requirements into data solutions
Drive adoption of Gen AI capabilities and contribute to overall AI strategy
Skills:
AI-ready data architecture and design, scalable data pipelines for AI/ML, semantic layer and RAG frameworks, AI data observability and monitoring, modern data stack and platform integration
Data quality management and SLA definition, vector search and embedding pipelines, metadata management and data modeling, Gen AI tools and ecosystem awareness, cross-functional collaboration with AI/ML and product teams
