Job Description: Senior QA Lead – Agentic AI Testing (Data Discovery & Classification)
Job Title: Senior QA Lead / Test Manager – Agent-Based Product Testing (Data Discovery & Classification)
About the Role
We are looking for a Senior QA Lead to own the quality strategy and execution for our AI agent–based Data Discovery and Classification product. This role requires someone who understands both traditional QA leadership (test planning, team management, release governance) and the emerging discipline of testing autonomous/agentic AI systems — where correctness isn't just "pass/fail" but includes accuracy of classification, data discovery coverage, model behavior, hallucination/false-positive rates, and decision traceability.
You will lead a team of QA engineers, define the test strategy for agentic workflows, and act as the final quality gatekeeper before releases involving AI-driven data scanning, classification, tagging, and remediation features.
Key Responsibilities
Leadership & Team Management
-
Lead, mentor, and manage a team of QA engineers/analysts; conduct performance reviews, skill-gap assessments, and career development planning.
-
Define test team structure, allocate resources across sprints/releases, and manage workload distribution.
-
Act as the primary QA point of contact for product managers, data science/ML teams, and engineering leads.
-
Drive hiring, onboarding, and training of QA team members on agentic testing methodologies.
-
Establish and run QA rituals: sprint test planning, defect triage, retrospectives, and release sign-off reviews.
Test Strategy for Agentic AI Products
-
Design and own the end-to-end test strategy for AI agent-based workflows involved in data discovery (crawling, scanning structured/unstructured sources) and classification (PII, PHI, PCI, sensitive data types, custom taxonomies).
-
Define evaluation frameworks for agent behavior: task completion accuracy, decision reasoning consistency, tool-call correctness, and multi-step workflow validation.
-
Establish metrics for classification quality: precision, recall, F1 score, false positive/negative rates, and drift over time.
-
Build test suites to validate agent behavior under varied data types, formats, volumes, and edge cases (structured DBs, unstructured files, cloud storage, SaaS repositories, etc.).
-
Define regression strategies for non-deterministic AI outputs (golden datasets, tolerance thresholds, statistical validation instead of exact-match assertions).
-
Own adversarial/red-team style testing to identify hallucinations, misclassification patterns, and agent failure modes.
Functional & Non-Functional Testing
-
Oversee functional testing of discovery connectors (databases, file shares, cloud storage, SaaS apps) and classification engines.
-
Ensure coverage of performance/scale testing (large datasets, high-volume scans), security testing (data handling, access controls), and compliance validation (GDPR, HIPAA, PCI-DSS relevant scenarios where applicable).
-
Validate integration points: APIs, agent orchestration layers, LLM/model endpoints, and downstream reporting/dashboarding.
-
Own test automation strategy — including automation for repeatable agent evaluation pipelines (prompt regression, model output comparison, synthetic data generation).
Process & Quality Governance
-
Define entry/exit criteria, release readiness checklists, and quality KPIs/dashboards for leadership visibility.
-
Own defect management lifecycle — triage, prioritization, root cause analysis, and closure tracking.
-
Partner with Product and Data Science teams to translate classification requirements and business rules into testable acceptance criteria.
-
Drive continuous improvement of QA processes, tooling, and test data management (including synthetic/sensitive test data handling).
-
Present test status, quality risks, and go/no-go recommendations to stakeholders and leadership.
Required Skills & Experience
QA Leadership
-
8+ years in QA/testing roles, with at least 3+ years leading QA teams.
-
Proven experience managing test planning, execution, and delivery across multiple concurrent releases.
-
Strong stakeholder management and communication skills — able to translate technical quality risks into business impact.
AI/Agentic & Data Product Testing
-
Hands-on experience testing AI/ML-based or agent-based products (chatbots, copilots, autonomous agents, or classification/recommendation engines).
-
Understanding of LLM-based agent architectures (prompts, tool calls, reasoning chains, orchestration frameworks like LangChain/AutoGen/custom agent frameworks).
-
Familiarity with data classification/discovery concepts: PII/PHI/PCI detection, taxonomy-based tagging, entity recognition, pattern/regex-based and ML-based classifiers.
-
Experience defining quality metrics for non-deterministic systems (precision/recall, confusion matrices, confidence thresholds).
Technical Skills
-
SQL and experience testing across structured and unstructured data sources.
-
Understanding of cloud platforms (AWS/Azure/GCP) and common data repositories (S3, databases, SaaS connectors like SharePoint, Google Drive, Salesforce).
Automation Expertise (Core Requirement)
-
6+ years hands-on test automation experience, with proven ability to build frameworks from scratch, not just maintain existing ones.
-
UI/API Automation: Strong command of Selenium, Playwright, or Cypress for UI; Postman, REST Assured, or Karate for API automation. Able to design page-object/modular frameworks for scale.
-
Programming/Scripting: Proficient in Python (preferred for AI/data testing) and/or Java; able to write reusable libraries, not just scripts.
-
Agent/AI-Specific Automation:Building automated evaluation pipelines for agent outputs — prompt regression suites, golden-dataset comparisons, LLM-as-judge scoring, and tolerance-based (non-exact-match) assertions.
-
Automating classification accuracy checks at scale (batch-running thousands of test documents/records through the classifier and auto-scoring precision/recall/F1).
-
Experience with tools/libraries such as LangChain/LangSmith, Promptfoo, DeepEval, TruLens, or equivalent LLM-eval/observability frameworks (any hands-on exposure is a strong plus).
-
Automating synthetic/sensitive test data generation (fake PII/PHI/PCI datasets) to validate discovery and classification without exposing real sensitive data.
-
Data Pipeline Automation: Ability to automate ingestion/validation checks across structured (DB/SQL) and unstructured (files, cloud storage, SaaS) data sources used by discovery agents.
-
CI/CD & Framework Ownership: Deep experience integrating automation suites into CI/CD pipelines (Jenkins, GitHub Actions, GitLab CI, Azure DevOps); ownership of test execution reporting, flaky-test triage, and parallel/distributed test execution (e.g., Selenium Grid, cloud device/browser farms).
-
Test Management & Traceability: Experience with JIRA, TestRail, Zephyr, or Xray for linking automated test coverage to requirements and defects.
-
Ability to define and track automation ROI metrics (coverage %, execution time reduction, defect leakage) and present these to leadership.
-
Prior experience setting up an automation Center of Excellence (CoE) or mentoring a team on automation best practices is a strong plus.
Domain/Compliance (Preferred)
-
Familiarity with data privacy/security regulations (GDPR, CCPA, HIPAA, PCI-DSS) as they relate to data classification.
-
Experience in data governance, DSPM (Data Security Posture Management), or DLP product testing is a strong plus.
Qualifications
-
Bachelor's/Master's degree in Computer Science, Information Technology, or related field.
-
ISTQB or equivalent QA certification preferred.
-
Certifications in AI/ML testing, data privacy (CIPT/CIPM), or cloud platforms are a plus.
Key Attributes We're Looking For
-
Strong analytical mindset — comfortable defining quality bars for outputs that aren't simply right/wrong.
-
Ability to balance rigorous process with the flexibility needed for fast-evolving AI products.
-
Hands-on leader who can dive into test case design while also managing team priorities and stakeholder expectations.
-
Curious and current on developments in agentic AI, LLM evaluation practices, and data governance trends.
