QA Engineer – GenAI Product Testing & Backend Automation
Experience: 2–3 Years
Location: Hyderabad
Work Mode: Work from Office
Employment Type: Full-time
About the Role
We are looking for a QA Engineer with 2–3 years of experience to test and automate quality assurance for enterprise GenAI-based products.
The candidate will work on AI-driven products such as chatbots, voice bots, agent assist systems, call quality audit platforms, workflow automation tools, and compliance-oriented AI applications.
This role is not limited to traditional UI testing. The main focus will be on backend testing, API testing, LLM interface testing, prompt validation, response quality testing, and automation of repeatable QA scenarios using Python-based frameworks.
The ideal candidate should be comfortable testing both conventional software workflows and AI-driven systems where responses may vary based on prompt, context, user input, and business rules.
Key ResponsibilitiesFunctional and Product QA
- Understand product requirements, business workflows, and customer use cases.
- Prepare test scenarios, test cases, test data, and test execution reports.
- Perform functional, regression, sanity, integration, and UAT support testing.
- Validate end-to-end workflows across frontend, backend, APIs, databases, and LLM-based components.
- Identify, document, and track defects with clear reproduction steps, screenshots, logs, payloads, and expected vs actual results.
- Test edge cases, negative scenarios, incomplete inputs, invalid data, ambiguous user inputs, and fallback flows.
- Support release validation and provide release-readiness inputs to the product and engineering teams.
Backend and API Testing
- Test backend APIs used by GenAI applications and workflow automation systems.
- Validate API request/response payloads, status codes, error handling, authentication, authorization, and data correctness.
- Test integration behavior between AI services, orchestration layers, customer systems, workflow engines, and databases.
- Use tools such as Postman and Python-based API test scripts for backend validation.
- Validate structured outputs such as JSON responses, classification outputs, extracted fields, workflow status, audit results, and decision outputs.
LLM Interface and Prompt Testing
- Test LLM-based interfaces by validating prompt inputs, model responses, business-rule adherence, and output consistency.
- Create and maintain prompt regression test suites for repeated testing across releases.
- Validate whether the AI system asks the right follow-up questions, handles missing information correctly, and avoids unnecessary or incorrect responses.
- Test LLM outputs for relevance, correctness, hallucination, tone, format, completeness, and compliance with expected behavior.
- Validate guardrails, fallback responses, refusal behavior, escalation logic, and restricted-topic handling.
- Test multi-turn conversations and ensure context is maintained correctly across user interactions.
- Compare LLM outputs against expected business outcomes and predefined evaluation criteria.
QA Automation
- Build and maintain Python-based automated test scripts for backend, API, and LLM workflow testing.
- Use frameworks such as pytest or similar Python-based test frameworks.
- Automate regression scenarios for prompts, APIs, workflows, and structured outputs.
- Use JSON, CSV, or YAML-based test datasets to run repeatable test cases.
- Validate API responses using schema validation, assertions, and business-rule checks.
- Generate automated test reports and maintain test execution history.
- Work with engineering teams to integrate automated tests into CI/CD pipelines where applicable.
GenAI Product Quality Evaluation
- Assist in creating evaluation datasets for GenAI product testing.
- Test AI responses across multiple scenarios, user personas, and input variations.
- Validate response consistency across product releases and prompt changes.
- Identify recurring failure patterns in AI responses and report them with examples.
- Support evaluation of model behavior during prompt updates, model upgrades, and workflow changes.
- Help define practical QA metrics for GenAI products, such as response accuracy, completion rate, fallback rate, hallucination rate, and business-rule compliance.
Required Skills
- 2–3 years of experience in software QA/testing.
- Strong understanding of manual testing, regression testing, sanity testing, integration testing, and UAT support.
- Good experience in API testing using Postman or similar tools.
- Basic to intermediate Python scripting knowledge.
- Experience or strong willingness to work with Python-based automation frameworks such as pytest.
- Ability to write automated tests for APIs, backend workflows, and structured responses.
- Understanding of JSON, REST APIs, request/response payloads, HTTP status codes, and error handling.
- Ability to prepare detailed test cases, test data, bug reports, and test execution summaries.
- Strong analytical thinking and attention to detail.
- Good written and verbal communication skills.
- Ability to understand business workflows and convert them into testable scenarios.
Good to Have Skills
- Exposure to GenAI, LLMs, chatbots, voice bots, or AI-based workflow products.
- Understanding of prompt testing, LLM response validation, hallucination testing, and guardrail testing.
- Experience with Python libraries for API testing, schema validation, and automated reporting.
- Exposure to tools such as pytest, requests/httpx, Pydantic, jsonschema, pytest-html, Allure, or similar tools.
- Basic understanding of CI/CD pipelines and automated test execution.
- Experience with databases and basic SQL validation.
- Exposure to conversational AI testing, call center automation, workflow automation, BFSI, travel, compliance, or enterprise SaaS products.
- Familiarity with LLM evaluation tools such as DeepEval, Ragas, Promptfoo, LangSmith, or similar tools.
Optional UI Testing Skills
- Basic UI testing experience is useful but not mandatory.
- Exposure to Selenium, Playwright, or Cypress will be considered an added advantage.
- The primary focus of this role will remain backend testing, API testing, LLM interface testing, and Python-based QA automation.
Ideal Candidate Profile
The ideal candidate should be a hands-on QA engineer who can go beyond manual UI testing and contribute to backend automation for AI-driven products.
The candidate should be able to test APIs, validate LLM responses, create prompt regression suites, automate repeatable QA scenarios, and think from both technical and end-user perspectives.
The candidate should be comfortable working in a fast-moving product environment where AI behavior, prompts, workflows, and integrations evolve frequently.
Key Product Areas
The candidate may work on different GenAI-based enterprise products, including:
- Conversational AI chatbots.
- Voice bot and speech-based AI workflows.
- Agent assist systems.
- Call quality audit and analysis platforms.
- Enterprise workflow automation tools.
- Compliance and document intelligence workflows.
- AI-powered dashboards and backend automation systems.
Pay: ₹500,000.00 - ₹600,000.00 per year
Work Location: In person
