Role Summary
We are looking for a detail-oriented QA Analyst to perform manual quality checks on AI-driven systems including LLM responses, automated email replies, and intent classification models. The role involves evaluating accuracy, relevance, tone, and consistency of outputs, and identifying improvement opportunities.
Key Responsibilities
1.LLM Output Quality Check
- Review AI-generated responses for:
- Accuracy and factual correctness
- Relevance to user query
- Clarity and completeness
- Identify incorrect, hallucinated, or misleading responses
- Ensure adherence to brand tone and communication guidelines
2.Email Bot Response Validation
- Perform manual QC of automated email replies
- Check for:
- Grammar, tone, and professionalism
- Contextual correctness of response
- Missing or irrelevant content
- Flag escalation-worthy responses or critical errors
3.Intent Prediction Evaluation
- Validate intent classification accuracy on sample data
- Compare predicted intent vs. actual intent
- Highlight misclassification trends and patterns
- Suggest improvements to training datasets
4.Error Analysis Reporting
- Maintain QC logs with detailed error categorization:
- Incorrect response
- Partial response
- Over-response
- Misclassification
- Share periodic reports with insights and recommendations
- Track quality metrics (accuracy %, error rate, etc.)
5.Process Improvement Feedback
- Provide actionable feedback to AI/ML and product teams
- Suggest rule improvements or prompt tweaks
- Assist in refining SOPs for QC processes
6.Adherence to QA Framework
- Follow defined QC sampling methodology
- Ensure consistency in evaluation criteria
- Support calibration sessions with stakeholders
Key Requirements
