AI Quality Analyst – Japanese
Job Type: Full-Time
Work Arrangement: Remote
Schedule: Flexible within your local time zone; full-time availability required
Language: Japanese + English
About the Role
We are looking for Japanese-speaking AI Quality Analysts to help evaluate and improve the quality of advanced AI models.
In this role, you will create realistic, multi-turn conversations and assess how effectively AI models understand and use personal context. You will compare model responses, identify subtle quality issues, and provide clear, detailed feedback that helps improve the next generation of AI systems.
This position is ideal for someone who combines strong Japanese language skills, analytical thinking, attention to detail, and an interest in AI.
Key Responsibilities
- Create realistic, creative, multi-turn prompts based on personal context and experiences.
- Evaluate AI responses for accuracy, relevance, naturalness, helpfulness, and personalization quality.
- Determine whether personalization is appropriately supported by available information.
- Identify incorrect personalization, unsupported assumptions, hallucinations, and forced connections.
- Evaluate whether personal information is integrated naturally rather than unnecessarily repeated or overexplained.
- Compare two AI-generated responses side-by-side and determine which provides the better overall experience.
- Assess responses for factors such as usability, clarity, naturalness, helpfulness, and enjoyment.
- Write concise, structured, and evidence-based rationales explaining evaluation decisions, including references to specific conversation turns.
- Provide detailed annotations and constructive feedback on model behavior.
- Review relevant model/debug information to verify that available data sources and conversation context were appropriately utilized.
- Follow strict data-hygiene procedures, including deleting evaluation conversations when required.
- Work independently while maintaining consistent quality and meeting project requirements.
Key Qualifications
- Japanese proficiency: Excellent ability to read and write Japanese, with strong comprehension of nuanced language and context.
- Strong written and verbal communication skills in English.
- Excellent analytical and critical-thinking abilities.
- Ability to evaluate nuanced or ambiguous AI responses objectively.
- Strong attention to detail and ability to identify subtle differences between responses.
- Ability to write clear, concise, and well-structured evaluation rationales.
- Strong understanding of context, intent, natural language, and conversational quality.
- Creative mindset with the ability to develop realistic and challenging prompts.
- Comfortable working independently in a remote environment.
- Strong organizational and time-management skills.
- Comfortable using AI tools and learning new evaluation platforms.
- Full-time availability in your local time zone.
- Desktop or laptop with a reliable internet connection.
Personal Account Requirement
Because this project evaluates personalized AI experiences, selected candidates may be required to use their primary personal Google account rather than a separate testing account and enable designated personal data sources for authentic evaluation.
Candidates should only participate if they are comfortable with the project's account and data-access requirements. All project-specific privacy, security, and data-handling procedures must be followed.
Preferred Experience
Experience in any of the following is a plus:
- AI evaluation or AI model testing
- LLM evaluation or prompt engineering
- Data annotation or data labeling
- Linguistic evaluation or localization
- Translation or language quality assurance (LQA)
- Content moderation or quality assurance
- NLP, computational linguistics, or language technology
- Technical writing or analytical research
Prior AI experience is helpful but not required if you have excellent Japanese language skills, strong analytical abilities, and a genuine interest in evaluating AI.
What You'll Do Day to Day
A typical workflow may involve:
- Creating a multi-turn conversation designed to test a specific personalization capability.
- Reviewing how the AI interprets and uses relevant personal context.
- Checking whether claims about the user are properly grounded in available information.
- Identifying hallucinations, incorrect assumptions, or unnatural personalization.
- Comparing two model responses side-by-side.
- Selecting the stronger response based on defined quality criteria.
- Writing a clear rationale explaining your decision.
- Reviewing required debug information and completing annotations.
- Following data-hygiene procedures before completing the task.
Who We're Looking For
We're looking for people who naturally ask questions like:
"Is this response actually supported by the information available?"
"Did the AI understand what I was really asking?"
"Does this personalization feel natural, or is the model forcing a connection?"
If you enjoy analyzing language, spotting subtle differences, challenging AI systems, and explaining why one response is better than another, this role could be a great fit.
Technical Requirements
- Desktop or laptop computer
- Reliable high-speed internet connection
- Ability to work remotely
- Ability to use your personal Google account as required for the project
- Comfortable working with web-based AI evaluation tools
Equal Opportunity
We welcome qualified candidates from diverse backgrounds and encourage applicants with strong Japanese language expertise, analytical skills, and an interest in AI to apply.
Apply now to help evaluate and improve the next generation of personalized AI.
Pay: ₹1,400.00 - ₹2,400.00 per hour
Expected hours: 10.0 – 40.0 per week
Benefits:
- Flexible schedule
- Paid time off
- Work from home
Work Location: Remote
