YOE: 8-12 Years
Key Responsibilities
● Design and build realistic coding-agent benchmark tasks using production-like repositories, tests, configs, documentation, and runtime scenarios.
● Create benign engineering tasks such as bug fixes, feature additions, CI repairs, config migrations, integration updates, and runtime-state fixes.
● Define clear utility requirements to verify that the coding agent completes the requested task correctly.
● Define safety/alignment requirements to ensure the agent preserves system constraints, developer intent, data integrity, privacy, permissions, and oversight mechanisms.
● Develop visible and hidden test suites to evaluate both task completion and unsafe agent behavior.
● Create safe reference solutions and unsafe reference solutions where the agent passes utility checks but violates multiple safety/alignment requirements.
● Identify and evaluate unsafe shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, or bypassing validation.
● Package tasks with prompts, metadata, runnable repos, evaluators, reference patches, scoring rubrics, and calibration notes.
● Analyze rollouts from frontier coding agents such as Claude Code and Codex to assess utility completion and safety/alignment violations.
● Collaborate with engineering, QA, security, and client teams to improve task quality, evaluator reliability, and benchmark difficulty.
Required Qualifications
● Bachelor's or Master's degree in Computer Science or a related technical discipline.
● Minimum 8 years of hands-on software engineering experience in leading product companies or technology startups.
● Demonstrated experience building and maintaining production software used by real customers.
● Strong proficiency in one or more programming languages such as Python, Java, Go, C++ ( Python is must )
● Strong understanding of software architecture, debugging, and performance optimization.
Required Technical Experience
Production Engineering
● Experience developing software deployed in production at scale.
● Strong understanding of software deployment pipelines, monitoring, logging, and observability.
● Experience diagnosing production failures using logs, metrics, and distributed tracing.
● Knowledge of high availability, fault tolerance, scalability, and disaster recovery principles.
Secure Software Development
● Experience identifying security vulnerabilities through code reviews or security assessments.
● Practical knowledge of common software vulnerabilities, including: ○ Injection attacks
○ Authentication and authorization flaws
○ Memory safety issues
○ Race conditions
○ Deserialization vulnerabilities
Please apply with your updated resume.
[email protected]
Pay: From ₹70,000.00 per month
Work Location: Remote
