Role description
We are seeking an experienced **AI Developer + Testing Specialist** with 6+ years of hands-on experience in AI/ML development, testing, and implementation across multiple AI platforms including Copilot, Claude, and other major AI tools. This role requires expertise in developing robust AI applications, comprehensive testing methodologies, quality assurance, and integration with enterprise systems. ## Key Responsibilities ### Development & Implementation - Design, develop, and deploy AI-powered applications and solutions using Copilot, Claude, and other major AI platforms - Create custom AI workflows, prompts, and integrations tailored to business requirements - Develop RAG (Retrieval Augmented Generation) systems, chatbots, and conversational AI applications - Implement API integrations with various AI models and services - Write clean, maintainable, and well-documented code following best practices and coding standards - Collaborate with product teams to understand requirements and translate them into technical specifications ### Testing & Quality Assurance - Develop comprehensive test strategies and test plans for AI applications - Perform functional, integration, performance, and security testing of AI-powered solutions - Conduct prompt engineering validation and optimization testing - Create and maintain automated test suites for AI/ML pipelines - Perform edge case and adversarial testing to identify model limitations and vulnerabilities - Validate output quality, accuracy, latency, and consistency across different AI models - Document test results, defects, and recommendations with detailed analysis ### AI Model Evaluation & Optimization - Evaluate and compare different AI models (GPT-4, Claude, Copilot, LLaMA, etc.) for specific use cases - Assess model performance, accuracy, cost-effectiveness, and suitability - Fine-tune prompts and configurations for optimal performance - Monitor model performance in production and recommend improvements - Conduct A/B testing and performance benchmarking ### Documentation & Knowledge Management - Create technical documentation for AI implementations and testing procedures - Develop training materials and best practice guides for AI tool usage - Maintain comprehensive records of testing methodologies and results - Document lessons learned and recommendations for future projects ### Collaboration & Communication - Work cross-functionally with data scientists, engineers, product managers, and stakeholders - Present findings, recommendations, and test results to technical and non-technical audiences - Participate in code reviews and peer testing sessions - Contribute to team knowledge sharing and continuous improvement initiatives - ## Required Qualifications ### Experience - **6+ years** of professional experience in AI/ML development or testing - **3+ years** of hands-on experience with Copilot, Claude, or similar LLM-based platforms - **2+ years** of dedicated testing and QA experience with AI/ML systems - Proven experience developing production-grade AI applications - Experience with multiple AI platforms and models (OpenAI, Anthropic, Microsoft, Google, Meta, etc.) ### Technical Skills - **Programming Languages:** Python, JavaScript/TypeScript, Java, or similar (Python preferred) - **AI/ML Frameworks:** LangChain, LlamaIndex, OpenAI API, Anthropic API, Azure OpenAI, Copilot - **Testing Frameworks:** pytest, Selenium, Jest, or similar automated testing tools - **API Development & Integration:** RESTful APIs, webhooks, real-time integrations - **Version Control:** Git, GitHub, GitLab - **Cloud Platforms:** Azure, AWS, Google Cloud (at least one) - **Prompt Engineering:** Advanced knowledge of prompt design, optimization, and validation ### Domain Knowledge - Deep understanding of Large Language Models (LLMs) and their capabilities/limitations - Knowledge of RAG (Retrieval Augmented Generation) systems and vector databases - Understanding of AI ethics, bias detection, and responsible AI practices - Familiarity with AI security, data privacy, and compliance requirements - Knowledge of NLP/NLU concepts and applications ### Education & Certifications - Bachelor's degree in Computer Science, Software Engineering, Computer Engineering, or related field - OR equivalent professional experience (6+ years) in place of formal education - Relevant certifications (AI/ML, Cloud, QA) are a plus - ## Preferred Qualifications - **Advanced Certifications:** AWS Certified Machine Learning, Microsoft Azure AI Engineer, Google Cloud AI Engineer - **Master's degree** in Computer Science, Artificial Intelligence, or related field - Experience with advanced testing methodologies (chaos engineering, security testing, load testing) - Hands-on experience with enterprise AI platforms and tools - Experience with Agile/DevOps methodologies and CI/CD pipelines - Published work, patents, or contributions to open-source AI projects - Experience with vector databases (Pinecone, Milvus, Weaviate, Chroma) - Knowledge of machine learning operations (MLOps) and model monitoring - Experience in financial services, healthcare, or regulated industries - Track record of building scalable, fault-tolerant systems - ## Required Technical Competencies ### AI/LLM Expertise - Proficiency with Claude (Opus, Sonnet, Haiku models) - Proficiency with Copilot (Microsoft Copilot, GitHub Copilot, Copilot Pro) - Experience with OpenAI models (GPT-4, GPT-3.5, fine-tuning) - Understanding of model selection criteria and trade-offs - Expertise in prompt engineering and few-shot learning ### Testing & QA - Test planning and strategy development - Automated testing and continuous integration - Performance and load testing - Security and penetration testing basics - Regression testing and test maintenance - Quality metrics and reporting ### Development Tools & Practices - Proficiency in relevant programming languages - Knowledge of software architecture patterns - Experience with microservices and distributed systems - Database design and optimization (SQL, NoSQL) - API design and development ### Cloud & DevOps - Cloud platform experience (Azure, AWS, or GCP) - Containerization (Docker, Kubernetes) - CI/CD pipeline implementation and management - Infrastructure as Code (IaC) - Monitoring and logging solutions ## Soft Skills Required - **Problem-Solving:** Ability to debug complex AI systems and identify root causes - **Attention to Detail:** Meticulous in testing and quality assurance - **Communication:** Clear documentation and ability to explain technical concepts to non-technical stakeholders - **Collaboration:** Strong teamwork and ability to work effectively in cross-functional teams - **Adaptability:** Ability to quickly learn new AI tools, models, and frameworks - **Ownership:** Takes responsibility for project outcomes and quality - **Initiative:** Self-motivated with ability to identify improvements and take action - **Time Management:** Ability to manage multiple projects and priorities - **Continuous Learning:** Passion for staying updated with AI/ML advancements ## Responsibilities Summary Table | Category | Key Tasks | |-|-| | **Development** | AI app development, API integration, RAG systems, workflow automation | | **Testing** | Test strategy, automated testing, performance testing, prompt validation | | **Evaluation** | Model comparison, performance assessment, A/B testing, benchmarking | | **Documentation** | Technical specs, test reports, best practices, training materials | | **Collaboration** | Cross-team coordination, stakeholder communication, code reviews |
Skills
LangChain, LLMs, MLOps, Python, LlamaIndex, OpenAI, NoSQL, NLP, SQL, Prompt Engineering, Machine Learning
About UST
UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.
