IND Staff Engineer, Reliability - GCC070
We’re determined to make a difference and are proud to be an insurance company that goes well beyond coverages and policies. Working here means having every opportunity to achieve your goals – and to help others accomplish theirs, too. Join our team as we help shape the future.
Position Summary
We are seeking a highly skilled T7 AI Operations & Site Reliability Engineer to join our engineering team in Hyderabad, India. This role is laser-focused on the availability, reliability, and performance of our production AI systems. You will own the operational health of AI-powered products — ensuring LLM-based services, agentic workflows, RAG pipelines, and ML inference platforms maintain enterprise-grade uptime while scaling to meet demand. You will build the observability, automation, and incident response capabilities that keep our AI products running 24/7.
Level: T7 (Senior Engineer)
Location: Hyderabad, India
Employment Type: Full-Time
Key Responsibilities
AI Platform Reliability & Uptime
-
Own the end-to-end reliability of production AI systems including LLM services, RAG pipelines, agentic workflows, and inference endpoints
-
Define and maintain SLOs/SLIs/SLAs for AI products — latency, availability, error rates, token throughput, and response quality
-
Design and implement high-availability architectures for AI workloads: multi-region failover, load balancing, auto-scaling, and graceful degradation
-
Build circuit breakers, retry logic, fallback models, and rate-limiting strategies to ensure AI services remain available under stress
-
Drive availability targets of 99.9%+ for critical AI-powered products
-
Establish disaster recovery procedures and regularly test backup/restore for AI data stores and model artifacts
Observability & Monitoring
-
Build and maintain comprehensive observability stacks for AI systems — metrics, logs, traces, and AI-specific signals (hallucination rates, model drift, token costs)
-
Implement real-time dashboards and alerting for AI service health, model performance, and infrastructure utilization
-
Design anomaly detection and proactive alerting to identify degradation before users are impacted
-
Monitor LLM provider dependencies (GCP Vertex AI, OpenAI, etc.) and implement automated failover when external services degrade
-
Track and optimize cost-per-inference, token utilization, and resource efficiency across AI workloads
Incident Management & Response
-
Lead incident response for AI system outages and degradations — triage, mitigate, resolve, and communicate
-
Build and maintain runbooks for common AI failure modes: model timeouts, context window overflows, embedding pipeline failures, vector DB issues
-
Establish on-call rotations and escalation procedures tailored to AI system failure patterns
-
Automate incident detection and remediation where possible — self-healing pipelines and auto-rollback
AI Infrastructure & Platform Operations
-
Operate and scale cloud-native AI infrastructure (GCP, AWS) including model serving platforms, Kubernetes containers, and vector databases
-
Implement and maintain Infrastructure-as-Code (Terraform) for AI platform environments
-
Automate deployment pipelines for model updates, configuration changes, and infrastructure scaling
Collaboration & Documentation
-
Partner closely with AI Engineers to ensure new features are built with operability, observability, and reliability in mind
-
Define production-readiness criteria for AI services — ensuring all systems meet reliability standards before launch
-
Maintain comprehensive operational documentation: architecture diagrams, runbooks, playbooks, and SOPs
-
Contribute to architecture reviews with a reliability lens — identifying single points of failure, blast radius, and operational risk
-
Participate in on-call rotations and drive continuous improvement of operational practices
Required Qualifications
-
Experience: 8+ years of professional experience in software engineering, DevOps, or site reliability engineering, with 1+ year operating AI/ML systems in production
-
Education: Bachelor's degree in Computer Science, Software Engineering, or related field (or equivalent experience)
-
SRE Fundamentals:
-
Deep understanding of SRE principles: SLOs, error budgets, toil reduction, incident management, and capacity planning
-
Proven track record maintaining high availability (99.9%+) for production systems at scale
-
Experience integrating into observability platforms (Prometheus, Grafana, Datadog, Splunk, or equivalent)
-
Strong incident response skills with experience leading war rooms and post-incident reviews
-
AI/ML Operations:
-
Understanding of AI-specific failure modes: model drift, hallucination spikes, token limit errors, embedding pipeline failures, and provider outages
-
Familiarity with LLM providers and platforms (GCP Vertex AI, OpenAI, AWS Bedrock) from an operational perspective
-
Cloud & Infrastructure: Advanced-level experience with cloud platforms (GCP, AWS), Kubernetes, containerization, and Infrastructure-as-Code (Terraform)
-
Programming: Strong proficiency in Python and at least one systems language; comfortable writing automation scripts, custom exporters, and operational tooling
-
Networking & Security: Solid understanding of networking, load balancing, DNS, TLS, and security best practices for cloud-native systems
-
CI/CD: Experience building and maintaining deployment pipelines (Jenkins, GitHub Actions, ArgoCD) with automated rollback capabilities
-
Communication: Excellent communication skills for incident coordination, stakeholder updates, and cross-team collaboration
Preferred Qualifications
-
Knowledge of AI cost optimization strategies — model routing, caching, batching, and tiered inference
-
Experience in regulated industries (insurance, finance, healthcare) with compliance and audit requirements
-
Cloud certifications (GCP Professional Cloud Architect, AWS Solutions Architect, CKA/CKAD)
-
Experience with AIOps — using AI/ML to improve operational intelligence and automated remediation
