Job Title: Associate Director – SRE & Observability Engineer (AI Infrastructure)
Role Overview
We are seeking a seasoned Site Reliability Engineering (SRE) and Observability leader to design, build, and scale reliability frameworks for AI/GenAI platforms and data-intensive workloads.
This role will focus on ensuring high availability, performance, scalability, and cost-efficiency across AI infrastructure (LLMs, model training/inference, vector databases, pipelines) by embedding SRE principles, observability, and automation into the platform lifecycle.
Key Responsibilities
1. SRE Strategy for AI Infrastructure
-
Define and lead SRE strategy and operating model for AI platforms across cloud (Azure, AWS, GCP) and hybrid environments
-
Establish SLIs, SLOs, and SLAs tailored to:
-
LLM inference latency and throughput
-
Model training performance and job success rates
-
Pipeline reliability (RAG, orchestration frameworks, agents)
-
Drive adoption of error budgets and reliability engineering practices across AI and platform teams
2. Observability Architecture for AI Workloads
-
Design and implement end-to-end observability frameworks for AI systems, including:
-
Metrics (latency, throughput, GPU utilization, token usage)
-
Logs (model behavior, system failures, prompt traces)
-
Traces (distributed AI workflows, API calls, orchestration flows)
-
Build observability for:
-
LLM pipelines and agent-based systems
-
Vector databases and retrieval layers
-
Data ingestion and feature pipelines
-
Enable deep visibility into model performance, drift, and degradation
3. Reliability Engineering & Automation
-
Implement self-healing systems, auto-remediation, and resiliency patterns
-
Design fault tolerance strategies:
-
Multi-region deployment
-
Model fallback and routing strategies
-
Graceful degradation in GenAI systems
-
Lead adoption of:
-
Chaos engineering for AI workloads
-
Canary deployments and A/B testing for models
-
Drive automation-first SRE practices using IaC and policy-as-code
4. AI System Performance Optimization
-
Optimize:
-
Inference latency and throughput
-
GPU/accelerator utilization
-
Distributed training efficiency
-
Work with engineering teams to:
-
Fine-tune model serving infrastructure
-
Implement caching, batching, and async processing
-
Drive performance benchmarking frameworks for AI workloads
5. Incident Management & Reliability Operations
-
Establish incident response frameworks tailored for AI platforms
-
Lead root cause analysis (RCA) for:
-
Model failures
-
Pipeline breakdowns
-
Infrastructure bottlenecks
-
Define and track MTTR, MTBF, availability, and reliability KPIs
-
Build runbooks, playbooks, and operational dashboards
6. Tooling & Platform Enablement
-
Implement and manage observability and SRE tooling such as:
-
Monitoring: Prometheus, Grafana, Datadog, Azure Monitor, CloudWatch
-
Logging & tracing: ELK stack, OpenTelemetry, Jaeger
-
AI observability: Langfuse, Weights & Biases, Arize, WhyLabs (preferred)
-
Develop custom telemetry pipelines for AI-specific metrics (token usage, prompt traces, response quality signals)
-
Integrate observability into CI/CD and MLOps pipelines
7. Governance & Risk Management
-
Define reliability guardrails and governance policies
-
Ensure compliance, security, and availability requirements for AI systems
-
Implement controls for:
-
Model drift and degradation detection
-
Data pipeline integrity
-
Responsible AI monitoring
8. Stakeholder Leadership & Advisory
-
Act as a trusted advisor to:
-
Platform engineering
-
Data science teams
-
Enterprise architecture and leadership
-
Translate reliability metrics into business impact (customer experience, revenue risk)
-
Drive enterprise adoption of SRE practices for AI
9. Thought Leadership & Innovation
-
Develop POVs, frameworks, and accelerators for:
-
AI SRE maturity models
-
Observability patterns for GenAI
-
Stay ahead of trends in:
-
AI reliability engineering
-
Observability tooling and standards
-
Lead internal capability building and external client workshops
Required Qualifications
Experience
-
12–15+ years of experience in:
-
Site Reliability Engineering / DevOps / Platform Engineering
-
Cloud infrastructure and distributed systems
-
4–6+ years working with AI/ML platforms, MLOps, or data-intensive systems
-
Proven experience in designing high-scale, highly reliable systems
Core Skills
-
Deep expertise in:
-
SRE principles (SLI/SLO, error budgets, incident management)
-
Observability (metrics, logs, tracing)
-
Distributed system design and failure modes
-
Strong understanding of:
-
AI/ML workloads (training, inference, pipelines)
-
LLM architectures and GenAI systems
Technical Skills
-
Cloud Platforms: Azure, AWS, GCP
-
Infrastructure:
-
Kubernetes, containers, serverless architectures
-
Observability stack:
-
OpenTelemetry, Prometheus, Grafana, ELK
-
Programming / scripting:
-
Python, Go, or similar
-
CI/CD & IaC:
-
Terraform, ARM, CloudFormation, GitOps
Leadership & Consulting Skills
-
Executive communication and stakeholder management
-
Ability to lead cross-functional, global teams
-
Strong problem-solving and analytical mindset
-
Experience in client-facing advisory and transformation programs
Preferred Qualifications
-
Certifications:
-
Kubernetes (CKA/CKAD)
-
Cloud Architect (Azure/AWS/GCP)
-
Exposure to:
-
AI observability platforms (Arize, WhyLabs, Langfuse, etc.)
-
FinOps alignment for AI workloads
-
Experience with:
-
Multi-cloud and hybrid deployment strategies
