Job Title: Senior Data Engineer – GenAI Data Platforms
Job Summary
We are seeking a highly skilled Senior Data Engineer to architect and build next-generation Generative AI (GenAI) data platforms that power Large Language Model (LLM)-based applications, AI copilots, and Retrieval-Augmented Generation (RAG) systems. The ideal candidate will have extensive experience in modern data engineering, distributed data processing, Azure cloud technologies, and production-grade machine learning pipelines.
This role requires hands-on expertise in designing scalable data platforms capable of processing structured, unstructured, and multi-modal datasets while enabling advanced AI capabilities such as vector search, embeddings, semantic retrieval, and intelligent agent workflows.
Key ResponsibilitiesGenAI Data Platform Development
- Design, build, and maintain scalable GenAI-ready data platforms for enterprise AI applications.
- Develop robust pipelines for structured, semi-structured, unstructured, and multi-modal data.
- Build production-ready Retrieval-Augmented Generation (RAG) pipelines, including document ingestion, chunking, embedding generation, indexing, retrieval, and prompt orchestration.
- Develop AI data services that support LLM-based applications, enterprise search, and AI copilots.
- Optimize vector indexing and semantic search performance for low-latency AI applications.
Machine Learning Platform Engineering
- Own the end-to-end Machine Learning lifecycle, including:
- Data ingestion
- Data preprocessing
- Feature engineering
- Model training
- Model evaluation
- Deployment
- Monitoring
- Retraining
- Rollback strategies
- Build production-grade ML pipelines using Azure-native services with automated CI/CD.
- Implement experiment tracking, model versioning, and model governance using Azure Machine Learning and MLflow.
Data Engineering
- Design and implement scalable batch and real-time data pipelines using Apache Spark, PySpark, and Azure Databricks.
- Build streaming architectures using Apache Kafka and Spark Structured Streaming.
- Process large-scale datasets including:
- JSON
- XML
- PDFs
- Documents
- Images
- Audio
- Application logs
- Multi-modal enterprise data
- Develop high-performance ETL/ELT pipelines optimized for scalability and reliability.
- Ensure data quality, lineage, governance, and monitoring across all pipelines.
Generative AI & Agentic AI
- Design and deploy enterprise-grade GenAI applications using Azure OpenAI Service.
- Develop embedding pipelines using Azure OpenAI embedding models and other embedding frameworks.
- Build intelligent Agentic AI workflows with:
- Multi-step reasoning
- Tool calling
- Function orchestration
- Memory management
- Guardrails
- Observability
- Reliability engineering
- Cost optimization
- Implement prompt engineering and prompt management strategies.
Vector Databases & Search
- Design and manage enterprise vector search systems using:
- Azure AI Search
- Pinecone
- FAISS
- ChromaDB
- Weaviate (Preferred)
- Optimize semantic retrieval performance.
- Develop hybrid search architectures combining keyword and vector search.
Cloud & DevOps
- Build cloud-native architectures on Microsoft Azure.
- Implement Infrastructure as Code (IaC) using Terraform or Bicep (preferred).
- Develop CI/CD pipelines using Azure DevOps or GitHub Actions.
- Automate deployment, monitoring, rollback, and infrastructure provisioning.
- Ensure platform security, scalability, reliability, and cost optimization.
Required Technical SkillsProgramming Languages
- Python
- PySpark
- SQL
- Scala (Preferred)
Big Data Technologies
- Apache Spark
- Spark Structured Streaming
- Apache Kafka
- Apache Airflow
- Delta Lake
- Azure Databricks
Cloud Technologies (Azure Preferred)
- Azure Data Factory
- Azure Databricks
- Azure Machine Learning
- Azure OpenAI Service
- Azure AI Search
- Azure Blob Storage
- Azure Data Lake Storage Gen2
- Azure Event Hubs
- Azure Key Vault
- Azure Monitor
- Azure Functions
Machine Learning & MLOps
- Azure Machine Learning
- MLflow
- Model Registry
- Experiment Tracking
- CI/CD for ML
- Model Monitoring
- Feature Engineering
- Model Deployment
- Feature Stores
Generative AI
- Large Language Models (LLMs)
- Retrieval-Augmented Generation (RAG)
- Embeddings
- Prompt Engineering
- Semantic Search
- AI Copilots
- Agentic AI
- Function Calling
- LangChain
- LangGraph (Preferred)
- LlamaIndex
- Vector Databases
Vector Search Technologies
- Azure AI Search
- Pinecone
- FAISS
- ChromaDB
- Milvus (Preferred)
- Weaviate (Preferred)
Data Engineering
- ETL / ELT
- Data Lake Architecture
- Data Warehousing
- Data Governance
- Data Quality
- Metadata Management
- Batch Processing
- Streaming Pipelines
DevOps & Version Control
- Git
- Azure DevOps
- GitHub Actions
- Docker
- Kubernetes (Preferred)
- Terraform (Preferred)
Required Qualifications
- Bachelor's degree in Computer Science, Information Technology, Data Engineering, Software Engineering, or a related field.
- Master's degree in Data Science, Artificial Intelligence, or Computer Science is preferred.
- Microsoft Azure Data Engineer Associate or Azure AI Engineer certification is an advantage.
Experience
- 6–8 years of experience in Data Engineering.
- Strong expertise in Python, PySpark, SQL, Apache Spark, Kafka, and Airflow.
- Hands-on experience with Azure cloud services.
- Experience building scalable enterprise data platforms.
- Practical experience working with unstructured and multi-modal data (JSON, logs, PDFs, images, audio, and documents).
- Proven experience developing RAG pipelines, vector search systems, and embedding workflows.
- Exposure to production-grade LLM and Generative AI solutions.
- Experience with MLOps, ML lifecycle management, and Azure Machine Learning.
Preferred Skills
- Experience with LangChain, LangGraph, or LlamaIndex.
- Familiarity with Hugging Face Transformers and open-source LLMs.
- Knowledge of Kubernetes-based AI deployments.
- Experience implementing observability for AI systems.
- Understanding of Responsible AI principles, governance, and AI security.
- Exposure to distributed systems and microservices architecture.
Key Competencies
- Strong software engineering fundamentals.
- Advanced problem-solving and analytical thinking.
- Ability to design highly scalable distributed systems.
- Excellent communication and stakeholder management skills.
- Strong understanding of cloud architecture and AI infrastructure.
- Passion for innovation in Artificial Intelligence and Generative AI.
Key Performance Indicators (KPIs)
- Reliability and scalability of data pipelines.
- Pipeline execution success rate and latency.
- Data quality and governance compliance.
- Performance of RAG and vector search systems.
- ML deployment frequency and model reliability.
- Infrastructure uptime and cost optimization.
- AI application response accuracy and retrieval relevance.
- Reduction in operational overhead through automation.
Why Join Us?
- Build cutting-edge GenAI platforms powering enterprise AI products.
- Work with Azure OpenAI, RAG, vector databases, and Agentic AI technologies.
- Collaborate with experienced AI engineers, data scientists, and cloud architects.
- Drive innovation in next-generation AI applications.
- Competitive compensation, continuous learning opportunities, and career growth in AI and cloud engineering.
Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)
