Job Title: Senior Data Engineer – GenAI & Unstructured Data Pipelines
Experience: 6–8 Years
Location: Offshore (2 Positions)
Employment Type: Full-Time
Job Summary
We are seeking an experienced Senior Data Engineer to build and optimize next-generation data platforms that power Generative AI and Large Language Model (LLM) applications. The ideal candidate will have strong expertise in designing scalable data pipelines for structured, unstructured, and multi-modal data while enabling Retrieval-Augmented Generation (RAG), vector search, embeddings, and AI-powered copilots.
This role requires hands-on experience in modern data engineering, Azure cloud technologies, MLOps, and GenAI workflows to develop production-grade AI data platforms and intelligent applications.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines for structured, unstructured, and multi-modal datasets.
- Own the end-to-end machine learning lifecycle, including data ingestion, feature engineering, model training, evaluation, deployment, monitoring, retraining, and rollback.
- Build and manage production-grade ML pipelines using Azure-native services with CI/CD automation and MLOps best practices.
- Utilize Azure Machine Learning and MLflow for experiment tracking, model registry, versioning, and governed model deployment across development, testing, and production environments.
- Design and implement Generative AI solutions using Azure OpenAI, embeddings, Retrieval-Augmented Generation (RAG), and vector search technologies.
- Develop Agentic AI workflows with multi-step reasoning, tool integration, guardrails, observability, reliability, and cost optimization.
- Build scalable batch and real-time data processing pipelines using Azure Databricks, Apache Spark, and Kafka.
- Develop robust ingestion pipelines for text, documents, PDFs, logs, JSON, and other unstructured data sources.
- Design and optimize RAG pipelines including document chunking, embedding generation, indexing, and semantic retrieval.
- Implement and manage vector databases and vector search platforms such as Azure AI Search and Pinecone.
- Collaborate with data scientists, AI engineers, and application teams to deliver scalable AI-ready data platforms.
- Monitor pipeline performance, data quality, scalability, and system reliability while implementing continuous improvements.
- Follow DevOps, MLOps, and Agile best practices for secure and automated deployments.
Required Skills & Qualifications
- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related field.
- 6–8 years of experience in Data Engineering with expertise in large-scale data processing.
- Strong programming skills in Python, PySpark, and SQL.
- Hands-on experience with:
- Apache Spark
- Apache Airflow
- Apache Kafka
- Azure Databricks
- Experience building batch and streaming data pipelines.
- Strong experience processing unstructured and semi-structured data including JSON, logs, documents, PDFs, and other text-based content.
- Experience with cloud platforms, preferably Microsoft Azure.
- Exposure to Azure Machine Learning, MLflow, and MLOps practices.
- Hands-on experience with Generative AI concepts including:
- Large Language Models (LLMs)
- Retrieval-Augmented Generation (RAG)
- Embeddings
- Vector Databases
- Semantic Search
- Experience with vector search platforms such as Azure AI Search, Pinecone, or similar technologies.
- Understanding of CI/CD pipelines, automation, monitoring, and production deployment strategies.
- Strong analytical, problem-solving, and communication skills.
Preferred Qualifications
- Experience with Azure OpenAI Service and AI-powered application development.
- Knowledge of Agentic AI architectures and AI orchestration frameworks.
- Experience working with multi-modal data pipelines and AI copilots.
- Familiarity with data governance, security, and cloud architecture best practices.
- Experience working in Agile development environments.
- Azure Data Engineering, Databricks, or Azure AI certifications are a plus.
Preferred Technologies
- Python
- PySpark
- SQL
- Apache Spark
- Apache Airflow
- Apache Kafka
- Azure Databricks
- Azure Machine Learning
- MLflow
- Azure OpenAI
- Azure AI Search
- Pinecone
- Vector Databases
- Retrieval-Augmented Generation (RAG)
- Embeddings
- Large Language Models (LLMs)
- CI/CD
- Git
- Docker
- Azure DevOps
This role is ideal for data engineering professionals who are passionate about building scalable GenAI data platforms, enabling enterprise AI applications, and delivering production-ready data pipelines that power next-generation intelligent solutions.
Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)
