Experience: 6–8 Years
Employment Type: Full-time
Job Summary
We are seeking a highly skilled Senior Data Engineer – GenAI & Unstructured Data Pipelines to build and scale next-generation AI data platforms that power Large Language Model (LLM) applications and intelligent AI solutions. The ideal candidate will have strong expertise in data engineering, cloud technologies, and modern AI/ML architectures, with hands-on experience in designing pipelines for unstructured and multi-modal data.
You will play a key role in developing Retrieval-Augmented Generation (RAG) pipelines, vector search systems, AI copilots, and production-grade machine learning workflows while collaborating closely with data scientists, ML engineers, and application teams.
Key ResponsibilitiesData Engineering & Pipeline Development
- Design, develop, and maintain scalable data pipelines for structured, semi-structured, and unstructured data, including documents, PDFs, logs, JSON, and multimedia content.
- Build high-performance batch and real-time data ingestion pipelines using Apache Spark, PySpark, Kafka, and Azure Databricks.
- Optimize data processing workflows for scalability, reliability, and performance.
Machine Learning & MLOps
- Own the end-to-end machine learning lifecycle, including data ingestion, feature engineering, model training, evaluation, deployment, monitoring, retraining, and rollback.
- Develop production-grade ML pipelines with CI/CD automation and governance.
- Utilize Azure Machine Learning and MLflow for experiment tracking, model registry, and controlled deployments across development, testing, and production environments.
Generative AI & LLM Solutions
- Design and implement Generative AI solutions using Azure OpenAI Services.
- Build Retrieval-Augmented Generation (RAG) pipelines including document chunking, embeddings generation, indexing, retrieval, and response orchestration.
- Develop AI copilots and intelligent applications powered by Large Language Models (LLMs).
- Design agentic AI workflows with multi-step reasoning, tool integration, observability, guardrails, reliability, and cost optimization.
Vector Search & Knowledge Retrieval
- Design and manage vector databases and semantic search solutions.
- Implement embedding generation, similarity search, and retrieval workflows using Azure AI Search, Pinecone, or similar vector databases.
- Optimize search relevance and retrieval performance for enterprise AI applications.
Cloud & Platform Engineering
- Develop cloud-native data engineering solutions on Microsoft Azure.
- Build scalable and secure architectures leveraging Azure Databricks, Azure Machine Learning, Azure OpenAI, and Azure AI Search.
- Implement infrastructure automation and deployment best practices.
Collaboration & Best Practices
- Work closely with Data Scientists, ML Engineers, AI Architects, and Business stakeholders to deliver enterprise AI solutions.
- Follow software engineering best practices, code reviews, testing, and documentation standards.
- Continuously evaluate emerging GenAI technologies and recommend improvements to the AI platform.
Required Skills & Qualifications
- 6–8 years of experience in Data Engineering.
- Strong hands-on experience with:
- Python
- PySpark
- SQL
- Apache Spark
- Apache Airflow
- Apache Kafka
- Experience handling unstructured data including JSON, documents, PDFs, logs, and text datasets.
- Strong experience with Microsoft Azure cloud services.
- Hands-on exposure to:
- Large Language Models (LLMs)
- Retrieval-Augmented Generation (RAG)
- Embeddings
- Vector Databases
- Semantic Search
- Experience building scalable batch and streaming data pipelines.
- Good understanding of MLOps, CI/CD, and production deployment practices.
Preferred Qualifications
- Experience with Azure Machine Learning and MLflow.
- Hands-on knowledge of Azure OpenAI Services.
- Experience with Azure AI Search, Pinecone, or other vector databases.
- Familiarity with agentic AI frameworks and AI orchestration platforms.
- Knowledge of cloud-native architecture, infrastructure automation, and DevOps practices.
- Strong problem-solving, analytical, and communication skills.
Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)
