Job Title: AI Data Engineer
Experience:5–8 Years
Employment Type: Full-Time
Position Summary
We are seeking an experienced **AI Data Engineer** to build and maintain scalable data pipelines that power AI/ML and Generative AI solutions. The ideal candidate will have strong expertise in **Apache Spark (Python/PySpark), SQL, Azure Cloud, Airflow, Kafka, and processing unstructured data**. This role involves designing modern data architectures, building ETL/ELT pipelines, managing large-scale structured and unstructured datasets, and enabling AI/ML teams with high-quality, reliable data.
Key Responsibilities
* Design, develop, and optimize scalable data pipelines for AI/ML and Generative AI applications.
* Build batch and real-time data processing solutions using Apache Spark (PySpark) and Kafka.
* Develop ETL/ELT workflows to ingest, transform, and process data from multiple sources.
* Process and manage structured, semi-structured, and unstructured data, including JSON, logs, documents, PDFs, and text data.
* Develop workflows using Apache Airflow for scheduling, orchestration, and monitoring.
* Design and optimize data models for analytics and AI workloads.
* Work with Azure cloud services to build secure, scalable, and high-performance data platforms.
* Collaborate with Data Scientists, ML Engineers, and AI teams to prepare feature-ready datasets.
* Optimize Spark jobs for performance, scalability, and cost efficiency.
* Implement data quality checks, validation, and monitoring processes.
* Ensure data governance, security, and compliance across data platforms.
* Troubleshoot production issues and optimize existing data pipelines.
* Maintain technical documentation, data lineage, and architecture diagrams.
* Stay current with emerging technologies in AI data engineering and big data ecosystems.
Required Technical Skills
* Strong expertise in Python and PySpark.
* Advanced SQL development and query optimization.
* Apache Spark for distributed data processing.
* Azure Cloud services (Azure Data Lake Storage, Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure Blob Storage).
* Apache Airflow for workflow orchestration.
* Apache Kafka for real-time data streaming.
* Experience handling unstructured and semi-structured data (JSON, XML, logs, PDFs, text, documents).
* Data ingestion, transformation, and pipeline development.
* Experience with Delta Lake and Parquet file formats.
* Git and version control.
* CI/CD pipelines for data engineering workflows.
* Linux/Unix command-line experience.
### Required Qualifications
* Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.
* 5–8 years of experience in Data Engineering or Big Data development.
* Hands-on experience building enterprise-scale data pipelines.
* Experience working with cloud-native data platforms, preferably Microsoft Azure.
* Strong understanding of distributed computing and big data architecture.
* Experience working in Agile/Scrum environments.
Preferred Skills
* Experience with Azure Databricks.
* Knowledge of Generative AI, LLMs, and Retrieval-Augmented Generation (RAG) data pipelines.
* Experience with Vector Databases such as Pinecone, Milvus, Azure AI Search, or FAISS.
* Familiarity with Azure Machine Learning and MLflow.
* Experience with data lakehouse architecture.
* Knowledge of data cataloging, metadata management, and data governance.
* Experience with Docker and Kubernetes.
* Exposure to Infrastructure as Code (Terraform or ARM/Bicep).
* Experience with monitoring tools such as Azure Monitor, Prometheus, or Grafana.
Preferred Certifications
* Microsoft Certified: Azure Data Engineer Associate (DP-203)
* Microsoft Azure Fundamentals (AZ-900)
* Databricks Certified Data Engineer Associate or Professional
* Apache Spark Certification (Preferred)
Key Competencies
* AI Data Engineering
* Apache Spark (PySpark)
* SQL Optimization
* Azure Cloud Data Services
* Workflow Orchestration (Airflow)
* Real-Time Data Streaming (Kafka)
* Unstructured Data Processing
* ETL/ELT Pipeline Development
* Data Modeling
* Performance Optimization
* Problem Solving
* Collaboration and Communication
Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)
