Location - Pune(locals can apply)Exp - 10 yrsMode- Hybrid Key Skills : Data Engineering, Data Warehousing, or Big Data platform development.,Azure Data Services,ETL/ELT pipeline development,Python.Data Pipeline & ETL/ELT Engineering Design, build, and optimize scalable ETL/ELT pipelines using Azure Synapse, Azure Data Factory, and Apache Spark. Implement incremental, batch, micro-batch, and real-time data processing using ADLS and Delta Lake. Work with Medallion architecture (Bronze → Silver → Gold) for data lake optimization. Data Governance, Quality & Security Implement data governance using Microsoft Purview, Data Catalog, RBAC, and secure access controls. Define and enforce data quality frameworks using Great Expectations or equivalent tools. Azure Platform Engineering & Integration Build and orchestrate workflows using Azure Logic Apps, Azure Functions, REST API integrations, and event-driven services (Event Hub/Service Bus). Python & Spark Development Develop Python modules and notebooks for automation, transformations, and ML integrations. Write optimized PySpark jobs for Synapse Spark or Databricks. Observability & Performance Optimization Monitor and optimize pipelines using Azure Monitor, Log Analytics, and Application Insights. Tune SQL queries and Spark jobs for improved performance.Collaboration, Agile Delivery & Documentation Collaborate with cross-functional teams including data architects, analysts, and business stakeholders. Document data flows, governance policies, and architecture diagrams. Implement CI/CD using Azure DevOps. AI-Driven Data Migration Skills & Strategies Use AI-assisted data profiling and discovery to assess legacy data platforms and migration complexity. Apply ML-based data quality and anomaly detection to identify inconsistencies, duplicates, and loss risks during migration. Leverage AI-assisted schema and SQL conversion techniques to modernize legacy databases into Azure SQL and Synapse SQL.AI, ML & GenAI Skills Design and build data pipelines to support Machine Learning (ML) model training, validation, and inference using Azure ML and Synapse. Enable MLOps workflows including dataset versioning, feature engineering, experiment tracking, and model monitoring. Support Generative AI (GenAI) and LLM-based solutions using Azure OpenAI and Retrieval-Augmented Generation (RAG) architectures. Build and manage embedding pipelines, vectorized data, and metadata enrichment for AI-driven search and copilots.
Pay: ₹1,809,356.57 - ₹3,266,829.63 per year
Work Location: In person
