Data Engineering & Pipeline Development: Design and implement end-to-end data pipelines (batch and streaming) using Azure Data Factory / Synapse Pipelines and Databricks.
ETL/ELT Development: Develop scalable ETL/ELT workflows using PySpark , SQL, and Databricks notebooks to process large-scale datasets efficiently.
Data Ingestion: Build and optimize data ingestion frameworks for structured and unstructured data from multiple enterprise and external sources.
Databricks & Lakehouse Architecture: Lead the design and implementation of Databricks-based lakehouse architecture, including Delta Lake and medallion (bronze, silver, gold) data layers.
Spark Optimization: Optimize Spark jobs for performance, scalability, and cost efficiency, including partitioning, caching, and cluster tuning.
Data Governance & Processing: Implement data versioning, incremental processing, and ensure high data quality, consistency, and governance standards.
Azure Data Platform: Utilize Azure services such as Data Factory, Data Lake Storage, Synapse Analytics, and Key Vault to build secure, scalable data solutions.
Application & Systems Integration: Design, build, and maintain integration solutions using Azure Functions, Logic Apps, and Service Bus / Event Grid to connect line-of-business applications, SaaS platforms, and external services with the data platform.
API Development & Management: Develop and expose secure, well-documented REST APIs and event-driven endpoints, handling publishing, versioning, throttling, and authentication through Azure API Management.
Event-Driven & Real-Time Messaging: Implement reliable, decoupled event-driven and near-real-time integration patterns using Event Hub, Event Grid, and Service Bus for resilient message and data exchange across systems.
Performance & Optimization: Monitor and continuously optimize data workflows for reliability, performance, and cost efficiency
DevOps & Governance: Manage code repositories, CI/CD pipelines, and deployments using Azure DevOps, ensuring high-quality documentation and best practices.
Collaboration & Stakeholder Engagement: Work closely with business stakeholders and technical teams to translate requirements into scalable data solutions and provide technical leadership.
