Key Responsibilities Design, develop, and optimize data pipelines using Azure Databricks. Implement ETL/ELT workflows leveraging Spark (PySpark/Scala). Integrate and manage data from multiple sources such as Azure Data Lake, SQL DB, Synapse, and external APIs. Design and enforce data quality rules, validation checks, and monitoring frameworks. Work with data quality tools to profile, cleanse, and validate data. Ensure data pipelines follow best practices for performance, scalability, and cost optimization. Collaborate with data engineers, analysts, and business stakeholders to ensure data reliability. Implement CI/CD pipelines and automation for Databricks deployments. Maintain documentation for data processes, pipelines, and quality rules.
