As an Azure Databricks Engineer, you will be responsible for designing, developing, and maintaining data pipelines and solutions on the Azure Databricks platform. You will work closely with data scientists, data engineers, and business stakeholders to understand requirements, implement solutions, and ensure the reliability and performance of data infrastructure. Responsibilities:
- Design, develop, and maintain data pipelines using Azure Databricks.
- Implement data ingestion, transformation, and loading processes.
- Optimize Databricks clusters for performance and cost efficiency.
- Collaborate with data scientists and engineers to build and deploy machine learning models.
- Monitor and troubleshoot data pipelines and infrastructure.
- Implement data security and governance best practices.
- Automate data pipeline deployments and infrastructure provisioning.
- Stay up-to-date with the latest Azure Databricks features and best practices.
