We are seeking a highly skilled and experienced Databricks Engineer to design, develop, and maintain robust and scalable data pipelines and solutions on the Databricks platform. The ideal candidate will have a deep understanding of big data technologies, cloud platforms, and data engineering best practices. You will be instrumental in transforming raw data into actionable insights, supporting data scientists, analysts, and business users across the organization.
- Design, develop, and implement scalable ETL/ELT data pipelines using Databricks, Apache Spark, Python, and SQL.
- Build and optimize data models in Delta Lake for efficient storage and retrieval of large datasets.
- Collaborate with data architects, data scientists, and business stakeholders to understand data requirements and translate them into technical solutions.
- Ensure data quality, integrity, and security across all data pipelines and data assets.
- Monitor, troubleshoot, and optimize the performance of Databricks jobs and clusters.
- Implement CI/CD practices for data pipelines and infrastructure on the Databricks platform.
- Manage and administer Databricks workspaces, clusters, and notebooks, ensuring optimal resource utilization.
- Develop and maintain documentation for data pipelines, data models, and operational procedures.
- Stay current with the latest Databricks features, big data technologies, and cloud services to recommend and implement improvements.
- Participate in code reviews and uphold best practices for data engineering and software development.
