- Key Responsibilities
- Lead the design and development of scalable data pipelines and ETL ELT workflows using Databricks on Azure
- Architect and implement data lakehouse solutions optimizing storage compute and query performance for large datasets
- Develop optimize and maintain Databricks notebooks jobs and workflows for batch and near real time data processing
- Define and enforce coding standards best practices and reusable frameworks for Databricks based data engineering
- Collaborate with data architects analysts and business teams to understand requirements and translate them into technical designs
- Implement robust data quality checks validation rules and monitoring to ensure reliability and trust in data products
- Optimize Spark jobs for performance and cost efficiency including cluster sizing caching strategies and partitioning
- Ensure security governance and compliance of data solutions including access control and data lineage within Databricks and Azure
- Troubleshoot complex data issues perform root cause analysis and drive long term fixes and improvements
- Mentor junior engineers provide technical guidance conduct code reviews and support knowledge sharing within the team
- Contribute to roadmap planning effort estimation and delivery management for data engineering initiatives
- Technology Cloud Platform Databricks AI Services Technology Data Engineering Databricks
- Knowledge of more than one technology
- Basics of Architecture and Design fundamentals
- Knowledge of Testing tools
- Knowledge of agile methodologies
- Understanding of Project life cycle activities on development and maintenance projects
- Understanding of one or more Estimation methodologies Knowledge of Quality processes
- Basics of business domain to understand the business requirements
- Analytical abilities Strong Technical Skills Good communication skills
- Good understanding of the technology and domain
- Ability to demonstrate a sound understanding of software quality assurance principles SOLID design principles and modelling methods
- Awareness of latest technologies and trends
- Excellent problem solving analytical and debugging skills
Technology->Data Engineering->Databricks,Technology->AI-AI Engineering->Databricks AI Engineering Services
