Job Summary
Sr Developer role in a global organization focusing on data engineering solutions for life and annuities insurance using Databricks SQL Databricks Workflows and PySpark. The role involves building scalable data pipelines optimizing analytical workloads and ensuring high quality data products in a hybrid work model with day shift and no travel.
Responsibilities
Design robust data pipelines in Databricks using Databricks SQL and PySpark to transform complex life and annuities insurance data into reliable analytics ready datasets
Develop reusable PySpark modules that standardize ingestion cleansing and enrichment of policy claims and actuarial data while meeting performance expectations and coding standards
Configure and manage Databricks Workflows to orchestrate end to end batch and near real time data processing jobs supporting reporting and analytics needs for insurance stakeholders
Optimize Databricks SQL queries and PySpark jobs by tuning partitions caching strategies and cluster configurations to achieve efficient resource usage and faster insights
Implement data quality checks validation rules and reconciliation processes that improve the accuracy and completeness of life and annuities insurance datasets used across business functions
Collaborate with business analysts and actuaries to translate insurance product rules policy events and financial calculations into clear and maintainable data engineering logic
Document technical designs job dependencies transformation logic and operational procedures to ensure transparency maintainability and smooth knowledge transfer within the team
Partner with testing and operations teams to define test cases support defect resolution and enable stable deployment of Databricks solutions into production environments
Monitor production workflows in Databricks investigate failures or performance regressions and implement corrective actions that protect data reliability and service commitments
Apply secure coding practices implement access controls and follow compliance guidelines relevant to financial and insurance data handling across all data engineering activities
Contribute to continuous improvement by identifying automation opportunities refactoring legacy code and adopting emerging Databricks features that deliver better outcomes for the company
Support business reporting and advanced analytics initiatives by exposing curated data models reusable views and standardized metrics for life and annuities insurance use cases
Work effectively in a hybrid environment by coordinating with on site and remote team members ensuring timely communication and consistent progress on shared deliverables
Qualifications
Possess professional experience between six and eight years in data engineering or related roles with a strong focus on large scale analytics solutions
Demonstrate deep proficiency in Databricks SQL including development of complex queries views and performance optimized data models for analytical consumption
Show hands on expertise in PySpark programming including data frame operations user defined functions and performance tuning techniques on distributed datasets
Exhibit practical experience configuring and managing Databricks Workflows including job scheduling dependency handling and monitoring of production pipelines
Bring solid domain knowledge in life and annuities insurance including familiarity with policies riders premiums benefits and related data structures
Display strong understanding of data warehousing concepts dimensional modeling and best practices for building curated layers supporting reporting and analytics
Apply sound knowledge of software engineering practices such as version control code review and modular design to maintain high quality data engineering solutions
Communicate effectively with cross functional partners by explaining technical designs in business friendly language and incorporating feedback into implementation plans
Adhere to organizational standards for security compliance and documentation while contributing ideas that enhance efficiency and long term maintainability of solutions
Certifications Required
Preferred certifications include Databricks Data Engineer Associate or Professional and cloud platform certifications relevant to data engineering
