Project Role : Custom Software Engineer
Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact.
Must have skills : Data Engineering
Good to have skills : NA
Minimum
5 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary:
As a Data Engineer, a typical day involves designing, developing, and maintaining scalable data solutions on the Azure platform. The role requires building and optimizing ETL/ELT pipelines, implementing data processing frameworks, and ensuring data availability for business and analytical needs. Collaboration with cross-functional teams, troubleshooting data workflows, and following agile development practices are key aspects of the role to deliver reliable and high-performing data solutions.
Roles & Responsibilities:
- Expected to be an SME, collaborate and manage the team to perform.
- Engage with multiple teams and contribute to key technical decisions.
- Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks, Python, and PySpark.
- Optimize Spark jobs, cluster configurations, and data processing workflows for performance and scalability.
- Develop and support data solutions leveraging Azure Storage Accounts, ADLS Gen2, and Delta Lake.
- Integrate Databricks with orchestration tools such as Apache Airflow for scheduling and monitoring data pipelines.
- Deploy and manage Databricks notebooks, jobs, workflows, and libraries across Development, QA, UAT, and Production environments.
- Troubleshoot data pipeline failures and ensure reliable data processing and delivery.
- Collaborate with stakeholders and development teams to deliver business-driven data solutions.
Professional & Technical Skills:
Must Have Skills:
- Strong hands-on experience in Azure Databricks with Python and PySpark.
- Good understanding of Apache Spark architecture, Spark configurations, performance tuning, and cluster management.
- Experience with Azure Storage Accounts, Azure Data Lake Storage (ADLS Gen2), and Delta Lake.
- Knowledge of authentication and authorization mechanisms, including Service Principals and Managed Identity.
- Experience integrating Databricks with Apache Airflow for job scheduling and monitoring.
- Understanding of Microservices architecture, REST APIs, and API integrations.
- Experience deploying Databricks notebooks, workflows, and libraries across multiple environments.
- Strong knowledge of Databricks Jobs, Workflows, and troubleshooting data pipeline failures.
- Ability to develop scalable ETL/ELT pipelines using PySpark and Python.
- Knowledge of ITSM processes and Production Support activities.
Additional Information:
The candidate should have a minimum of 8 years of experience in Data Engineering.
Experience in Azure Cloud-based Data Engineering solutions is preferred.
This position is based at our Bengaluru office.
A 15 years full-time education is required.
