Job Description:
The Senior Data Engineer will be responsible for leading the support, optimization, and troubleshooting of enterprise data engineering pipelines and analytics platforms. This role requires strong hands-on experience with Snowflake, Databricks, AWS Glue, SQL, Python/PySpark, ETL/ELT development, data modeling, and production support. The resource should be able to work independently on complex data issues, guide other engineers, and coordinate with client SMEs for L2/L3 resolution and production changes.
- Enhance and support ETL/ELT pipelines using Snowflake, Databricks, AWS Glue, SQL, Python, and PySpark.
- Perform advanced troubleshooting for data pipeline failures, data refresh issues, performance problems, and data quality defects.
- Optimize Snowflake queries, Databricks jobs/notebooks, AWS Glue jobs, and data processing workflows for reliability, performance, and cost efficiency.
- Support data modeling, schema design, ingestion patterns, transformation logic, and reusable data engineering frameworks.
- Guide intermediate and junior data engineers on development standards, code review, testing, release support, and support documentation.
- Coordinate with Customer SMEs, Database Operations, application owners, and security teams for L2/L3 issue resolution and production approvals.
Contribute to future migration planning from AWS Glue-based pipelines to Databricks, including assessment, design input, and migration readiness.
-
Core Skills: Snowflake, Databricks, AWS Glue, SQL, Python, PySpark, ETL/ELT development, data warehousing, data modeling, data quality validation, performance tuning, job scheduling, production support, incident triage, and L2/L3 troubleshooting.
Good-to-Have Skills: ServiceNow, xMatters or other alerting tools, CloudWatch or equivalent monitoring tools, AWS S3, IAM, ODI, Oracle, PL/SQL, SFTP, API-based ingestion, DevOps/CI-CD, documentation, release management, and AWS Glue to Databricks migration readiness.
