Required Skills:
- PySpark / Spark
-
dbt
-
Redshift
-
Apache Airflow
-
Lakehouse / Iceberg
-
Python
-
CI/CD & GitHub
-
Data Modeling
Nice to Have:
- Terraform
-
Data Governance
Job Title: Data Engineering Tech Lead About the Role: We are seeking a Data Engineering Tech Lead to design and deliver scalable data solutions. You will lead a team of engineers to build and optimize data pipelines, implement Lakehouse architecture, and ensure high-quality, reliable data for analytics and reporting. Key Responsibilities: Lead the design and development of data pipelines and workflows using AWS Glue, Spark, and PySpark. Implement Lakehouse architecture leveraging Apache Iceberg for efficient data storage and querying.-Redshift Develop and maintain transformation logic using Python and dbt. Manage orchestration and scheduling using Apache Airflow or AWS-native services. Optimize data models and queries for Redshift and other analytical databases. Ensure best practices in CI/CD, version control (GitHub), and automated testing. Collaborate with stakeholders to translate business requirements into technical solutions. Mentor and guide the data engineering team on coding standards and architecture decisions. Required Skills & Experience: Strong experience with AWS Glue, Spark, PySpark, and Python. Hands-on expertise in dbt for data transformations and testing. Knowledge of Lakehouse architecture and Apache Iceberg. Experience with Apache Airflow for workflow orchestration. Proficiency in Redshift and SQL performance tuning. Familiarity with GitHub workflows and CI/CD pipelines (GitHub Actions, Jenkins, or AWS CodePipeline). Solid understanding of data modeling, partitioning, and performance optimization. Nice to Have: Experience with Terraform or other Infrastructure-as-Code tools. Familiarity with data governance and lineage tools. Exposure to other big data frameworks (Delta Lake, Hudi).
