Roles and Responsibilities :
- Design, develop, test, deploy and maintain large-scale data pipelines using Airflow to extract insights from various sources such as Kafka and ETL processes.
- Collaborate with cross-functional teams to identify business requirements and design scalable solutions for data processing and storage on AWS Glue.
- Develop high-quality code in Python using PySpark, Spark SQL, and Kubernetes to ensure efficient data processing and deployment.
- Troubleshoot complex issues related to data quality, performance optimization, and system reliability.
Job Requirements :
- 4-7 years of experience in Data Engineering with expertise in Airflow, AWS Glue, ETL/Kafka/Data Bricks/Spark.
- Strong understanding of big data technologies including Hadoop ecosystem (HDFS) and NoSQL databases like Apache Cassandra or MongoDB.
- Proficiency in writing efficient code in Python using PySpark/Scala/Kotlin programming languages.
Pay: ₹800,000.00 - ₹1,600,000.00 per year
Benefits:
- Health insurance
- Leave encashment
- Life insurance
- Provident Fund
Work Location: Hybrid remote in Bengaluru, Karnataka
