Job Summary:
We are looking for a skilled Data Engineer to design, develop, and optimize enterprise-grade data pipelines within an on-premises, containerized data platform environment. The ideal candidate should have strong expertise in Apache Spark, Python, SQL/PL-SQL, and container orchestration technologies to support scalable data ingestion, transformation, and analytics workloads.
The role involves building and maintaining robust data solutions that enable Business Intelligence, Analytics, AI/ML initiatives, and regulatory reporting across the organization.
Key Responsibilities:
- Design, develop, and maintain scalable ETL/ELT pipelines using Apache Spark (PySpark/Scala).
- Develop data ingestion frameworks for structured, semi-structured, and unstructured data sources.
- Deploy and manage containerized data workloads within OpenShift or Kubernetes environments.
- Build and maintain Lakehouse architectures using Medallion Architecture (Bronze, Silver, Gold layers).
- Implement data quality validation, monitoring, and alerting mechanisms.
- Optimize Spark workloads for performance, scalability, and resource efficiency.
- Develop and manage workflow orchestration using Apache Airflow.
- Collaborate with Business, Analytics, DevOps, and Data Governance teams to deliver enterprise data solutions.
- Ensure adherence to data governance, compliance, and security standards.
- Participate in code reviews, troubleshooting, and performance tuning activities.
Required Skills:Technical Skills:
- Strong experience in Python programming.
- Hands-on experience with Apache Spark (PySpark preferred; Scala is an advantage).
- Expertise in SQL, PL/SQL, and relational databases such as Oracle and PostgreSQL.
- Experience with Red Hat OpenShift (OCP) or Kubernetes.
- Knowledge of Apache Airflow for workflow orchestration.
- Experience with Git and CI/CD pipelines.
- Understanding of Data Warehousing concepts and Lakehouse architecture.
- Familiarity with Delta Lake or equivalent open-source storage formats.
- Experience with Data Quality frameworks and monitoring solutions.
Preferred Skills:
- Exposure to Medallion Architecture (Bronze, Silver, Gold).
- Experience working in large-scale enterprise data environments.
- Certifications in Spark, Kubernetes/OpenShift, Python, or Data Engineering technologies will be an added advantage.
Qualifications:
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- Relevant certifications in Data Engineering technologies are preferred.
Soft Skills:
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management abilities.
- Ability to work independently as well as in cross-functional teams.
- Strong ownership mindset with a focus on quality and timely delivery.
Pay: ₹60,000.00 - ₹70,000.00 per month
Benefits:
- Flexible schedule
- Paid sick time
- Provident Fund
Ability to commute/relocate:
- Pune, Maharashtra (Pune District): Reliably commute or planning to relocate before starting work (Required)
Application Question(s):
- What is your current CTC ?
- What is your Expected CTC ?
- What is your notice period?
Experience:
- Data Engineer: 3 years (Required)
Work Location: In person
