Job Title: Big Data Engineer (EL3)
Experience: 4+ Years
Location: Onsite
Employment Type: Full-Time
Job Summary
We are seeking an experienced Big Data Engineer to design, develop, and optimize scalable big data solutions. The ideal candidate should have strong expertise in Scala, PySpark, AWS Glue, and data validation frameworks, along with experience in building high-performance data pipelines and working with large-scale datasets.
Key Responsibilities
- Design, develop, and optimize scalable, rule-based data validation engines capable of validating data across multiple data pipelines.
- Build and maintain high-performance big data pipelines using Scala, PySpark, and AWS Glue.
- Write clean, efficient, and well-tested code while utilizing optimized file formats such as Parquet for efficient data storage and retrieval.
- Perform data validation, profiling, and quality checks to ensure data consistency and accuracy across various data systems.
- Leverage enterprise-approved AI coding tools (e.g., GitHub Copilot, Codex) to improve software development productivity and automate routine tasks.
- Collaborate with cross-functional teams to translate business requirements into scalable technical solutions.
- Research and evaluate emerging AI technologies and big data trends to improve solution architecture and innovation.
- Participate in code reviews, testing, debugging, and performance optimization to ensure high-quality software delivery.
Required Skills & Qualifications
- 4+ years of software engineering experience in developing big data pipelines.
- 3+ years of hands-on experience with Scala and PySpark.
- 3+ years of experience working with AWS, including AWS Glue.
- 3+ years of experience working with big data file formats such as Parquet.
- 1+ year of experience in data validation, data profiling, or building rule-based data engines.
- 1+ year of experience using AI-assisted coding tools such as GitHub Copilot, Codex, or similar AI code-generation tools.
- Strong understanding of distributed data processing and scalable data architectures.
- Excellent problem-solving, debugging, and analytical skills.
- Strong communication and teamwork abilities.
Work Location: Hybrid remote in Noida, Uttar Pradesh
