Job Summary
We are seeking an experienced Big Data Engineer (EL3) to design, develop, and optimize scalable big data solutions and data validation platforms. The ideal candidate will have strong expertise in Scala, PySpark, AWS, AWS Glue, and Parquet, with hands-on experience building high-performance data pipelines. This role involves developing rule-based data validation engines, ensuring data quality, and leveraging AI-assisted development tools to improve productivity and innovation.
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related field.
- 4+ years of software engineering experience developing big data pipelines.
- 3+ years of hands-on programming experience with Scala and PySpark.
- 3+ years of experience building solutions on AWS, including AWS Glue.
- 3+ years of experience working with big data file formats, especially Parquet.
- 1+ year of experience in data validation, data profiling, or developing rule-based data engines.
- 1+ year of experience using AI coding assistants or code-generation tools (e.g., GitHub Copilot, Codex) as part of software development.
Key Responsibilities
- Design, build, and optimize a highly scalable, rule-based data validation engine capable of validating data across multiple data pipelines.
- Develop robust, high-performance big data pipelines using Scala, PySpark, and AWS Glue.
- Write clean, efficient, and well-tested code utilizing optimized data formats such as Parquet for high-performance storage and retrieval.
- Perform data validation and profiling to ensure data quality, consistency, and integrity across diverse data systems.
- Utilize enterprise-approved AI development tools to streamline software development, automate repetitive tasks, and improve engineering productivity.
- Collaborate with cross-functional teams to translate business requirements into scalable technical solutions.
- Evaluate emerging AI technologies and big data trends to enhance data validation frameworks and solution architecture.
- Participate in code reviews, testing, debugging, performance tuning, and profiling to ensure high-quality, maintainable code.
Required Technical Skills
- Scala
- PySpark
- AWS Cloud
- AWS Glue
- Big Data Pipelines
- Parquet
- Data Validation
- Data Profiling
- Rule-Based Data Engines
- Software Engineering
- Code Reviews
- Debugging & Performance Optimization
- GitHub Copilot / AI Coding Assistants
- Cross-functional Collaboration
Preferred Skills
- Experience designing scalable distributed data processing systems.
- Strong analytical and problem-solving abilities.
- Knowledge of software development best practices and performance optimization.
- Familiarity with AI-assisted software development tools and workflows.
- Excellent communication and teamwork skills.
Experience
- 4+ years of experience in Big Data Engineering or Software Engineering.
- 3+ years of experience with Scala, PySpark, AWS, and AWS Glue.
- 1+ year of experience in data validation, profiling, or rule-based data engines.
Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)
