Job Title
Big Data Engineer (EL3)
Experience Required
4+ Years
Employment Type
Full-Time
Job Summary
We are looking for an experienced Big Data Engineer to design, develop, and optimize scalable big data pipelines and data validation solutions. The ideal candidate should have strong hands-on experience in Scala, PySpark, AWS, and AWS Glue, along with expertise in data validation, profiling, and modern big data technologies. The role also involves leveraging AI-assisted development tools to improve engineering productivity and software quality.
Required Skills & Experience
- 4+ years of software engineering experience developing Big Data pipelines.
- 3+ years of hands-on programming experience using Scala and PySpark.
- 3+ years of experience building cloud-based data solutions on AWS, including AWS Glue.
- 3+ years of experience working with Big Data file formats such as Parquet.
- 1+ years of experience in data validation, data profiling, or developing rule-based data validation engines.
- 1+ years of experience using AI-powered coding assistants or code-generation tools such as GitHub Copilot, OpenAI Codex, or similar tools in daily software development.
Key Responsibilities
- Design, develop, and optimize scalable, rule-based data validation engines capable of validating data across diverse data pipelines.
- Build robust, high-performance Big Data pipelines and processing platforms using Scala, PySpark, AWS, and AWS Glue.
- Develop clean, maintainable, and well-tested code while utilizing optimized storage formats such as Parquet for efficient data storage and retrieval.
- Perform comprehensive data validation, data profiling, and quality assurance to ensure data accuracy, consistency, and reliability across multiple data sources.
- Utilize enterprise-approved AI development tools to streamline software development, automate repetitive tasks, and improve engineering productivity.
- Collaborate with cross-functional teams including Business Analysts, Data Engineers, Architects, and Product Owners to translate business requirements into scalable technical solutions.
- Evaluate emerging Big Data technologies and AI-driven engineering practices to enhance solution architecture and innovation.
- Participate in code reviews, unit testing, debugging, performance tuning, and optimization to ensure high-quality software delivery.
Technical SkillsProgramming Languages
- Scala
- PySpark
- Python (preferred)
Big Data Technologies
- Apache Spark
- PySpark
- Parquet
- Data Validation Frameworks
- Data Profiling
Cloud Platforms
Data Engineering
- ETL/ELT Development
- Data Pipelines
- Data Quality
- Rule-Based Validation Engines
- Data Processing
Development Practices
- Code Reviews
- Unit Testing
- Debugging
- Performance Optimization
- Software Development Best Practices
AI-Assisted Development
- GitHub Copilot
- OpenAI Codex
- AI Coding Assistants
Preferred Qualifications
- Experience designing scalable data validation frameworks.
- Strong understanding of distributed data processing and optimization techniques.
- Familiarity with enterprise software development lifecycle (SDLC) and Agile methodologies.
- Excellent analytical, problem-solving, and communication skills.
- Ability to work effectively in cross-functional and collaborative teams.
Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)
