Greetings !!!
Job Title: Data Engineer – Datastore Migration
Client: Tech Mahindra (for a leading global investment banking client)
Experience: 5+ Years
Location: Bengaluru,
Job Overview
We are seeking a skilled Data Engineer to join the Datastore Migration Factory team responsible for executing end-to-end datastore migration from an on-premises Data Lake environment to an AWS-hosted Lakehouse platform. This is a high-visibility and business-critical transformation initiative involving large-scale data migration, modernization, validation, and stakeholder engagement.
Key Responsibilities
1. Pipeline Migration
- Refactor and migrate extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment.
- Execute physical migration of datasets while ensuring data integrity and consistency.
- Collaborate with data owners and stakeholders to facilitate migration sign-off and business validation.
2. Consumption Pattern Migration
- Convert and optimize legacy SQL and Spark-based consumption patterns for compatibility with Snowflake and Apache Iceberg.
- Analyze existing usage patterns and deliver required data products.
- Partner with stakeholders to validate migrated assets and ensure business requirements are met.
- Support data reconciliation and quality assurance activities.
3. Data Validation & Reconciliation
- Perform rigorous data validation and reconciliation to ensure migrated datasets are functionally equivalent to existing production data.
- Work with reconciliation frameworks and internal data management platform teams.
- Quickly learn and adapt to new workflows, tools, and language constructs as required.
Required Qualifications
Education
- Bachelor's or Master's degree in Computer Science, Engineering, Applied Mathematics, or a related quantitative discipline.
Experience
- 3–5+ years of hands-on software development or data engineering experience.
- Strong SQL troubleshooting and scripting capabilities.
- Experience working in collaborative, team-based environments.
Programming Skills
- Strong proficiency in Python or Java.
- Experience with Software Development Life Cycle (SDLC) methodologies.
- Knowledge of CI/CD practices and Kubernetes (K8s) deployments.
Core Data Engineering Skills
Candidates should possess a strong understanding of:
- Temporal Data Modeling (e.g., SCD Type 2)
- Schema Evolution and Schema Enforcement concepts
- Data Partitioning and Clustering strategies
- Normalization vs. Denormalization principles
- Natural Keys vs. Surrogate Keys
- Data Reconciliation and Validation techniques
Technical Skills
Data Processing & Integration
- ANSI SQL
- Apache Spark
- Kafka
- FTP
Data Formats
Platforms & Technologies
- Hadoop (HDFS, Hive)
- Snowflake
- Apache Iceberg
- AWS Data Lake/Lakehouse ecosystem
- Sybase IQ
Preferred Candidate Profile
- Experience in data migration, data modernization, or cloud migration projects.
- Strong analytical and problem-solving skills.
- Ability to work with cross-functional and global teams.
- Experience engaging directly with business stakeholders and data owners.
Core Competencies
- High integrity and ethical conduct.
- Strong communication and stakeholder management skills.
- Excellent collaboration and teamwork abilities.
- Ownership mindset with strong delivery focus.
- Ability to work effectively across global teams and time zones.
- High energy, urgency, and commitment to quality.
- Intellectual curiosity and continuous learning attitude.
Pay: ₹1,800,000.00 - ₹3,000,000.00 per year
Work Location: In person
