Required Skills: Data Engineering
Country: India
Role Summary:
We are looking for a skilled and experienced Data Engineer with 6+ years of expertise in designing, developing, and managing scalable data platforms and ETL pipelines.
The ideal candidate should have strong hands-on experience in AWS cloud technologies, particularly AWS Glue, Redshift, QuickSight, Python, PySpark, SQL, data integration, data migration, and large-scale data processing systems.
The candidate should be capable of developing end-to-end data engineering solutions, supporting enterprise integrations, optimizing data pipelines, and collaborating with cross-functional teams to deliver analytics and reporting solutions.
Key Roles & Responsibilities:
Data Engineering & ETL Development
- Design, develop, and maintain ETL/ELT pipelines using Python, PySpark, SQL, and AWS Glue.
- Build and support data ingestion frameworks for batch and real-time processing.
- Develop reusable and optimized data transformation solutions.
- Create workflows for structured and semi-structured datasets.
- Automate data processing and monitoring activities.
AWS Cloud & Big Data
- Develop and maintain AWS Glue jobs for data extraction, transformation, and loading.
- Design and optimize Amazon Redshift data warehouse solutions and SQL queries.
- Implement scalable cloud-based data solutions using AWS services.
- Monitor Glue jobs, Redshift clusters, and cloud infrastructure performance.
- Support AWS resource optimization and operational efficiency initiatives.
SQL & Database Management
- Develop SQL queries, stored procedures, views, and performance tuning solutions.
- Perform query optimization and database performance analysis.
- Work with enterprise databases including Redshift, SQL Server, Oracle, Snowflake, and PostgreSQL.
- Implement data validation, reconciliation, and auditing processes.
Python & PySpark Development
- Develop Python and PySpark scripts for large-scale data processing and transformation.
- Handle distributed data processing using Spark frameworks.
- Build automation scripts for data validation, reconciliation, monitoring, and migration activities.
- Troubleshoot and resolve production-level data engineering issues.
Data Migration & Integration
- Support data migration projects between enterprise systems and cloud environments.
- Perform data mapping, transformation, cleansing, and validation activities.
- Integrate multiple enterprise applications and source systems.
- Ensure data consistency and integrity across integrated systems.
Data Cleaning & Data Quality
- Develop data cleansing and standardization processes.
- Identify and resolve data quality issues, duplicates, and inconsistencies.
- Implement data quality checks across data pipelines.
- Perform root cause analysis for data discrepancies and failures.
Master Data Management (MDM)
- Support enterprise MDM processes and workflows.
- Manage customer, product, supplier, and business master data integrations.
- Implement data validation and synchronization across multiple systems.
- Collaborate with business teams to improve master data quality and governance.
Reporting & Analytics
- Develop dashboards and analytical reports using AWS QuickSight.
- Support reporting and business intelligence requirements.
- Build datasets and data models for analytics teams.
- Optimize dashboard performance and visualization capabilities.
Production Support & Troubleshooting
- Provide production support for data platform issues and integration failures.
- Monitor SLA timelines and resolve production incidents.
- Perform root cause analysis and implement preventive solutions.
- Support business users during releases and go-live activities.
Collaboration & Team Support
- Collaborate with business analysts, architects, QA teams, and stakeholders.
- Participate in architecture discussions, code reviews, and solution design activities.
- Prepare technical documentation, SOPs, and operational runbooks.
- Mentor junior engineers and support knowledge-sharing initiatives.
- Contribute to continuous improvement and best practices across the data engineering team.
Required Skills:
- 6+ years of experience in Data Engineering and ETL development.
- Strong hands-on experience with Python, PySpark, SQL, and AWS Glue.
- Experience with Amazon Redshift and cloud-based data platforms.
- Knowledge of data migration, data integration, and MDM concepts.
- Experience working with enterprise applications and large-scale integrations.
- Strong understanding of database concepts, query optimization, and performance tuning.
- Experience in AWS QuickSight or similar reporting/BI tools.
- Good problem-solving, analytical, and troubleshooting skills.
- Strong communication and collaboration abilities.
