Project Role : Custom Software Engineer
Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs.
Must have skills : AWS Glue
Good to have skills : NA
Minimum 7.5 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary
As a Senior AWS Glue Data Engineer / AWS Glue Architect, you will design, develop, and implement enterprise-scale ETL and data integration solutions using AWS Glue and AWS cloud-native data services. The role requires strong hands-on expertise in AWS Glue ETL development, PySpark programming, AWS Lambda, IAM security, VPC networking, and serverless data engineering architectures.
The engineer will build scalable data pipelines, metadata-driven frameworks, and cloud-native integration solutions using AWS Glue, Glue Data Catalog, Lambda, S3, Athena, Redshift, Step Functions, EventBridge, and other AWS services. The role involves data ingestion, transformation, performance optimization, security implementation, monitoring, and production support following AWS best practices.
Roles & Responsibilities
1. Design, develop, and maintain enterprise-scale ETL pipelines using AWS Glue.
2. Develop and optimize AWS Glue Spark jobs using PySpark for large-scale data processing.
3. Create and manage AWS Glue components including Jobs, Crawlers, Data Catalog, Workflows, Triggers, Connections, and Job Bookmarks.
4. Build metadata-driven ETL frameworks using AWS Glue Data Catalog and configuration-based processing approaches.
5. Develop data ingestion and transformation pipelines from multiple sources including databases, APIs, files, and cloud platforms.
6. Integrate AWS Glue with AWS services including Lambda, S3, Athena, Redshift, Step Functions, EventBridge, and SQS.
7. Develop AWS Lambda functions for data processing, automation, event-driven workflows, and Glue job orchestration.
8. Configure and manage AWS IAM Roles, Policies, Permissions, and Security Controls for Glue jobs and AWS services.
9. Design and configure AWS VPC components including VPC endpoints, subnets, security groups, route tables, and network connectivity for AWS Glue workloads.
10. Configure AWS Glue connections for databases using JDBC connectivity.
11. Implement secure data integration solutions using JDBC, REST APIs, SFTP, and event-driven architectures.
12. Perform AWS Glue job tuning using Spark optimization techniques, partitioning, compression, caching, and performance best practices.
13. Implement data quality checks, validation frameworks, error handling, and reconciliation processes.
14. Monitor and troubleshoot AWS Glue jobs using Amazon CloudWatch, CloudWatch Logs, and AWS monitoring services.
15. Implement CI/CD pipelines for AWS Glue deployments using Git, Jenkins, GitHub Actions, AWS CodePipeline, or AWS CodeBuild.
16. Develop Infrastructure as Code (IaC) solutions using AWS CloudFormation or AWS CDK.
17. Support production incidents, troubleshooting, performance issues, and operational improvements.
18. Collaborate with data architects, data engineers, application teams, and business stakeholders to deliver AWS data solutions.
19. Mentor team members on AWS Glue development standards, PySpark optimization, and AWS data engineering best practices.
Professional & Technical Skills
1. Strong hands-on experience with AWS Glue for ETL development, data integration, and data pipeline implementation.
2. Strong programming skills in Python, PySpark, and SQL for data transformation and processing.
3. Experience with AWS services including Lambda, S3, Athena, Redshift, Step Functions, and EventBridge.
4. Strong knowledge of AWS IAM Roles, Policies, Security Configuration, and VPC setup for secure data solutions.
5. Experience designing and optimizing AWS Glue ETL pipelines, Spark jobs, and data processing workflows.
6. Good understanding of Data Lake architecture, AWS Data Catalog, metadata management, and data governance.
7. Experience with CI/CD, Git, Infrastructure as Code (CloudFormation/Terraform), Agile, and DevOps practices.
Additional Information
1. The candidate should have a minimum of 10 years of experience in AWS Glue.
2. This position is based at the Hyderabad office.
3. A minimum of 15 years of full-time education is required.
15 years full time education
