Project Role : Custom Software Engineer
Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs.
Must have skills : AWS Glue
Good to have skills : Python (Programming Language), PySpark
Minimum
7.5 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary:
As a Custom Software Engineer, you will engage in the development of custom software solutions that are designed to meet specific business needs. Your typical day will involve collaborating with cross-functional teams to design, code, and enhance various components across systems or applications. You will utilize modern frameworks and agile practices to ensure the delivery of scalable and high-performing solutions, while also addressing any challenges that arise during the development process. Your role will require a proactive approach to problem-solving and a commitment to continuous improvement in software development practices.
Roles & Responsibilities:
Build & Maintain Data Pipelines: Develop scalable, reliable ETL/ELT workflows using Python and AWS Glue (Jobs/Workflows), Lambda, and Step Functions where applicable.
Data Lake Engineering: Design and optimize S3-based data lakes (partitioning, lifecycle policies, storage classes) with robust metadata strategies using AWS Glue Data Catalog.
Security & Governance: Implement AWS IAM policies/roles, fine-grained access controls for S3, KMS-based encryption, and audit-ready logging (CloudTrail/CloudWatch).
Automation & CI/CD: Familiarity with Git, code reviews, and automated testing for data code. Package and deploy jobs/functions, maintain unit/integration tests, and set up CI/CD pipelines.
Serverless & Event-Driven: Build event-driven data processing with S3 events, Lambda triggers optimize for cost, concurrency, and resilience.
Data Quality & Observability: Implement validation, schema enforcement (e.g., Glue schemas), logging/monitoring, and alerting for pipelines and jobs.
Collaboration: Work closely with data platform, analytics, and governance teams to ensure reliability, scalability, and compliance.
(If Databricks): Develop notebooks/jobs, Delta Lake optimization (Z-ordering, OPTIMIZE/VACUUM), cluster/pool usage, and job orchestration.
Professional & Technical Skills:
- Must Have Skills: Proficiency in AWS Glue, Python (Programming Language), PySpark.
- Strong understanding of data integration and ETL processes.
- Experience with cloud computing services and architecture.
- Familiarity with agile methodologies and software development life cycle.
Additional Information:
- The candidate should have minimum 7.5 years of experience in AWS Glue, Python (Programming Language), PySpark.
- This position is based at our Bengaluru office.
- A 15 years full time education is required.
