Project Role : Custom Software Engineer
Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs.
Must have skills : PySpark
Good to have skills : NA
Minimum
3 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary:
As a Custom Software Engineer, a typical day involves creating tailored software solutions by designing, coding, and improving various components within systems or applications. The role requires working with contemporary frameworks and following agile methodologies to ensure the delivery of scalable and efficient solutions that meet unique business requirements. Collaboration with team members and adapting to evolving project needs are integral parts of the daily workflow, fostering innovation and continuous improvement in software development processes.
Roles & Responsibilities:
- Strong experience in designing, developing, and maintaining ETL pipelines for large-scale data processing.
- Hands-on experience with data extraction, transformation, and loading from various structured/unstructured sources.
- Experience with data quality checks, data validation, and data lineage tracking.
- Knowledge of incremental loads, CDC (Change Data Capture), and data partitioning strategies.
- Data Processing: pandas, numpy,
- Data Extraction & Loading: boto3 (AWS SDK), requests,
- Database Connectivity: psycopg2, pymysql
- Testing: pytest, unittest
- The candidate should have a minimum of 5-6 years of experience in Pyspark
- AWS certifications at the Professional or Specialty level will be an added advantage
- Expected to perform independently and become an SME.
- Required active participation/contribution in team discussions.
- Contribute in providing solutions to work related problems.
- Collaborate with cross-functional teams to understand project requirements and deliver effective software solutions.
- Maintain and enhance existing software components to improve performance and reliability.
- Document development processes and solutions to support knowledge sharing within the team.
- Assist junior team members by providing guidance and support to foster their professional growth.
Professional & Technical Skills:
- Experience in PySpark for distributed data processing on Hadoop or Spark clusters.
-Ability to write optimized Spark SQL, DataFrame, and RDD transformations. -Strong experise using PySpark ETL libraries -pyspark.sql, pyspark.sql.functions
- Strong expertise in Python for ETL development.
- Must To Have Skills: Proficiency in PySpark.
- Experience in developing scalable software solutions using modern programming techniques.
- Strong problem-solving skills with the ability to analyze and optimize code performance.
- Familiarity with agile development practices and continuous integration/continuous deployment pipelines.
- Ability to work effectively in a collaborative team environment and communicate technical concepts clearly.
Additional Information:
- The candidate should have minimum 3 years of experience in PySpark.
- This position is based at our Bengaluru office.
- A 15 years full time education is required.
