Position: Senior Data Engineer
Requirement: Ab-initio, Kafka, ETL
Experience: 6+ Years
Working Hours: 3 hours per day (Between 7 PM IST to 1 AM IST)
Working Days: Monday to Friday
Location: Remote
Job Description / Task Overview
We are looking for a Spark Developer with hands-on experience in big data processing to implement data pipelines involving Parquet file creation and consumption, along with integration into Ab Initio workflows.
Key Responsibilities
1. Create Parquet File using Apache Spark
- Develop Spark jobs (using PySpark/Scala) to ingest data from source systems (CSV, JSON, database, etc.)
- Transform and cleanse data as per business rules
- Write processed data into Parquet format with optimized schema and partitioning
- Ensure efficient storage using compression (Snappy preferred)
2. Read Parquet File using Apache Spark
- Load Parquet datasets into Spark DataFrames
- Perform transformations, aggregations, or validations
- Optimize read performance using partition pruning and predicate pushdown
3. Read Parquet File in Ab Initio
- Configure Ab Initio graph to ingest Parquet files generated by Spark
- Use appropriate components such as:
- Reformat
- Input File / Dataset component (via external table or conversion layer)
- Convert Parquet data into Ab Initio compatible format (e.g., DML structure)
Approaches:
- Use Parquet-to-CSV/Avro conversion before ingestion
- OR leverage custom scripts / connectors to read Parquet directly
- Define DML schema matching Parquet structure
Required Skills
- Strong experience in Apache Spark (PySpark/Scala)
- Hands-on with Parquet file format
- Knowledge of data partitioning and compression techniques
- Experience with Ab Initio (GDE, Co>Operating System)
- Understanding of ETL pipelines and data engineering best practices
Good to Have
- Experience with Hive / HDFS / Data Lake architectures
- Familiarity with Avro / ORC formats
- Performance tuning of Spark jobs
Expected Outcome
- Successfully create optimized Parquet files using Spark
- Efficiently read and process Parquet data in Spark
- Seamlessly integrate and consume Parquet data within Ab Initio workflows
Job Types: Part-time, Freelance
Contract length: 6 months
Pay: From ₹30,000.00 per month
Benefits:
Experience:
- Spark: 6 years (Required)
- Parquet processing: 5 years (Required)
- Ab-initio: 5 years (Required)
Work Location: Remote
