Job Type: Contract (C2C)
Experience: 7–10+ Years
Location: Remote (India) / Onsite (US)
Job Summary
We are seeking an experienced Senior Data Engineer to design, develop, and optimize large-scale data extraction and delivery solutions for an enterprise Compliance Data Platform. The ideal candidate will have strong expertise in Snowflake, Python, Amazon S3, and cloud-based data engineering, with experience building scalable batch processing pipelines, REST APIs, and microservices.
This role involves designing high-performance data delivery pipelines, optimizing query performance, integrating APIs, and collaborating with cross-functional teams to deliver secure and scalable data solutions.
Key Responsibilities
- Design, develop, and optimize scalable data extraction and delivery pipelines using Snowflake and Amazon S3.
- Develop data extraction logic using Snowflake, including External Tables and Iceberg Tables.
- Build efficient batch processing and API-driven data retrieval solutions.
- Develop reusable microservices for data extraction, transformation, validation, and enrichment.
- Integrate data delivery services with FastAPI-based REST APIs.
- Optimize query performance, indexing strategies, and data retrieval efficiency.
- Work with Parquet data formats and cloud storage solutions.
- Implement audit logging and monitoring for data extraction and delivery processes.
- Collaborate with Solution Architects, API Developers, QA, Security, and Business teams.
- Ensure data security, governance, and compliance standards are followed.
Required Skills
- 7–10+ years of experience in Data Engineering or Data Platform Development.
- Strong experience with Snowflake, including query optimization and External/Iceberg Tables.
- Experience with Amazon S3 and Parquet data formats.
- Strong Python programming skills for data engineering and pipeline development.
- Experience building enterprise-scale batch data extraction pipelines.
- Strong understanding of indexing, partitioning, and data retrieval strategies.
- Experience developing and consuming REST APIs.
- Experience with Microservices architecture.
- Hands-on experience with Docker and/or Kubernetes.
- Strong SQL and query optimization skills.
- Excellent analytical and problem-solving abilities.
Preferred Skills
- FastAPI or similar Python frameworks.
- Apache Spark or distributed data processing frameworks.
- Serverless data processing experience.
- Audit logging and metadata-driven platforms.
- Healthcare, Insurance, or other regulated industry experience.
- Knowledge of PII/PHI compliance requirements.
- Experience with SFTP or enterprise batch delivery solutions.
Why Join Us?
- Work on enterprise-scale cloud data platforms.
- Opportunity to build modern, high-performance data delivery solutions.
- Collaborative Agile environment with cutting-edge technologies.
- Fully Remote (India) / Onsite opportunities (US).
- Long-term enterprise project with a global client.
Pay: ₹80,000.00 - ₹100,000.00 per month
Experience:
- Snowflake: 3 years (Required)
- Kubernetes: 3 years (Required)
- apache spark: 4 years (Required)
Work Location: Remote
