Job Description: Data Engineer – Business Insights
Location: Bangalore [Hybrid]
Experience Required: 5–7+ years of experience in Data Engineering, Big Data, or Analytics Engineering.
Role Overview and Key responsibilities
We are looking for a highly skilled Data Engineer to build and manage scalable data pipelines for our 4PL (Fourth-Party Logistics) Business Insights platform. The ideal candidate will design and implement robust ingestion, transformation, and analytics-ready data infrastructure that powers AI-driven business insights and operational intelligence.
• This role will be responsible for building end-to-end pipelines from Existing Kafka spine + Debezium CDC + Apache Flink for streaming transformation along with supporting bulk ingestion from CSV and other flat-file sources
• Would need the candidate to have working experience with Apache Iceberg on Amazon S3
• Should be familiar with ClickHouse for building customer dashboards and Trino/Athena for historical queries
• Design, develop, and maintain scalable data pipelines for ingesting logistics and operational data into the analytics platform.
• Strong SQL skills and experience optimizing analytical queries.
• Familiarity with containerization and cloud-native deployments.
• Proficiency in Python, Scala, or Java.
Data Lake & Warehouse Management
• Manage and optimize data flow from Kafka topics into S3-based storage layers. Build ETL/ELT pipelines to transform and load data into ClickHouse for high-performance analytical querying.
• Design partitioning, indexing, and schema strategies in ClickHouse for low-latency AI and BI workloads.
AI & Analytics Enablement
• Enable AI agents query ClickHouse datasets. and analytics applications to efficiently
• Ensure data quality, consistency, and availability for downstream AI-driven insights.
• Collaborate with AI/ML teams to expose optimized datasets and semantic models.
Platform Reliability & Optimization
• Monitor and optimize pipeline performance, storage efficiency, and query latency.
• Implement observability, alerting, and retry mechanisms for ingestion pipelines.
• Ensure scalability, fault tolerance, and data governance best practices.
Collaboration
• Work closely with: o Product teams o Business Insights teams o AI/ML engineers o Platform engineering teams
• Participate in architecture discussions and contribute to long-term data platform strategy.
Required Skills & Qualifications
Technical Skills
• Strong experience in building distributed data pipelines.
• Hands-on expertise with: Data Engineering Concepts
• ETL/ELT pipeline design
• Data modeling for analytics
• Data partitioning and indexing strategies
• Schema evolution and metadata management
• Monitoring and observability
Nice to Have
• Experience with logistics, supply chain, or 4PL platforms.
• Exposure to AI/LLM-based analytics systems.
• Familiarity with vector search or AI retrieval architectures.
• Experience with dbt or modern data stack tools.
• Knowledge of Iceberg, Delta Lake, or Parquet optimization.
Preferred Qualifications
• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• Experience working in high-scale analytics or real-time data environments.
Success Metrics
• Reliable real-time and batch ingestion pipelines.
• Optimized ClickHouse performance for AI-agent querying.
• Reduced latency for analytics and reporting workloads.
• High data quality and pipeline uptime.
What We Offer
• Opportunity to build next-generation AI-powered logistics insights platforms.
• Work on large-scale distributed data systems.
• High ownership and architecture influence.
• Collaborative engineering culture focused on innovation and scalability.
