Data Pipeline Development & Engineering
Design, build, and maintain scalable, reliable, and efficient data pipelines to support analytics and reporting needs
- Develop and manage ETL/ELT workflows using Apache Airflow to orchestrate complex data movement and transformation processes
- Optimize data ingestion, transformation, and loading processes to ensure timely and accurate data availability
Troubleshoot and resolve pipeline failures, data quality issues, and performance bottlenecks
Data Platform & Infrastructure Management
Work with large-scale distributed data platforms including Hive and Presto for data storage, querying, and processing
- Manage and optimize data warehouses and data lake architectures on AWS (S3, Redshift, Glue, EMR, Lambda, etc.)
- Ensure high availability, scalability, and performance of data infrastructure
Implement data partitioning, indexing, and query optimization strategies to improve performance and reduce cost
Process Excellence & Automation
Data Quality & Governance
Implement robust data validation, quality checks, and monitoring frameworks across pipelines
- Ensure data accuracy, consistency, and integrity across all data sources and reporting systems
- Collaborate with analytics and business teams to define and enforce data governance standards
Maintain comprehensive documentation for data models, pipelines, and data dictionaries
Technical Development & Advanced Analytics Support
Perform advanced data extraction, transformation, and analysis using Python and SQL
- Build reusable data models and transformation logic to support multiple analytics use cases
- Work with structured and unstructured datasets from diverse sources including transactional systems, marketing platforms, and third-party APIs
Support data scientists and analysts by providing clean, well-modeled, and readily accessible datasets
Cross-Functional Collaboration
Partner with Data Analytics, Product, Marketing, Operations, and Technology teams to gather data requirements and deliver engineering solutions
- Drive alignment and execution across multiple stakeholders and geographies
- Translate complex technical concepts and data architecture decisions clearly to non-technical leadership
Manage multiple priorities in a fast-paced and dynamic environment
Requirements
Preferred Skills
Experience with real-time/streaming data pipelines (Kafka, Spark Streaming, Kinesis)
- Exposure to data quality frameworks and observability tools (Great Expectations, Monte Carlo, etc.)
- Familiarity with dbt for data transformation and modeling
- Experience with Infrastructure as Code (Terraform, CloudFormation)
- Understanding of statistical and analytical concepts to better support data science teams
Exposure to marketing, customer, or product data domains
Behavioral Competencies
