Job Description - Senior Data Engineer
What We're Looking For:
CivicDataLab is seeking a Senior Data Engineer to support our current and upcoming interventions through open data platforms and open-source tools, focusing on building robust data pipelines for automated data mining, cleaning, standardisation and transformation for AI readiness. We are looking for people who are strongly aligned with our values and have an innate sense of problem-solving, automating processes and adapting well to dynamic environments. They will work alongside data strategists, public policy researchers and other stakeholders to develop automated ETL (Extract, Transform and Load) data pipelines and know how to model, store and manage bulk datasets. This will help us co-create comprehensive data analytics tools and dashboards for our diverse stakeholders.
At CivicDataLab, we believe in using data, tech, design, and social science to strengthen civic engagement and drive evidence-based decision-making. Our projects centre on building data strategy, platforms, and applications to foster data-driven governance.
Key Responsibilities:
Take ownership over developing scalable data infrastructure for our existing platforms and products from data orchestration pipelines using Prefect and Airflow, to Data Sharing Protocols for our open data collaboratives.
Create and oversee data APIs responsible for collecting, managing, and analysing data from diverse public data sources.
Maintain and monitor our existing open data platforms like Open Budgets India, Justice Hub, Open Contracting India, and oversee dataset migration into new platforms.
Thoroughly document code, processes, and all activities performed by the data team, ensuring clarity and comprehensiveness. This includes documenting algorithms, methodologies, data transformations, and the overall workflow.
Lead strategic planning and scoping for data engineering projects, aligning them with product roadmaps and stakeholder needs.
Mentor and provide technical guidance to a team of data engineers, conducting code reviews and capacity-building sessions.
Quality & compliance: Run structured checklists (licensing/ownership validation, PII checks, metadata completeness) and coordinate fixes with publishers.
Any graduate, however, candidates with a B.Sc./M.Sc. (or higher) in Computer Science, Data Science, or a related field will be preferred.
5+ Years of experience as a Data engineer.
Proficiency in Python, SQL, and Excel.
Experience building dashboards (Superset, Tableau, or Python frameworks).
Excellent written/verbal communication and documentation skills.
Public-sector experience: Demonstrated work with government departments/PMUs or allied institutions; comfortable with consultations and public-sector processes.
Onboarding literacy: Familiarity with dataset discovery, metadata, licensing/ownership, and basic PII/privacy checks; rigorous documentation and follow-through.
Operations & collaboration: Excellent coordination and communication skills; readiness for periodic intra-state travel.
Extensive working knowledge of API or Stream-based data extraction processes.
Ability to gauge requirements for the project, make decisions and coordinate with other team members.
Prior experience in actively working and contributing to FOSS (Free and Open Source Software) communities.
Familiarity with working with Agile methodologies and Scrum processes.
Experience with scalable infrastructure practices such as microservice architecture, infrastructure as code, distributed systems, scaling methods, load balancing and more.
How we work
CivicDataLab is based out of Delhi and has the presence of project teams located in Assam and Himachal Pradesh. We follow a hybrid model where our bandhus work out of the office for a minimum of 12 days per month (i.e.) 3 days a week. We use open-source tools and agile methodologies in organising our work.
Perks of Working with Us
At CivicDataLab, we encourage everyone to take up things that help one grow professionally, and you get an annual kitty of INR 60,000 to do so. This includes attending or speaking at conferences and workshops, taking courses, acquiring hardware or software licenses or even joining summer schools. We feel that learning a skill should never be a hurdle to solve important problems for the community.
