The Data Engineer will be responsible for the end-to-end design, development, deployment, and maintenance of the Metadata Registry Platform for the State Data Authority (SDA) of Uttar Pradesh. The role is positioned at the DataHub at Bharti Institute of Public Policy, Mohali, to work closely with the SDA implementation team at a BIPP and the State Transformation Commission Lucknow.
The Metadata Registry is one of the initial goals in the roadmap of SDA development. This is a platform that enables departments of the Government of Uttar Pradesh to discover, classify, and govern their data assets in line with the state’s vision 2030.
The successful candidate will oversee the data pipelines, and backend development, oversee UAT, ensure deployment on the UPSDC servers, and build the monitoring dashboard.
The Data Engineer will carry out the following key activities:
- Platform Architecture & Design: Design the Registry’s system architecture, database schema, and API layer in alignment with BIPP’s technical blueprint. Ensure the architecture is modular, open-source, and deployable on the UP State Data Centre (UPSDC) infrastructure.
- Backend Development: Build RESTful APIs and backend services using Python/Node.js; develop data ingestion pipelines for dataset metadata; implement role-based access control (RBAC) for Data Owners, Stewards, and administrators.
- Compliance Dashboard: Build a real-time compliance dashboard tracking dataset registration status, classification completeness, and DPDP Act alignment across onboarded departments.
- UAT & Deployment: Lead User Acceptance Testing with STC IT team and pilot departments; resolve bugs; deploy the applications to the UPSDC-compatible production environment.
- Junior DE Supervision: Mentor and technically supervise two Junior Data Engineers; review code, assign tasks, and ensure delivery quality.
- Frontend Development: Although, there will be Full Stack developers working along in the team, but the candidate is required have some knowledge of frontend frameworks to oversee and align the development work.
