Education Data Portal

From codebook to endpoint

Each release started with Urban’s data team, who cleaned the new data and documented it in codebooks and a release specification. We reviewed those codebooks to work out what each release changed: a new year of data, new variables, or new endpoints.

Then we decided how to make that change in the Django API in a way that was repeatable and testable, so the next release of the same dataset would follow the same steps and could be checked the same way.

A release with checkpoints

A release moved through stages, and each stage had an owner. The data team loaded the new data and metadata into the staging database. We deployed the matching code to staging, and the data team checked the endpoints. The Explorer was updated to match.

Next came a soft release: the production site pointed at the staging data, so the public could use the new data while production was updated. Then the full release switched everything over.

Faster responses

The data only changes at release time, so we added Redis caching to production: a response is cached until the next release replaces the data, and the cache is rebuilt after each deployment.

A build that fails loudly

In 2023 we moved the portal’s API, documentation site, Explorer and proxy server into Docker containers, configured once and overridden per environment: development, staging, pre-release and production. Pushing to a branch deploys it through CircleCI, and the deploy script stops the build if any step fails rather than bringing up a broken release. Logs now go to AWS instead of filling up the servers.

We also wrote the developer documentation: setup, deployment, architecture and environment variables, so the next developer would not have to rely on a handover call.

Our role

We shipped data releases, deployed the new Education Data Explorer in place of the old one, modernized the build and deployment, and trained Urban’s own developers to run releases themselves.