Jurying Tool


Bill & Melinda Gates Foundation, LDC


Jurying Tool

Services Provided
  • Product Design
  • Project Management
  • Web Development
November 2012
Project kick off
December 2012
Remote user feedback
May 2013
First jurying session with new tool
December 2013
Transition to CoreTools module authoring
January 2015
Professional development incorporated into CoreTools

The background

Back in 2009, a set of high-quality academic standards, known as the Common Core, was being created in the United States. The standards outlined what a student should know, divided into two categories:

College and career-readiness standards (CCRS), addressing what students are expected to know and understand by the time they graduate from high school.
K-12 standards, addressing expectations for elementary school through high school. The goal was ensuring all students graduate from high school with the skills and knowledge necessary to succeed in college, career, and life, regardless of which state they live in.

Anticipating the release of these standards, the Bill & Melinda Gates Foundation assembled the Literacy Design Collaborative (LDC), a design team of literacy and curriculum experts to develop a strategy that would not only incorporate the new literacy requirements, but also meet demands for high performance in core content instruction. It was mostly a grassroots, teacher-driven movement that through research, experience, and an iterative process in the field, conceived a framework that incorporates the CCRS to focus instruction on critical-thinking tasks that could be applied to English Language Arts & Literacy, social studies, and science courses.

Testing LDC Systems with Teachers

The challenge

The framework consisted of Microsoft Word templates that were introduced to a wide range of partners involved in the support of literacy instruction. As use of the templates was gaining momentum, the LDC learned that teachers had difficulty using the templates. They needed support to adopt the process, easier ways to collaborate with peers and coaches, and methods to track and measure impact on the ground.

At Mobility Labs, we initially got involved with the project because LDC wanted to improve the quality of the lesson plans, as well as grow and scale the use of the LDC Framework. The best way at the time to do that was through jurying events. Building a jurying tool would allow other teachers and experts to analyze lesson plans and decide if they had covered all of the necessary requirements for it to be quality instruction.

User-centered design and solution

While the solution for a quality assurance tool seems simple and straightforward, we started at the beginning to understand the problem. This involved including teachers deeply in the process. We used user-centered design methods to translate the offline process into an interactive online experience for jurors.

The first step was watching the process in action. We needed to understand what the process was, what they were doing, and where technology could improve it. A big part of transforming the process from pen and paper to digital was coming up with ideas and designing concepts, showing it to teachers and asking if we did it right. Instead of doing individual sessions, we created micro-sites and micro-surveys to ask them about the new features. Wireframes and sketches were used initially, then videos that talked the teachers through what they were about to see.

The teachers would review the proposed designs and interactions and give us feedback. We’d receive, prioritize, and implement requests directly from teachers. Because this was done remotely, we were given feedback from teachers all over the country, very quickly and at low cost. The amount of time spent was minimal for what we were getting out of it. Every design and prototype was checked to see if it was actually solving problems from their perspective. By the time we showed them the actual software and asked them if it worked, they were delighted.

We included teachers in every step. They were part of our design process. Anytime we went to in-person events, we talked to them, asked them what was missing, what they needed, and what their biggest challenges were, even outside of LDC. Realizing that LDC is a teacher tool, and that as teacher’s needs change it may have to change, we were constantly asking questions about their challenges. The development of the jurying tool was quick and gave us room to propose the next step in helping teachers: A lesson-plan authoring tool called CoreTools.

When the Bill & Melinda Gates Foundation found Chad Vignola, LDC transitioned from being an ad hoc, teacher-driven initiative to one that had an organization shepherding that process of how to improve LDC. We have continued to work with LDC, helping to scale both the technology and the concept aligned with Chad’s vision. As a result, what LDC needed to achieve that vision technology-wise. We worked closely with LDC and other teams during this early time, helping him grow his internal team in a very open and collaborative process.

Continued partnership

The jurying tool was the first step for LDC to become an organization with its own innovative technology. Fueled by the success of this tool, we helped them build capacity for teachers by creating an authoring environment, work embedded professional development, and a resource website. As LDC has grown, Mobility Labs has been there to work on a blended technology team.

Contact Us