There is a popular quote attributed to management expert Peter Drucker: “What gets measured gets improved.” In education, the mantra is equally true. However, since I began working in edtech five years ago, I have been continuously surprised by how little emphasis there is on measuring changes and progress in individual learning, especially in higher education.
This is particularly concerning given that it’s difficult to improve learning if we can’t measure it. However, I think attitudes around why it’s important to track a learner’s progress, and how to accomplish it, are starting to change. With better tooling and more emphasis, we'll see significant progress in 2018.
The most obvious sign that measuring learning is not a priority in higher-ed is that administrators and educators throw away so much data about it. Instructors grade the vast majority of high-stakes summative assessments on paper, and collapse student performance down to a single number for entry into the grade book. Instructors spend hours looking at how each student arrives at an answer, figuring out where they went right or wrong, giving them feedback—but then compress all of that information down to a single grade that omits the nuance and specific areas of improvement for a student.
Holding on to this kind of data, however, is crucial for measuring students abilities and learning. And keeping information that would have otherwise been discarded after grading could lead to better outcomes within a course. For example, if it were easier for instructors to record student performance on specific learning outcomes, instructors could track progress across assignments. This kind of insight could give an instructor a data-informed prediction of how students might do on future assignments, and enable them to provide targeted interventions to give students extra support before the test.
Without technology, this process may involve a professor going back and digging through old assignments to get an idea of where a student was once at, and comparing it to their latest work. Over the next year, we will see more technologies stepping in to do that digging for instructors, giving them an easy way to view a student’s performance over time and thus more bandwidth to focus on students themselves.
One trend that is picking up steam is the demands by educational technologists that institutions have access to all of the data that pass through technology solutions, often using standard formats like Caliper and xAPI. Administrators I have spoken with say they have set up data warehouses that allow learning analysts to build these types of tools using the full range of data available on campus.
Data isn’t a silver bullet, however. The more an instructor is able to provide meaning to the data, the more useful the insights derived from the data will be. Another emerging trend in higher education is the push for instructors to be more rigorous about defining learning objectives for their course. Taking it one step further, more instructors are also starting to assess their students via standards-based grading, where students are assessed against defined learning objectives. Together, these trends will make it much easier for instructors to understand the strengths and weaknesses of their students.
The same data would help in improving repeated offerings of the same course. If instructors could effortlessly measure student progress on every learning outcome, they would have the information they need to understand which parts of their courses need the most attention.
Data on how students perform across a sequence of courses is plentiful. Ad-hoc analyses of this data are periodically performed, for example, when planning the curriculum for a major. These analyses often lead to information such as “students who take Math 1 before CS 2 generally score a half letter grade higher.” The ad-hoc nature of these kinds of studies, however, limit their ability to regularly capture information. Automating these kinds of trackers would thus allow for instructors to regularly use act upon student data, and improve their teaching in the process.
Benefits of this kind of technology would start start at day one. For instance, instructors could be presented with a report about their students on the very first day of the course, eliminating the need to test students on prerequisite material.
Gathering and synthesizing information about student learning and performance won’t only be useful for faculty to see, but students could gain valuable information about their own progress as well. For example, when instructors define and assess against learning objects, student could see their progress in attaining those objectives.
Every industry in the world is hungry for data to improve its workings. Education is unique in that data is actually plentiful—it just hasn't been captured and connected in the right way yet. I look forward to less data being discarded and seeing silos broken down in 2018.
