These days there’s a wave of new edtech products hitting the market, and teachers and professors are increasingly making teaching videos and other materials for their classes. But one group is often left out of the design process: students.
“Many educational products are never shown to students until they have already been designed,” said Elliott Hedman, a consultant who works with edtech companies, in a talk this month at the SXSW EDU festival.
These days most major media and consumer tech products undergo extensive testing, he pointed out, but he says that many grown-ups are fine with a much lower standard for anything designed for kids.
“The budget for one episode of Game of Thrones was $15 million. It was probably worth it — it was a very good episode,” he quipped. “I get hired sometimes to write reading stories for kids in classrooms, and I get paid $80. So that’s how much we’re willing to pay for a kid’s experience in the classroom.”
That’s perhaps an extreme example, Hedman admits, but he says that lack of testing really shows when students are given the materials. As he visited school computer labs for one research project, he says, “I found that a lot of the kids, they alt-tab off the education program and start watching YouTube for an hour.”
In an interview after his presentation, Hedman, who has a Ph.D. from the MIT Media Lab and has been working on design of educational materials for more than a decade, said it’s not that edtech companies don’t do any testing. But he says the testing they do is often ineffective or too limited, such as relying mainly on family members of company employees, who are “way above average in school — and they are very white.”
He notes that there aren’t good incentives for edtech companies to spend the time and effort on more-detailed testing with students. “They’re selling to the government, to the administration, to the district,” he points out. “They’re not selling to the child — the child has no purchasing power. The kids never really get heard and the teachers rarely get heard. Then they throw it into the classroom and then you’re testing, ‘Did the scores go up?’”
And he argues that doing more user testing with students during the design process is different than doing efficacy research of an edtech product — which a growing number of education leaders have called for in recent years.
“I’m not saying efficacy research is a bad thing,” he says, but he says it can “get in the way” of doing the design research, since most research on whether a teaching approach works or not does not focus on making changes that would correct any failings.
He says the ideal approach is to put an edtech product or learning material in front of a diverse group of students — without a researcher in the room — and videotape how they use the tool. Then designers should make small improvements based on what they learn, he says, and keep doing that in an iterative way throughout the development process. In that way, it’s more of a co-creation with students rather than adults building something and assuming it will work for kids.
Experts have also called for more teachers and educators to be involved with the development of edtech products.
Instructors as Designers
Increasingly, instructors themselves are the ones creating learning materials — such as, say, a short video lesson if they are trying a “flipped classroom” approach of having students learn some basic material at home so that class time can be used for more interactive activities. And some experts say that students are too often left out of the design process for these materials as well.
“The challenge of having materials that are not good user experiences is they make the learning more difficult,” says Kayla B. McNabb, an assistant director of teaching and learning engagement at Virginia Tech. “If you want students to have good outcomes in your course, then you have to lower the barriers to them engaging in the learning experience.”
McNabb, who earned a Ph.D. in rhetoric and composition from Virginia Tech, co-authored a paper in 2021 calling on educators to do more to incorporate user testing when designing learning materials.
In many cases, what instructors learn by, say, running a short sample of a video they made for a class by a few students, is not about the content, but about the user experience, McNabb told EdSurge in an interview. And often the issues raised by students can be easy to fix and can be applied to other projects as well.
“Maybe the transition into the video is too jarring,” she says. “In that case, let's put a different header image to make it clearer.”
McNabb says one major challenge at colleges is that many instructors have never been trained in how to teach, much less how to design learning materials. But she says that even spending a small amount of time sharing some part of course materials with students to get input can help an instructor make changes that will greatly improve impact.
“The number one thing people can do is ask for feedback,” she says, noting that any time in the development process can work. “Any time is better than never.”
And if instructors can’t find students to test their materials on, she suggests asking a colleague to test it, or even present it at a conference. “Any feedback is better than no feedback,” she argues.
Changing Times
The need for product testing may become even more important as companies rush to add new AI features into their products, says Hedman.
“How kids use AI tools is very different then we think they do,” he says. “I have to remind companies: You have never shown that to a child. It’s going to be very important as companies rush out with AI-infused products.”
But he says that doing so will mean a change in the culture of many companies.
“In an ideal world, I would like the voice of the student to be at the front and center of every edtech company,” he says.
One way to address the challenge may be to increase collaboration between college researchers and the edtech industry, argued Yu-Chen Chiu, a user design researcher, in a recent post on Medium. The goal, she argues, is “to develop impactful products that are not only ‘effective’ but also ‘usable’ and ‘desirable.’”
Involving students is also important now that students have more ways than ever to find information online, or using AI chatbots. That means that instructors should focus more on teaching the skills and ways of thinking in their discipline, rather than spending so much time on specific content or details that might change rapidly or that students can find elsewhere.
Essentially, she argues, the rise of AI will mean that many instructors will need to update materials they made in the past, and she hopes more educators will involve students in that redesign process.
“That doesn’t mean you have to drop everything and get a master’s degree in instructional design,” says McNabb. “It means you should be thinking clearly about user experience throughout the course.”
