This story was published in partnership with The Moonshot Catalog.
In the late 1960s, Nobel Prize-winning economist Herbert Simon posed the following thought exercise: Imagine you are an alien from Mars visiting a college on Earth, and you spend a day observing how professors teach their students. Simon argued that you would describe the process as “outrageous.”
“If we visited an organization responsible for designing, building and maintaining large bridges, we would expect to find employed there a number of trained and experienced professional engineers, thoroughly educated in mechanics and the other laws of nature that determine whether a bridge will stand or fall,” he wrote in a 1967 issue of Education Record. But at a university? “We find no one with a professional knowledge in the laws of learning, or the techniques for applying them,” he wrote.
Teaching at colleges is often done without any formal training. Mimicry of others who are equally untrained, instinct, and what feels right tend to provide the guidance. As a result, teaching is, to use another building metaphor, not up to code.
There are widespread beliefs about the best way to teach and learn that have been proven wrong by science, yet they persist. Reading back over a textbook or taking lecture notes with a highlighter at the ready is often done by students, for instance, but these practices have proven of limited merit, and in some cases even counterproductive in aiding recall. And while many educators believe that word problems in math class are tougher for students to grasp than ones with mathematical notation, research shows that the opposite is true.
Simon spent the latter part of his career as a professor at Carnegie Mellon University, making the case for bringing in a new kind of engineer to help improve teaching. He knew it would mean a major change in how instruction of complex subjects happens, moving it from a “solo sport” of a sage on the stage to a community-based one where teams build and design learning materials and experiences — and continually refine them.
He also knew that the notion of introducing what he called “learning engineers” would face resistance from faculty convinced they already knew perfectly well what they were doing in their classrooms.
“A substantial part of the nation’s resources are being devoted to higher education,” Simon said. “The nation has a right to expect more than talented amateurism.”
In recent years, Simon’s ideas have found new traction, thanks to new computing technologies that would have seemed alien in the ’60s. Today, students frequently work in digital environments to read course materials, take tests and complete assignments. Those activities often leave data trails, making it possible to quickly measure how well, say, a section of an online textbook is conveying the knowledge a teacher hopes to impart, or whether the material needs to be revisited and revised.
For instance, an online biology textbook might include a short section about protein synthesis, followed by a question. If few students answer the question correctly, the software can flag the professor, or the textbook author, to consider revising the content to make it clearer. And as they revise, they can see how long learners spent looking at that passage and other details about how they moved through the digital tool, since every action leaves breadcrumbs to analyze.
Also in concurrence with Simon’s vision of more dynamic, data-guided teaching, colleges have begun hiring instructional designers. Working alongside faculty members, they are tasked with helping professors apply findings from learning research into classroom practice by collaborating on the design of learning materials and activities. Ten years ago, only about 1,300 instructional designers worked at U.S. colleges, but that has grown to more than 10,000 today.
Even so, we’re still a long way from having a mature practice of learning engineering in place. But proponents of the approach say they are beginning to build the infrastructure necessary for their moonshot of turbo-charging the speed and the quality of learning. Some learning engineers believe they can help students reach mastery of complex subject matter as much as 10 times faster than with traditional approaches.
If they are right, it would mean short-circuiting the famous “10,000-hour rule” based on studies by education researcher Anders Ericsson and popularized by bestselling author Malcolm Gladwell in his book “Outliers.” The rule—and that number—has many detractors, but the underlying principle is that it requires deep and focused work to achieve mastery. The hope is that if what would normally take 10,000 hours could be shortened to 1,000, and be done via methods that are more affordable and accessible, many more people can become experts.
If these new teaching approaches can deliver what their proponents promise, they could perhaps turn around pitfalls in college, like introductory math courses, which are part of a nationwide college completion crisis. Only 58% of students who started college in 2012 had graduated 6 years later. More than 4 out of 10 college students wind up in remedial math or English courses, and those that do are even less likely than other students to finish college. At a time when 9 out of 10 new jobs are going to those with a college degree, a teaching method that would help underprepared students whiz back on track academically could boost the prospects of millions and raise global productivity.
As this article was being written, the COVID-19 pandemic swept across the world, forcing a global experiment in online education as schools and colleges closed their doors and moved teaching to online formats. As a result, educators have been forced to rethink how they teach using digital instructional tools and practices.
While many of these hastily-created online experiences are improvisational rather than well-engineered learning programs, the increased use and awareness of the kinds of digital educational tools available could underlie a new culture of more evidence-based teaching.
Taking flight
Learning engineers, such as Kenneth Koedinger of Carnegie Mellon University, like to point to the Wright brothers as inspiration.
After all, for most of human history, humans couldn’t fly, and some said it could never be done. Now, 117 years after that famous first flight on the sand dunes of North Carolina, air travel has become routine and affordable (at least before the current pandemic).
But the Wright brothers didn’t rely on any one big new idea or invention in their Kitty Hawk garage that led to the Wright Flyer. Instead, “they deconstructed the problem into subproblems, like lift and drag,” says Koedinger, a professor of human computer interaction and psychology at CMU and a leading learning engineering researcher. “They were doing iterations — not on the whole-plane level but on the subproblems. It didn’t happen overnight, and there were a lot of incremental improvements in the engine and the wings and the weight and the fuel and lots of different dimensions.”
Learning engineers are taking the same approach, says Koedinger, breaking the problem of effective teaching into more-manageable subproblems, and bringing expertise from different disciplines, including neuroscience and psychology, to bear on each.
That deconstructed approach may even be more crucial in learning engineering than it was for flight, since teaching and learning arguably have more variables. In a paper Koedinger co-published in Science, he found that as instructors consider their course design and teaching options during a typical college course, they’re picking from trillions of possibilities.
Among the choices educators make are what instructional technique to use, when and how to give feedback, and when and how to test student knowledge. What’s more, for each of these choices, there are additional decisions about which media to use (video, audio, hands-on), whether to give concrete examples, and more.
Koedinger sees three main areas of learning engineering, which together give flight to the learner. The first is to hone and clearly scope out what students need to learn in any given situation. That’s referred to as the cognitive aspect. The second is to improve strategies for how students take in that information and retain it. That’s known as the metacognitive realm. The third is motivational — the fuel that keeps students pushing forward when they get stuck on difficult material. Getting substantial gains in learning, Koedinger says, takes “getting the details right” in all of those areas.
For him, improving the speed of learning is just part of it. “Imagine students in a control group go from 50% on a pre-test to 60% on a post-test, whereas in the treatment group they go from 50% to 80%,” he explains. “That’s 30% vs 10%, or a 3x bigger improvement in learning effectiveness. If the control involved 12 hours of work a week over one semester, but the treatment required only 4 hours a week, that’s a 3x improvement in learning efficiency. If both happen together, that’s a 9x improvement in the rate of learning.”
Refining an approach
On the other side of the country in Sunnyvale, California, John Newkirk has been persistently applying that philosophy to a teaching approach he’s been refining for 20 years. He has gotten a lot of support in the form of more than $50 million in funding from U.S. government agencies including the Defense Advanced Research Projects Agency (DARPA), the Department of Defense’s research shop for big new ideas.
Newkirk had previously served as a professor at Stanford University where he ran a research lab that pioneered integrated-circuit design. When he decided to tackle the challenge of education, he tried, in the spirit of Herbert Simon, to put aside any assumptions about how teaching ought to work — in Newkirk’s words, “to step back and say, ‘How do we make this better?’”
And Newkirk wasn’t interested in slight improvements. “The issue here is: how do you improve education by a factor of 10?”
The specific moonshot he recently set his sights on is to revolutionize the teaching of mathematics, specifically for college students who lack basic competencies to meet admission or graduation requirements. It’s a major pain point in the education system: Between 40 percent to 60 percent of college students now need some form of remedial math, or English, or both, and the United States ranks 36th out of a comparison group of 79 countries in math proficiency, according to the 2018 Programme for International Student Assessment.
Newkirk calls his company Acuitus, in hopes of encouraging sharpness of thought. He co-founded the company in 1999 with Maria Machado, who also got her start in the semiconductor industry before turning her attention to education. The Acuitus strategy that has evolved over 20 years involves mixing a digital tutor with in-person instructors. Newkirk concedes they are not the first to use these techniques, but the combination he and his colleagues have devised is getting results beyond what others have reported.
The starting point for Newkirk is to try to deconstruct what experts know — the cognitive realm of learning engineering. To do that, his team carefully analyzed how human tutors work with students, videotaping such interactions and looking for patterns.
What they’ve found is that the most effective tutors give just enough information or guidance to get the learner back on track, often posing questions rather than giving answers. As Carole Balcells, who helps develop curricula for Acuitus, puts it: “The one doing the thinking is the one doing the learning.”
That sentiment is backed by learning-science research in a concept called the “doer effect.” Studies have consistently shown that students tasked with responding to interactive exercises, like answering online quizzes, retain more than those tasked with passive learning activities such as reading or watching videos.
But Newkirk wanted to improve students’ motivation as well, so he brought in Mark Lepper, a psychology professor at Stanford who has studied how best to keep learners on task.
One insight Lepper brought is that when education software tools simply list all the errors students made and points out what they should have done instead, what many end up hearing is, “You’re wrong, you’re wrong, you’re wrong.” For students, this is a discouraging engagement, Lepper says.
“That kind of feedback would be perfect if you had a robot learner on the other end,” he says. “The robot learner would be delighted to have you say, ‘Okay, you made three errors in problem number one,’ and being a robot learner, they’d be able to take out those bugs and do better the next time. Real kids, especially real kids who are kind of phobic about math and who think they can’t do it, they leave and say, ‘See I can’t do it.’”
Acuitus’s earliest client was the U.S. Navy, and the company’s first mission was to train sailors in information technology (IT) support, so they could fix any computer or network problem that a crew at sea might encounter. The experiment was part of a DARPA program ambitiously called the Education Dominance Program, which awarded the company some $35 million.
“We are probably the most-studied educational program in history,” quips Newkirk, since [the government] required constant documentation on its progress. Those studies showed steady improvements over time as his team refined the computer tutor and the overall teaching protocol, including how much time students spend in in-person classes. And the approach produced graduates who vastly outperformed cohorts trained using traditional methods.
In a 2012 experiment, for instance, teams of IT graduates participated in a timed competition where they solved as many “trouble tickets” — as user complaints are called in tech support — as possible in a set amount of time. One team that had trained with the digital tutor solved more than 120 problems, and another did more than 140. Meanwhile, a team that had been on the job for 10 years and trained with traditional methods solved 41 problems, and another with similar training and background solved only one.
In the past year, the company started a version of that training program for civilians, creating an intense 5-month program to teach basic networking concepts to people with little background in computers. The program costs $35,000. Students pay nothing up front, but instead contribute a percentage of their salary once they have landed a job in the field until they pay off the tuition — a model called an income-share agreement.
So far, the classes have been small — about 15 students at a time — and the learning happens in the company’s office park in Sunnyvale, California. “This is the classroom,” Newkirk said with a smile during my visit in February, as he pointed to three rows of tables where students sat at computers wearing headphones. At first sight, one might think they were employees coding the software, rather than the ones learning from it.
There’s nothing glitzy about the software itself. (Newkirk likes to point out that the Wright brothers largely used technology for their Wright Flyer that had been available for decades.) The interface features two windows positioned side by side. The window on the right looks like a typical computer desktop running the Windows operating system. The window on the left is the digital tutor, essentially a chatbot that offers brief instructions and asks questions.
Students are given tasks such as, “Help a user figure out why they can’t print,” and the system monitors every move they make in the Windows environment as they try to come up with a fix. The digital tutor asks questions or gives nudges depending on how close the student is to completing the task correctly. At any time, students can ask for a hint, but even those are only clues about how to proceed. If a student is still floundering, the system sends a message to a human teacher in the room to come help. Even that human, though, is told never to give the student the answer, but only to ask more questions. The company’s research shows that this Socratic approach leads to the most lasting learning.
About once a day, all the students gather with a human instructor for a brief in-person lesson known as “study hall.” It’s a chance for the students to ask questions, share with each other, and take a break. “You’ve got a limit on the amount of mental energy that you’ve got,” says Newkirk.
Newkirk is also careful about limiting distractions. He says research shows that students who can maintain focus and concentration reach a state of “flow” that helps learning and retention. So the program doesn’t allow them to look at their smartphones or personal computers while in the building.
Erals Delao was one of the students working through the program on the day I visited. The 31-year-old had been working in ice cream manufacturing before deciding he wanted a career change. That’s when he saw an ad for the Acuitus program on Craigslist.
“It’s really different than anything else I’ve interacted with,” he says of the digital tutor. “It does have somewhat of a personality,” he adds, describing it as seeming “helpful” even though “it isn’t going to tell you the answer.” He notes that the computer sessions are constructive and instructive, but that he would prefer more time in study hall with other humans. “The hardest part for me is learning to just sit still in the chair so long,” he says.
Newkirk says that the company’s latest internal studies show that the company’s approach can deliver the kind of sped-up deep learning he set out to achieve.
Before the pandemic, Newkirk was in talks with the community college system in California to pilot the system for students in entry-level mathematics courses, though that is on hold for now while campuses are locked down. When campuses do reopen, the need for students to catch up on things like math instruction will likely be even greater than before.
The COVID-19 outbreak has forced Acuitus to temporarily shut down its in-person teaching — but that has allowed it to embrace an online format that could one day help the company’s approach reach a broader audience. Newkirk says he had long resisted online-only teaching because he worried that students would not be disciplined and focused enough when interacting with the tutor at home, and he felt the in-person teaching sessions were key. But now he and his team are forced to adapt to a world where coming to the building and sitting side-by-side at computers in an office park is not currently possible for health reasons.
Even in more normal times, Newkirk’s overall strategy has limitations.
For one thing, it’s expensive and time-consuming to develop. The 1,000 hours of content the company has developed specifically for IT troubleshooting took more than a decade to build. But he argues that for some subject areas, like college-level introductory mathematics, the payoff will be worth the effort since it can be used for years with as many students as one wants to teach. Newkirk believes that his model will work for other STEM fields, including chemistry and physics.
Another potential shortfall rests in whether the approach can work for nontechnical subjects such as humanities disciplines, where there is less agreement on what the right answers are, which makes it harder for a digital tutor to monitor how well a student is doing.
Moving forward will take time and money, on the order of $20 million, Newkirk estimates. Such an infusion of resources would enable him to run a project at a large enough scale to show others what is possible. He believes the student performance results will convince even skeptical educational institutions to adopt the model.
Turning classrooms into learning laboratories
For the learning engineers back at CMU, the goal is not to put digital tutors in every classroom. Instead, they want to deploy ways to better measure learning, no matter what teaching style a professor prefers. That way, instructors can apply a scientific approach to what they’re already doing in their classrooms, propose hypotheses for improvement, and see which tweaks work.
“It’s allowing every classroom to become a learning laboratory, and every educator to become a learning scientist,” argues Norman Bier, a CMU professor who leads an effort to encourage learning engineering at the university and beyond. The project is called the Simon Initiative, in tribute to Herbert Simon. “The key,” says Bier, “is doing so in an instrumented way.”
“Instrumenting” classrooms means being able to track what students are doing as they go through learning materials, such as digital textbooks and online labs, and seeing which behaviors tend to lead to the best performance on quizzes, exams or other measures of student learning.
In fact, over the past several decades, CMU has built a series of digital learning tools that address all three of the broad categories of learning engineering — cognition, metacognition, and motivation—that Koedinger outlined.
With funding from the National Science Foundation, researchers at CMU have developed an analytics tool called LearnSphere. The software can pull in data that learners generate as they move through software that colleges already deploy, such as learning-management systems including Blackboard and Canvas. The goal is to give professors a dashboard that shows trends in student activities so they can identify spots in a course that are working or need to be improved.
And for professors who want to create their own version of a digital tutor, CMU has built software known as the Open Learning Initiative (OLI). The toolkit has been used to build online tutors that have demonstrated significant gains. In one statistics course, for instance, students who learned with OLI software saw an 18 point gain, where students in a traditional section of that same course saw a 3 point gain. That’s equivalent to more than twice the learning in half the time, says Bier.
Now that such instrumentation exists, perhaps the biggest challenge is convincing professors to wire their classrooms to use it — and teaching them how to work all that software.
To that end, Carnegie Mellon began a bold effort last year to make all the learning engineering software it has developed over the past decade free and open source so that any institution in the world can adopt it. The open-source status allows users to get under the hood of the tool and assures them the availability of the software does not depend on the solvency of any company. The university estimates that more than $100 million of research funding has gone into building what they’re calling the OpenSimon Toolkit.
But as the old saying goes, free software is free the way a free puppy is. Devoting staff and faculty time and energy to learning and deploying these learning-engineering tools will cost colleges significant amounts, and it could take years before any gains are discernible.
“What would be really interesting is if they had donors or foundations make $10 million or some amount available to help universities implement these tools,” Brandon Muramatsu, associate director for special projects at the Massachusetts Institute of Technology’s Open Learning project, which supports online education at MIT and for other colleges through free courses and resources, told EdSurge last year.
Meanwhile, other colleges are taking a lighter approach to learning engineering by trying to apply insights from the science to recommend one specific tool or intervention, rather than asking professors to fully instrument a class.
One example is at Duke University, where its Learning Innovation center has built a tool called Nudge. It is based on a hypothesis called the Ebbinghaus Forgetting Curve, which shows that people forget new facts and details after a few days or weeks unless they are actively recalled . (Forgetting is the mind’s way of sweeping little-used information to make room for what seems more important.) But if details are recalled at certain intervals, then the learner will remember them for longer. Ideal results, some studies show, happen when following the “2–2–2 method,” prompting learners to recall information two days after learning it, then two weeks after learning it, and again two months after learning it.
The Nudge tool is a system for scheduling text messages that pose short questions to students that prompt them to recall things they’ve learned in class after a certain amount of time. The system sends students a text or e-mail 24 to 48 hours after a class, with one multiple-choice question about the material. The idea is to bring material from, say, a Monday lecture, back to mind before the Wednesday lecture so students can better build on the information.
“We now have research that shows that students improve their performance in a class by several percentage points just using this intervention,” said Matthew Rascoff, associate vice provost for digital education and innovation at Duke University in Durham, North Carolina, on the EdSurge Podcast.
Interestingly, the students end up getting higher grades even if they respond with the wrong answers on those short text-message questions, says Kimberly Manturuk, assistant director for research and development at Duke. “Simply the act of interacting with the information again moves it to the forefront of your memory,” she says.
Changing the culture
Convincing professors to adopt a learning engineering approach an be a tough sell.
That pushback often comes from professors who are convinced that what they are doing in the classroom is working, even when they are presented with evidence to the contrary. That was the finding of a study by CMU anthropologist Lauren Herckis. “For faculty who believe that teaching is an art, that it is just something that you develop with experience and time, that you can’t learn from a book, no amount of exposure to learning-science research is going to” change that belief, Herckis said in a podcast interview with EdSurge last year.
It’s a refrain among many working in learning engineering. David Wiley, CEO of Portland, Oregon-based Lumen Learning, which makes an online textbook platform that attempts to apply learning-science principles, says that given how professors often do research for their academic work, he has been surprised by how reluctant they are to embrace experimentation and data for their own teaching.
“The same way that there are sort of climate-science deniers, I swear there are learning-science deniers that just don’t want to believe that anything about learning can be quantified,” he says.
Herckis points out, though, that precisely because professors take research seriously, they may not feel they have the time to learn how to do proper research on teaching in an effective way.
And some professors say that instructors already do a form of learning engineering without digital data, and that informal feedback is more valuable than measuring clicks. “I collect data all the time from my students, but it’s qualitative data,” says John Warner, a longtime writing teacher and author of The Writer’s Practice. That data, he says, comes from him asking questions like ‘What did you learn this semester?’ and ‘what can you do now that you couldn’t’ do at the start of the class?’
Another factor that could help explain why not everybody is onboard with learning engineering could be described as “innovation fatigue” among educators, who are wary of overhyped solutions in education. After all, many high-tech ideas for remaking higher education have made splashy headlines but fail to deliver. Large-scale online courses called MOOCs (massive open online courses), for instance, were touted as possible low-cost replacements for residential colleges, but proved to have completion rates of less than 10%.
Another dashed hope was the digital tutor made by a New-York-based company called Knewton, which one education consultant rated as “snake oil” in an NPR story in 2015. In that report, the company’s CEO, Jose Ferreira, described the tool as something “like a robot tutor in the sky that can semi-read your mind and figure out what your strengths and weaknesses are, down to the percentile.” That led to a backlash among some educators, and after the software failed to catch on, the company was sold off for a fraction of what investors had put in.
Ferreira told EdSurge that his system was effective, however, and that he had more data on its success than he was able to publish to make his case. He said his comments were meant not as an overheated claim, but as a way to explain a new approach.
In recent years learning engineers have begun organizing and trying to make a stronger case for their work.
In 2017, the Industry Connections program of the Institute of Electrical and Electronic Engineers established a special interest group, the Industry Consortium of Learning Engineers (known as ICICLE), which aims, according to its website, to “develop learning engineering as a profession and as an academic discipline.”
And the consultant who said Knewton was peddling snake oil, Michael Feldstein, now runs a group called the Empirical Educator Project to promote an evidence-based approach to teaching. “As the reality sinks in that the shift to online education will continue indefinitely — and some of it permanently — now is a particularly good time to re-examine our beliefs about effective teaching,” he wrote in a recent op-ed.
And over the past few months, CMU has seen a surge of interest in its open-sourced digital tutor system. In a period when they expected about 80 new instructors using the system, they’ve had 1,000, says Bier. “We’ve been spending a lot of time and effort supporting new users,” he says.
And Herckis said that she is now studying whether the move to remote teaching during the pandemic is leading to greater adoption of more evidence-based teaching practices.
“A lot of people adopted tools and practices that they never would have entertained under other circumstances,” she says. “For some people they’re going to say, ‘I never would have tried this on my own but now I’m going to use it in all my classes.’”
The first flight by the Wright brothers wasn’t far—just 120 feet. “The planes that they built were not airliners,” says Feldstein. Only through iteration and careful testing did the inventors overcome the obstacles that kept other designs from staying airborne longer. “It’s the questioning of your underlying assumptions,” Feldstein concludes, “that enables the possibilities of a real breakthrough.”
How many hours rule?
When Anders Ericsson first published the research that described the 10,000-hour rule in 1993, his focus of study was the teaching of violin — because it was a domain where there is widespread agreement of what expertise looks and sounds like. Ericsson, who is now a professor of psychology at Florida State University in Tallahassee, says that since his rule has become well known, he has been contacted by music teachers who say that the research has inspired them to make improvements in their teaching to get the hour count down to a few thousand.
Essentially, these teachers are now taking a learning engineering approach to their own instruction.
“I’m working with one guy who is a musician at the Metropolitan Opera and in New York City,” Ericsson says, adding that that teacher is now trying a new technique by asking students to film the lessons and rewatch them later, and then measure whether adding that technique reduced the hours needed to master a piece.
At the heart of the 10,000-hour rule research is the assumption that it takes that long to become an expert, even when a teacher observes students regularly to give them feedback and the students practices with that feedback in mind. But Ericsson said this teacher realized that students often forget the key feedback he gives during lessons. By recording video so that students can review the key feedback and apply it as they practice, the teacher said that his students are now learning faster. That teacher is now considering asking students to videotape themselves practicing, hoping that might further improve the efficiency.
The point is, there’s nothing set in stone about the 10,000-hour rule. Teaching can be improved through careful trial and error — the very experimentation that professors so often do in their academic research.
It’s not clear how much more efficient teaching can be, and how much faster people can achieve mastery of skills and knowledge. But as Herbert Simon challenged more than 50 years ago, more educators could at least be open to learning from the data and evidence that is theirs for the taking just about every time they interact.
