Like all district administrators, Dr. Ayindé Rudolph, superintendent at Mountain View Whisman School District (MVWSD), wants to see students improve, be challenged and get excited about learning. But that’s not always easy when students perform at different proficiency levels and speeds.
It was about this time last year, in January 2016, when Rudolph was working on a strategic plan that he hoped could address the needs of all MVWSD students. Among the tools the district found was Teach to One: Math (TTO), a personalized learning model by New Classrooms, a New York-based nonprofit, that adapts to a student’s individual skills and gaps.
But the program didn’t work out the way the district had hoped or planned. Just weeks into the pilot last fall, parents wanted to see it removed from their children’s classrooms. Now, only about halfway through the pilot, the district has decided to pull the plug.
“Effective immediately, the district will discontinue using Teach to One,” Rudolph wrote in a letter to parents on Jan. 12. “We are committed to personalized learning, but can't continue a program that does not meet the needs of all of our students.”
A few days later, at a study session with parents on Jan. 17, Rudolph explained the motivation behind the decision: “It’s hard to argue with personalized learning… but when everybody is sitting there saying ‘yes, this looks good,’ maybe we should ask some more critical questions.”
What MVWSD Wanted
Teach to One had many of the ingredients that Rudolph was looking for. Content is delivered at different paces based on a student’s needs. High achievers can accelerate into above-grade level standards, while those who move slower can get extra practice. TTO is also aligned with Common Core standards and runs on the Chromebooks that the district had invested in.
The district’s proximity to big technology companies at the heart of Silicon Valley also seemed like a cultural fit for this kind of futuristic learning. “Khan Academy is in our backdoor, and a lot of parents are used to technology,” Rudolph said in an interview with EdSurge. “That's one of the hallmarks on our community.”
Students work through the TTO curriculum at their own speed via 90-minute math lessons that combine digital content with peer and teacher-led experience. At the end of the session, a personalized five-question assessment (known as an “Exit Slip”) is given to test concepts they individually worked on that day. New Classrooms then analyzes those quiz results and generates a customized learning schedule for each student’s next day.
After gaining board and trustee approval in June 2016, the district decided to run a one-year pilot program for the 2016-2017 school year for sixth graders at the district’s two middle schools, Graham and Crittenden. After a year, Rudolph says the plan was to evaluate its results and decide whether to not to expand the program. The cost of the pilot was $521,000, according to the Mountain View Voice.
Canaries in the Coal Mine
But those plans soon saw setback, and shortly into the pilot program’s run, issues emerged. First was with the devices themselves. The district had ordered new Chromebooks for students, but those didn't arrive in time for the start of the school year, causing some delays in implementation.
“When you're using outdated computers, you run into technical issues,” says Rudolph. “Our intent was to have the books ready to go.”
There were also issues with the program itself. Rudolph says some students experienced trouble filling out their Exit Slips, including difficulties with submitting the assessment. This was problematic, since these results are used by the program to assess a student’s performance to create their next day’s curriculum.
Unhappy Parents
Technical issues aside, some parents weren’t happy with having their students spending significant time with a computerized math curriculum. “Not every child responds to learning on a computer,” one mother said at the study session. “Don’t experiment with our children.”
“I want my 5th grader to be well-educated… Tapping away at multiple choice questions is not high level math,” another parent said. “These bright guessers are going to be in big trouble in high school if we don’t push them beyond that.”
These statements echoed their concerns in a letter signed by 180 parents on Dec. 16, urging the district to discontinue the program.
“The TTO program is still under development and the kinks have not been worked out,” the letter states. “Continuing to require the program, now that we have first-hand experience with the problems, would be a disservice to our children.”
The letter lists specific complaints such as a student's ability to pass Exit Slips without fully mastering concepts, and that some of the instruction is “factually incorrect.”
“Glitches in the system/internet mean that students often miss instructional time, are unable to complete their Exit Slip, or are late to their next class,” the parents wrote. “Unfortunately this has happened quite a few times to many of our children.”
Hearing their pleas, the district announced in a letter on Dec. 20 that it would be cutting TTO instruction time in half—with the other half turning to a non-digital Eureka Math curriculum—“to strike a better balance between technology-assisted and teacher-led instruction.”
Pulling the Plug
The decision to cut the TTO pilot in half was made without any student benchmark or assessment data available. The district acknowledged that parents had valid complaints, but said the program also had supporters among teachers, students and other parents. “Our teachers and students stated that they find value in the math advisory, teacher-led instruction, virtual instruction and reinforcement, and task portions of TTO,” the district letter reads.
Yet the new plan was barely in effect when the district suddenly announced on Jan. 12 it would drop TTO altogether. In a letter sent to parents, the administrators referenced assessment data that suggested the number of students performing at grade level dropped from 58 to 52 percent since using the program.
But the data cited in the letter was problematic. For starters, the 58 percent figure came from the CAASPP, while the 52 percent figure came from an entirely different assessment, the Northwest Evaluation Association Measure of Academic Progress (MAP) exam. There was also data that showed the program had actually worked: according to results from teacher-administered assessments, students demonstrated higher performance in some standards compared to previous years.
“With conflicting data points, it is hard to ascertain if TTO is having a positive impact on student performance because the latest data reports show the results are mixed,” Rudolph said in the Jan. 12 letter. “Some students aren't performing as well as we had hoped.”
But the mixed-results, coupled with negative qualitative feedback from parents, proved to be too much, and the district decided to call it quits.
Communication is Key
Still many questions and lessons remain. What might have happened if the pilot ran its full course? Is mixed data enough to call a program a failure? How do you know when it’s time to end a pilot, versus time to push through inevitable bumps in a new program?
For its part, New Classrooms believes it could have helped the transition to a new curriculum go a bit smoother.
“We and the district should have done a better job helping parents understand how it works and why it’s a far better way for their children to learn,” Jennifer Kohn, spokesperson for New Classrooms, said in a prepared statement.
On the district’s end, Rudolph echoes how better communication could have fostered a more successful pilot experience. “We focused on training teachers and working with coaches to make sure the process would be executed, but part of it is that we didn't do enough to bring people along the process,” he said at the Jan. 17 study session. “We could have done a better job at that.”
Moving Forward
Though MVWSD will now go back to teacher-led instruction with Eureka Math, Rudolph and other board members say they’ll continue to look for tools and methods that can help personalize learning for students. “At the heart of innovation is the need to look back and reflect on what took place,” Rudolph says. “This is a great opportunity for our district to learn, adapt and grow.”
In addition to improving communication with the community when new programs are introduced, they’ve started brainstorming what to do better next time. First, Rudolph says they will rethink their sample size and not test a program across an entire grade level. Second, he says the district should work to gather more quantitative data earlier. “We had a lot of qualitative data, but you want to be able to see whether or not [the pilot] is working,” he says.
Had the pilot gone for the full year, New Classrooms believes the outcome might have been different. “Formulating any assessment on the impact of any educational program after such a short period of time, much less one that operates so differently than a traditional classroom model, is highly unusual,” New Classrooms wrote in a letter to MVWSD board after its decision to end the pilot.
Rudolph says a district leader must look at data holistically, taking into account both parent feedback and qualitative information—rather than just relying on numbers—to make a decision that feels right for the community.
“Part of our job is to look at all of the data,” he says. “If you can’t get a clear picture of what’s taking place, then you have to make the best assessment possible, and that's what we did.”
