On the front page of the Science Times of September 2, 2013, Gina Kolata celebrates the U.S. Education Department’s efforts that are “starting to get some real data” about what works in science and math education. The key to replacing guesses and hype, writes Kolata, is a “method that has transformed medicine: the randomized clinical trial.” This kinds of research makes possible clear findings about the effectiveness of circumscribed, standardized interventions, such as several cited by Kolata, including that one popular math textbook was demonstrably superior to three competitors, and that a highly touted computer-aided math-instruction program had no effect on how much students learned.
The problem is that when it comes to the more complex aspects of change, those that go beyond highly circumscribed units of analysis, randomized trials are of very limited help. For a dramatic illustration, we need only go as far as the inside pages of the same special issue of the Science Times, featuring “Ideas for Improving Science Education,” proposed by nineteen American “scientists, educators, and students with a stake in the answer.” None of these ideas are circumscribed programs or projects. All involve improving education by changing strategies, policies, and mindsets. They include such ideas as helping students to “understand that hard work and persistence are much more important to scientific success than natural ability,” or seeing “what science and math can do when they are used by a creative mind,” or how to stop “over-emphasiz(ing) content when, in fact, it’s context that matters,” or to assure “that creativity and invention became the central focus of STEM courses and that the traditional skills be viewed as what they are: tools to empower creativity.”
Resources invested by large foundations and the Federal government can make a big difference in life outcomes, especially for students that have traditionally not fared well in our education system. When funders ask for evidence of “what works,” they should be demanding a broad range of evidence, and not just the evidence produced by randomized trials. Otherwise important avenues that could profoundly improve results will remain untraveled.
