The following are articles and speeches by Lisbeth Schorr drawing on lessons learned over the last several decades from research, theory, and practice about multiple efforts to solve social problems, new opportunities to act on what we now know, and new approaches to assessing “what works”.
The Stanford Social Innovations Review posted my blog post of January 8, 2016, Reconsidering Evidence: What It Means and How We Use It –The tide that has swept experimental program evaluation to the forefront of knowledge building about social policy is suddenly ebbing. . Here I am exploring the possibility that experimental program evaluation may be receding as the prime way to build knowledge about social policy.
“To Achieve Big Results from Social Policy, Add Continuous Real-time Learning to After-the-Fact Program Evaluations” Huffington Post Politics, Anthony Bryk and Lisbeth Schorr.
“An Evidence Framework to Improve Results”, 2014 Harold Richman Public Policy Symposium: The Future of Evidence. Lisbeth Schorr and Frank Farrow with Joshua Sparrow – The question addressed in thsi paper is not whether we need evidence, or even whether we need more evidence. The question we explore here is, How do we use a framework of continuous learning to obtain and apply the kinds of evidence that will be most useful in achieving significantly greater outcomes?
Keynote Address,Community Action Program Legal Services (CAPLAW), National Training Conference, Boston, June 19, 2013 – a discussion of the challenges of documenting accomplishments and creating a culture of continuous learning and improvement.
“Keynote Address, NeighborWorks Symposium”, December 12, 2012 – An exploration of why we need a broader definition of evidence in designing, understanding, and evaluating efforts to solve the pressing problems facing children, families, and communities in our country today.
“Broader Evidence for Bigger Impact” Stanford Social Innovation Review”, Fall 2012 – This article calls for the reconciliation of competing approaches to the kinds of evidence that funders, policy makers, and program designers should consider credible. It advocates using a range of methods to obtain proof of impact, to establish accountability, to assist funders in allocating resources to what works, to encourage the development of innovative interventions, and to achieve defined results. The combined insights of the Experimentalists and the Inclusionists would enable hardworking practitioners throughout the country to achieve the breakthrough impacts so urgently needed by the children, families, and communities now at the margins or American society.
“Risks Worth Taking” – When philanthropists opt for the low-risk strategy of funding only proven programs, they may be trading off the chances of realizing more powerful results from interventions that have a high probability, but less certainty, of effectiveness.
“Expanding the Evidence Universe: Doing Better by Knowing More” Lisbeth B. Schorr and Frank Farrow, December 2011
“Expanding the Evidence Universe: Doing Better By Knowing More,” Executive Summary – Thanks to the last two decades of research and experience, we now know so much more than ever before about what it takes to improve outcomes for disadvantaged children and families — whether measured in healthy births; school readiness; school achievement; robust physical, and mental health; or neighborhood support and safety. But our expanded knowledge has not led to better outcomes at a magnitude that matches the need. Among the reasons: we have too often failed to marshal the full extent of available evidence and to generate new, real-time knowledge from experience. The paper explores how we might make use of all the evidence we now have from multiple sources, including research, theory, practice, as well as the findings from program evaluations, and how we might aggressively gather new evidence about the nuanced and powerful strategies for change that are currently emerging.
“Integrating Evidence and Practice to Reduce Disparities: Developing an Inclusive Framework for Effective Mental Health Interventions,” Keynote Address to the Conference of the National Alliance of Multi-Ethnic Behavioral Health Associations, May 11, 2010 – Current experience around the country suggests that complex efforts to improve outcomes for disadvantaged children and families are best begun with a focus on results. When communities agree on outcomes and results for children and families, they can encourage innovation and local problem-solving that responds to unique conditions and cultures, by replacing rigid regulation of inputs with rigorous accountability for accomplishments. A results frame makes it easier to bridge diverse constituencies and points of view, to support collaboration across boundaries, to mobilize and sustain efforts over time, and to map backwards from agreed-upon results to determine what collection of interventions and supports are most likely to achieve the results. New approaches to knowledge building do not reject — on grounds of messiness or complexity — information that can shed light on real-world efforts that promise to improve outcomes. Non-experimental evaluations must insist on rigor even in the absence of certainty or proof, in the belief that credible evidence of effectiveness can be attached to promising interventions if they are found to have a basis in strong theory; a converging accumulation of empirical evidence of effectiveness from similar or related efforts that may not rise to the level of causal proof; consensus among informed experts based on a combination of theory, research, and practice-based evidence.
“Innovative Reforms Require Innovative Scorekeeping,” Education Week, August 26, 2009 – Serious social reformers today agree that rigorous efforts to determine “what works” are essential. But if these efforts are not to sabotage or marginalize the most innovative attempts to solve intractable social problems, funders must develop more inclusive ways of establishing effectiveness and accountability. If government agencies and private grant makers support only those interventions that are shown effective by experimental methods, we will be robbed of (1) good programs that do not lend themselves to experimental evaluations, (2) reforms that are deeper and wider than individual programs, and (3) innovations of all kinds.
“To Judge What Will Best Help Society’s Neediest, Let’s Use a Broad Array of Evaluation Techniques,” The Chronicle of Philanthropy, August 20, 2009
Mismatch between Complex Interventions and “Gold Standard” Evaluations, .Chart, 2009 – Comparison of the attributes of effective complex interventions and the attributes associated with “gold standard” evaluations.
“A Lot to Lose: A Call to Rethink What Constitutes ‘Evidence’ in Finding Social Interventions that Work,” January 2009 – Malcolm Wiener Center for Social Policy, Harvard Kennedy School, Working Paper Series (Co-authored with Katya Fels Smyth) – The authors describe the characteristics of What It Takes organizations, which their work suggests support lasting change in the lives of highly marginalized and vulnerable people. They identify the risks inherent in the continued privileging of experimental designs over all others, and suggest that the risks are heightened in periods of great economic stress, when the pressure for accountability is increased.
“The O’Connor Project: Intervening Early to Eliminate the Need for Racial Preferences in Higher Education,” Judicature, September 2004, Volume 88, Number 2 September-October 2004.
“The O’Connor Project, Can We End Racial Discrimination Without Affirmative Action? Here’s What it Will Take,” The American Prospect, January 2004– In her opinion in Grutter v. Bollinger, Associate Justice Sandra Day O’Connor declared her expectation that racial preferences in higher education will no longer be necessary 25 years from now. These articles lay out what we can and must do with present knowledge to reduce or eliminate racial disparities early in life and thereby eliminate the need for racial preferences at the university level.
“From Knowledge Management to Knowledge Building: An Essential Foundation Journey,” New Directions For Philanthropic Fundraising, Fall 2004, No. 45, © Wiley Periodicals, Inc. – By taking a bold, generous, and inclusive approach to knowledge building, foundations can equip community organizations, service providers, and policymakers at every level with actionable information that will enable them to more effectively achieve their objectives.
“Usable Information About What Works: Building A Broader and Deeper Knowledge Base,” Journal of Policy Analysis and Management, Fall 2003, Vol. 22, No. 4. (Co-authored with Patricia Auspos) – The authors contend that to foster and support effective strategies that will actually improve lives, the way we think about and process knowledge in the social policy world must change:
- Prevailing conventions about what counts as credible knowledge must be reexamined and modified.
- Our focus must change from making yes-or-no judgments about individual interventions to discerning patterns from an accumulation of research and experiences.
- Our knowledge-building activities must cross systems and disciplinary boundaries, even if service delivery and funding continue to operate predominantly within self-contained silos.
“Determining “What Works” in Social Programs and Social Policies:Toward a More Inclusive Knowledge Base,” Children’s Roundtable, The Brookings Institution, February 2003 – Lisbeth Schorr argues that rigorous research on impacts is needed but is not sufficient. Too many programs are multidimensional, cannot or should not be standardized, evolve or adapt through time, require participants’ active involvement, or are dependent for success on good implementation, not just good design. Schorr argues for more flexible forms of evaluation that require experts and practitioners to hypothesize the linkages between actions and outcomes, identify interim indicators of success, and pay more attention to the attributes of programs and the institutional contexts that are essential to success. While recognizing the inevitable trade-off between knowing a few things very well and more things with less certainty, Schorr argues for greater efforts to understand broad patterns that connect activities to results, even if this means compromising the search for absolute truths.