This blog posting inaugurates a new feature, series blogs. When I think an topic is particularly important, I will blog several times about the topic. Each series will get its own listing in categories, so you can quickly find the whole sequence.
I am starting with a series on outcome measures because they remain somewhat controversial within the access to justice community, and I think it is important to lay out where we are and where we can and should be going.
Let me start by going back to the medical analogy. Those of us who are thinking about treatments and drugs, and potentially life-changing choices constantly worry about things like impact on life expectancy. While we may quibble (with energy) about the importance of designing quality of life outcome measures that value more than just time gained — and indeed I am enrolling in just such a study to validate such measures with respect to my disease — the fact is that we would be lost without that data on outcomes. Imagine if we had to rely on an oncologist saying “Oh, we think that we have found from experience that this drug is better than that,” or “we tell you to hope for the best and prepare for the worst.” (One of my favorite lines from Terms of Endearment is when the Shirley Mclaine character, in response to a doctor saying that that is what he tells all his patients, she shoots back, “And they let you get away with it.”)
But without validated and legitimate outcome measures that’s exactly where we are in the legal system. And, just as it would be hard for the medical system to get much research money if they had not committed to studies based on outcome measures a few years before they they went for the big federal money, we are going to find it very difficult to get innovation and research money unless we can show that we are learning from the investments we make. Indeed, while some would worry that it is unethical to do randomized experiments, I would argue that it may be more unethical to continue to “treat” people’s legal problems if we refuse to use available techniques to get data from which we could make better choices about how to do so.
But even at the substantive level, how can we even argue for innovation if it will not teach us anything? How can we propose simplification, for example, if we have no way of looking at what the impact of any changes are? How can we design incentives for simplification if we have no way of measuring what the benefit is?
How, similarly, can be we make decisions as to who would need and benefit from what services, unless we have a way of measuring the impact on outcomes of different services in different situations? Without outcome measures, any triage systems will be intuitive at best, and surely inconsistent in their application.
Perhaps most importantly in this moment of unqie opportunity, how can we develop and test strategies for 100% access without the “realistic and measurable outcomes” urged by the CCJ/COSCA Resolution.
So we, and by “we” I mean both courts and advocacy organizations, have to get over this hump and design accepted, legitimate, and validated outcome measures.
I would note that one of the hesitations about doing so comes from the continuing confusion about the relationship between what courts need from such measures and what representation/advocacy organizations need. The two sets of needs have to be different, because advocates want to get the best possible result for each of their clients, while courts want to be fair, neutral, and accessible to all. I have come to believe that it is possible to develop an overall system of outcome measures that meets the goal of reflecting these different needs and perspectives, and yet ultimately allows the two kinds of institutions to be assessed as an integrated whole.
Stay tuned.
Richard Zorza's Access to Justice Blog 
I don’t think either the courts nor the advocacy groups have the right perspectives for appropriate outcome measures. Instead, they should reflect the views of the litigants themselves. That is the only legitimate perspective.
Pingback: Outcome Measures #2: LSC Outcomes Measures, Good News, Bad News, and A Challenge | Richard Zorza's Access to Justice Blog
Hi Richard – I agree as well. You raise an important issue (or maybe obstacle is a better word) in your last paragraph. In my experience it has been difficult to get stakeholders to agree on what outcomes to measure. There are many options – economic, social, service efficiency, satisfaction, and on and on. There is also a fear of the unknown – will outcomes tell people what they do not want to hear. From the development perspective, we are often pulled into ‘less-than-innovative’ outcome measurement because donors and policy makers want easily digestible measures (eg. how many people served). But having a more beneficial measure of outcomes is important for the reasons you mention. In the development world, I need to explain to decision-makers why we should invest in justice as opposed to sectors such as health, education or infrastructure. Which as you can imagine is not an easy sell.
Another issue is how to measure outcomes. Again, quite a few options. I’d been experimenting with an impact evaluation of legal aid services on poor women in Jordan. The internal resources here at the Bank basically forced an RCT model. But the lack of flexibility around how to randomize subjects and the projected costs, among other things, made me uneasy and the initiative has been canceled.
I am now launching a research initiative on the links between unresolved legal problems and poverty. It’s the pilot phase of data collection in four upper middle income countries – Jordan, Mexico, Colombia and Peru – that involves collection of existing household survey and administrative data on the types of legal problems most affecting the poor (though we can more or less guess what most of these will be). The subsequent challenge is refining the methodology to allow for measuring poverty-related impacts. The next phase of the research will involve surveys of households affected by certain types of legal problems to measure poverty impacts through a variety of to-be-determined indicators, which hopefully include coverage not only of human development (health and education) and economic (income, consumption, access to assets, financial security, labor force participation) factors, but also issues around social factors and agency (eg. opportunity). Hopefully, this will add to the general discussions of outcomes.
Richard, Great post! We would supplement your discussion of outcome measures a brief mention of the call for “indicators” on access to justice that is set forth in the recently adopted United Nations Sustainable Development Goals, specifically Global Goal 16. The UN adopted the Global Goals in September 2015, along with Global Targets to help implement them. The UN is currently working to design global indicators. As part of the process, the UN has called on all countries to develop “national indicators”. See Resolution 70/1 (October 21, 2015)
http://www.un.org/ga/search/view_doc.asp?symbol=A/RES/70/1&Lang=E
In the US, the Obama administration recently issued a Presidential Memorandum charging the White House Legal Aid Interagency Roundtable (LAIR) and the US Dept of Justice’s Office on Access to Justice with responsibility for implementing Global Goal 16.
https://www.whitehouse.gov/the-press-office/2015/09/24/presidential-memorandum-establishment-white-house-legal-aid-interagency.
Federal agencies will be expected to contribute ideas and data to the US government’s efforts to develop national indicators for access to justice.
The adoption of Global Goal 16, the issuance of the Presidential Memorandum, and the earlier issuance of the CCJ/COSCA 100%
resolution, create a unique opportunity for establishing national access to justice indicators to track progress to increase access to justice in the US.
In selecting indicators, substantial work will need to be done. Existing indexing systems, with their respective indicator sets, may offer a place to start. See e.g., the Justice Index, http://www.justiceindex.org, created by the National Center for Access to Justice at Cardozo Law School (“NCAJ”), and the WJP Rule of Law Index, http://worldjusticeproject.org/rule-of-law-index, created by the World Justice Project. Another source to draw on is the toolkit posted by the Legal Services Corporation to guide LSC recipient programs toward adopting indicators to measure outcomes. http://clo.lsc.gov/home/.
The Columbia Law School Human Rights Institute and the National Center for Access to Justice at Cardozo Law School are coordinating an Expert Working Group that will be meeting initially this week to begin a process of developing recommendations to submit to the US government on national indicators on access to justice for the US. For more information see our past posts, http://ncforaj.org/2015/10/02/us-govt-takes-step-toward-collecting-more-data-to-measure-access-to-justice/ and http://ncforaj.org/2015/08/03/the-global-access-to-justice-goal/
We would welcome inquiries by people interested in learning more about Global Goal 16 and the process to establish national indicators for measuring access to justice. David Udell, National Center for Access to Justice, udell@yu.edu; Risa Kaufman, Columbia Law School Human Rights Institute, Risa.kaufman@law.columbia.edu.
Hi, Richard, perhaps it’s not a surprise that I’m 100% with you on this one. I had a chance to post something about how a discussion of measuring outcomes begins and ends with hard philosophical questions, here (http://harvardlpr.com/2015/03/23/whats-access-to-justice-for-lets-get-more-philosophical-in-a-hurry/) and here here (http://harvardlpr.com/2015/03/27/whats-access-to-justice-for-lets-get-more-philosophical-in-a-hurry-part-ii/). Rarely are the hard questions in this area statistical. Instead, they consist of a focus on why we’re doing what we’re doing. At least in my view, that’s often the hardest question of all to answer.