Jim Greiner at Harvard Law School is deeply committed to what he calls “gold standard” research on access to justice. By this term he means truly randomized studies in which it is possible to compare two otherwise equal groups, the members of one of which receive a service or intervention, and the members of the other do not.
Now available in draft on the Social Science Research Network, is the first of his studies, in this case of help with unemployment benefit hearings provided by the students at the Harvard Legal Aid Bureau.
There is a lot worthy of close attention here.
First, and likely to be the headline, the study did NOT show a statistically significant impact from the use of second year law students to represent unemployment benefit claimants at their hearings (or to be more precise, from the offer to provide representation). It is important to note that the lack of such a showing, while surely disappointing, is not the same as a showing of lack of impact. But the study, which should perhaps be seen as a pilot in the process of learning about the impact of providing different forms of help, does warn us that we can not assume that the impact of help is always massive. It is also important to note that most of these students were handling their first case in the real world, and that the rate of findings for the claimant in the hearings was high overall, making it harder to have an impact. (There are other complex technical issues in the study, which should be read in full.)
Second, the study showed, not surprisingly, that there was a significant delay in the hearings when student-counsel were involved. This makes sense, and should not be forgotten. Programs should think about what processes they might adopt to minimize this impact.
Thirdly, and perhaps most significantly at this stage in the process, the paper reviews a very long list of studies that tried to measure the impact of providing counsel, and finds that with only two exceptions, discussed below, they were all so seriously methodologically flawed as to not having proved their cases. (As a general matter, the flaw was that the populations observed were not randomized. In other words, those with counsel were compared with those without counsel, but the study could not compensate for the fact that those with counsel had the energy and/or resources to get counsel.)
With with respect to the statistically valid studies, quoting from page 67 of the Greiner & Pattanayak paper: “Seron et al. [in a study of New York City Housing Court] concluded that an offer of counsel caused a case to take about a month longer to adjudicate, but that tenants offered lawyers defaulted less often (16% versus 28%), suffered fewer adverse judgments (32% versus 52%) and warrants for eviction (24% versus 44%), and obtained more orders requiring repairs (19% versus 3%) and rent abatement (46% versus 28%).”
The other methodologically rigorous study, of juvenile court delinquency cases, found impact in certain procedural structures and not in others.
This is what I take from all of this right right now, at the beginning of what will surely be a long and valuable process of research and discussion:
- It is very important to do these studies — we are only now really learning how little we know about the impact and structure of counsel interventions.
- We can not assume that all interventions are obviously and massively helpful.
- When we do pilots and research, we must be very specific in defining what is being newly provided, what it is being compared to, and for what population.
- We need to keep an eye on delay issues.
- We must honor those programs that have the courage to open themselves up to such research. They are heroes and what they are doing will immeasurably strengthen all access to justice work. (In an ideal world, funders would provide a bonus grant for such programs.)
I hope we see much more of this work. it is critical to our effectiveness and our ability to make the case for access to justice.
Note that the New York Times Freakonomics Blog is also covering this study.
The link to the paper brings you to the abstract. Sometimes, but not always, there is a “one click download” option at the top for the full text. If that does not appear, try clicking on Greiner’s name, and you should get a list of all his papers on the site, and can click the download button next to this paper.