Note: This our first “NewsMaker Interview,” designed to go under the surface of important developments in access to justice.  Readers of this blog are encouraged to suggest future interviewees and topics.

Jim Greiner and Cassandra Wolos Pattanayak’s draft article, What Difference Representation?, has been the subject of considerable attention and debate, including in the New York Times Freakonomics blog.  My prior post is here.

So the AccessToJustice Blog thought it might be useful to explore some of the implications of this study of the outcome and time impact of offers to provide student representation by Harvard Legal Aid Bureau students at unemployment hearings in Massachusetts.  (Disclosure: I actually used to help people at these hearings before going to law school.  This is authorized by the Mass statute.)

Jim has generously agreed to answer our questions.  He will also monitor and respond to comments posted to this blog entry.  Thanks, Jim.

This is the first of what I hope will be a continuing series of such Newsmaker interviews, hopefully getting a bit more in depth behind the headlines of access to justice.

Q.        Jim, can you tell us first why you got into this area of work, and why you did this study.

I’ve been interested in the delivery of legal services since picking up pro bono cases during my 6-year stint as a litigator in D.C.  I believed then, and continue to believe, that lawyers must provide critical access to justice services to economically less fortunate clients.  I wondered even then, though, whether we lawyers knew enough about how much we were helping people, and whether we were doing so efficiently.  Efficiency seemed important given the vast demand for free or low-cost legal services.  Supply of free or low-cost legal services seemed totally inadequate to meet the demand.  After I got bitten by the quantitative bug, I decided to pursue this research.

Q:        Jim, a lot of people seem to feel that your study said that you showed that representation did not help unemployment claimants.  Is this right?  Could you tell us what your study did and did not show?

Our study did not show that an offer of representation from the Harvard Legal Aid Bureau (“HLAB”), the student group that participated in this study, caused no increase in the probability that a unemployment claimant would receive benefits.  No study could be so definite; all results from studies of this kind are limited by statistical uncertainty.  The data did suggest, though, that if there was a positive effect for claimants, that effect was probably not very large. 

Q.        So, what you are saying is that there might have been an impact on hearing outcomes, but that this study did not pick it up.  Can you help us understand from the numbers what kind of effect might have been missed, and what is the chance that there might be some effect that you have missed.

If one is willing to tolerate a little statistical modeling (as we are), then the true increase due to an offer of HLAB representation in the probability that a claimant would prevail was 95% likely to be between -.06 and .09.  (Negative numbers correspond to a decrease in the probability of a win.)  Numbers near the middle of this interval (.01, or .02, say) much more likely to be true than numbers near the extremes (such as -.05 or .08).

In other words, suppose that a claimant without an offer of HLAB representation has about a .65 probability of getting benefits (which is about what we observed in our study).  Then with an offer of HLAB representation, it’s very likely that this same claimant’s probability of getting benefits is somewhere between .59 and .74.  But the most likely probabilities are somewhere around .66, .67, in that range.  In other words, it’s pretty likely that there is essentially no change due to an HLAB offer in the probability that a claimant will prevail.

For me, perhaps the most important point here NOT whether true increase due to an HLAB offer in the probability of a claimant victory is .01 or .05 or a negative number.  For me, the key point is that we went into this study thinking that an offer of HLAB representation would increase the probability of a claimant victory by a minimum of .15 or more.  In my hypothetical, we thought a claimant that had a probability of winning of around .65 without an HLAB offer of representation would have a probability of winning of around .80 or .85 with one.  But it turns out that such a large increase is highly unlikely to be true; if an HLAB offer had made that much of a difference, we probably would have seen it in the data, and we did not.

I’m speaking non-technically here, and my friends in the statistical community will get after me about some of the imprecise language I’m using, but that’s the basic idea.

Q.        Some people have asked why you focused on offers of representation, rather than on actual use of representation.  Could you explain?

First, note that we did try to tease out a causal effect of actual use of representation.  The data didn’t let us say very much here, though.  But you’re right to say that we focused more on the offer.

Why we focused on offers is hard for we lawyers to grasp at first, but it’s one of the most important things I learned in my statistical training.  Here’s an analogy that I find helpful in understanding:  Suppose that laboratory experiments demonstrate that a chemical kills a certain kind of intestinal parasite, in the sense that when the chemical is placed in a petri dish with the parasite’s cells, those cells die at high rates.  In a petri dish, the chemical is really, really effective, just wipes out the parasites.  Should the FDA approve, on the basis of such tests, a drug company’s proposal to market an oral pill containing the chemical to the public?  Obviously not.  Among other things, we do not yet know what will happen when a patient eats the chemical.  Perhaps the chemical metabolizes into something else in the stomach before it can reach the intestine, and that something else might be ineffective in killing the parasite or even harmful to the patient.  Perhaps to be effective the chemical must be introduced directly into the intestine.  Or perhaps the drug company, try as it might, cannot formulate an effective way to deliver the chemical.  The point is that even if the chemical in some sense “works” on parasitic cells, the mechanism by which the chemical is delivered is critical to deciding whether it can form the basis of an effective treatment.  If we want to evaluate the effectiveness of a real-world treatment, which is surely a highly relevant (perhaps the most important) question for a policy maker deciding funding, then we cannot look only or even primarily to how the chemical behaves in the lab.  We must instead look to how a treatment (including a delivery mechanism) works in the real world.

Here, “actual representation” is the chemical.  That’s what we want to deliver to the patient/client.  But in this country, we deliver actual representation by offering to do so.  Offers are the means of delivery.  In the US, we don’t force people to accept an offer of representation from a particular service provider; and if a person is turned down by a particular service provider, we don’t prohibit that person from finding representation elsewhere.  All a service provider can do is offer.

In evaluating the effectiveness of any program designed to deliver a “treatment,” you have to include the delivery mechanism in the mix..

To be clear:  again, we are also interested in the effect of actual use of representation.  We think a person designing or reforming an adjudicatory system would want to know whether actual use of representation makes a difference because that may say something about how pro se accessible the system is.  But, when we’re thinking about that, we have to remember that we cannot randomize who actually gets representation.  Potential clients have to decide on their own whether to accept an offer, and they can do as they wish if they don’t receive an offer.  All we can randomize is who gets offered representation.  So we have to be really, really careful.

Q.        And, some people feel that you should have compared those who actually received assistance with those who did not.  Can you explain?

Again, we can’t randomize who actually gets assistance; we can only randomize who gets an offer.

When you’re trying to figure out the effect of something, you need to compare apples to apples, meaning you need to compare two groups of claimants who are the same (except for statistical variation) in all ways except for that something.  Otherwise, you can’t tell whether it is the something that is really making the difference.  Randomization is the only way to make sure two groups are basically the same.

Think of it this way:  suppose I’m trying to figure whether representation makes a difference in unemployment litigation.  I divide a group of unemployment claimants into two groups.  In Group A, I put all of the articulate, persistent, and organized claimants who speak English fluently and who have dealt with official bureaucracies a lot.  In Group B, I put in all of the inarticulate, less persistent, disorganized claimants, many of whom can’t speak English and who hate forms.  I get all of the Group A but none of the Group B people lawyers.  I then follow the results for both groups and find, lo and behold, that Group A succeeds more often than Group B.  Have I proved that lawyers made a difference?  I don’t think so.  It seem likely to me that what’s going on is that articulate, persistent, and organized claimants are more likely to have better facts in their cases, better able to help their lawyers, make better witnesses, and are more able to find witnesses and documents on their own.  But all of those skills also suggest that these Group A folks might have done OK without a lawyer.  Meanwhile, the Group B folks, due to lack of articulateness and persistence and personal organization, etc., are probably both less like to help lawyers (if they’d had them) with their cases as well as less likely to win without lawyers.  In short, Groups A and B are very different in ways other than just having a lawyer, so we can’t tell whether any change in likelihood of winning is due to the lawyers or due to these background differences.

People who obtain legal representation may not be (are probably not, really) the same as people who don’t.  Getting a lawyer, including a legal aid lawyer, is not an automatic process.  It takes some work.  A potential client has to find a legal aid office that takes the kind of case, get through to the office when it’s open, explain the problem that she’s having, fill out information and consent forms, etc.  Folks that are able to do this are probably more persistent, articulate, etc., than those who don’t or can’t.  So if we just compare the outcomes for folks who had lawyers to the outcomes for folks who don’t, we’d probably have the Group A versus Group B I just explained.

Q.        Do you think we can generalize to other counsel situations?  What might be similar or different?

It’s not yet clear how generalizable our results are.  We need more study.  Right now, we can’t even tell whether we can generalize these results to other service providers who work within the Massachusetts unemployment system.  It could be that the kind of person who calls HLAB is a particularly articulate, persistent, and organized kind of person who doesn’t need legal representation to succeed in unemployment litigation, whereas folks who call other service providers have less in the way of these skills.  We don’t know without further study.  We also do not know whether the use of second year students appearing in their first hearing – no matter how well trained and dedicated – is or is not a typical intervention, although in this particular study, the quality of the law students involved was exceptionally high, the oversight they received was unusually strong, and the law student’s work was respected among the adjudicators in the Massachusetts unemployment system.

On the other hand, these findings have us worried.  What if it is true that there are certain settings in which legal assistance doesn’t help very much?  What if the legal services community currently has a bunch of resources devoted to providing representation in settings where representation doesn’t matter much?  (For what it’s worth, there are at least two other major legal services providers that offer representation in unemployment litigation in the Boston area.  Both have suffered budget issues in the current recession.)

Note that IF it turns out that few if any forms of representation are helpful for unemployment claimants, that might mean that the unemployment system really is open enough to pro se litigants to let them succeed about as well as those with lawyers do.  If it turns out that this is true, now might ask, how did the system get so pro se friendly?  My guess is that legal services providers had a big role in molding the system so as to make it accessible to folks without lawyers; the designers and implementers of the adjudicatory system probably had to work hard as well.  If so, all involved deserve a great deal of credit.

Q.        Could it be that the fact that lots of the people you turned down in fact got help from other sources made the HLAB representation look less effective?

We evaluated the need for and effectiveness of HLAB’s program.  If there’s lots of representation available from sources other than a particular legal services provider (here, HLAB), shouldn’t that fact be important in deciding whether to fund or continue offering legal assistance in this areas?  The argument might be that we have too many needs in various places that aren’t being met to devote resources to areas in which representation is already available to many.

Anyway, the fact is that about 92% of the folks offered HLAB assistance ended up represented, whereas only 49% of the folks NOT offered HLAB representation ended up represented.  That’s a decent-sized difference; probably enough to let us see a change in the win rate due to an offer of representation had that change been sizeable.  Apparently the win rate difference was not all that large.

Q.        You did find a statistically significant delay for those cases in which the claimants were offered student counsel.  You also suggested that this meant that state money was being wasted by the delay, in the small number of cases in which the grant of benefits was ultimately overturned after a delayed hearing.   How significant do you think this is, and does it matter that much?

It’s hard to tell because the Massachusetts unemployment agency couldn’t tell us how much of the money erroneously paid to claimants it recovers.  Note that to recover funds, the agency can deduct the amount overpaid from income tax refunds as well as from any future unemployment benefits, but the claimant can ask that the recoupment be waived, based on need.

If you ignore the recovery issue, our numbers here were fairly rough.  We came up with something like $2000 per case, but there was a wide range, and didn’t have a whole lot of data to go on.  So that’s a pretty back-of-the-envelope calculation. Note that when I said $2000 “per case,” that means “per case in which there was an initial grant of benefits, where the initial grant was overturned on appeal.”  We don’t know how many of these cases there are in the system as a whole, so we can’t tell how much money is involved here.

Q.        You reviewed lots of studies for your paper.  What did you find, and what are the implications for future studies?

As far as we can tell, in the history of the United States, there have been only two other randomized studies done to measure the effect of representation in civil cases.  One was done in NY housing court in the early 1990s, and it showed that counsel made a big difference.  The other was actually two separate studies in juvenile delinquency.  One of those juvenile delinquency studies showed that counsel didn’t help at all; the other one showed that counsel made a medium-sized difference.

As for the dozens upon dozens of non-randomized studies that tried to measure how much of a difference a representation makes, we don’t believe any of them (which is different from saying we know that they’re wrong!).  Researchers don’t know enough about how people get lawyers to be able to use statistical techniques, such as regression, to adjust for differences between the kind of person who gets a lawyer and the kind of person who does not.  That’s what you’d have to do to make a non-randomized study work, and we don’t think such adjustment is possible at present.

As far as we’re concerned, for future studies, the only real way to learn anything is via randomization.

Q.        What about the ethics of this?  Is there any problem with lawyers (who naturally think that our work is helpful to our clients) being part of studies that actually deny some people our help?

In almost all cases where there’s more demand than a legal services provider has capacity (meaning when more people want help than the provider can serve), there’s absolutely no problem with randomizing.  In that situation, the provider IS going to turn some folks down, no question about it.  The only question is how to decide who gets turned down.  And it seems like using a coin flip (or a roll of the die or whatever) to decide who gets representation is a fair way to allocate a scarce resource.  Again, in this situation, a study (randomized or otherwise) isn’t the reason that the legal services provider will “actually deny some people our help.”  That will happen, study or no study.  The only question is whether we use the unfortunate facts of too many people and not enough help to figure out how to make our help more efficient and effective.

Further, the whole debate about whether it’s ethical for lawyers to participate in studies that “deny some people our help” assumes that our help is good for them.  What if it isn’t?  What if it hurts them, even a little (which is what we found in the HLAB study, as a result of the delay)?  Shouldn’t we find this out?  And at least at present, randomized studies are the only way to do so.

Q.        I understand that you are working on other studies of the impact of counsel.  Could you tell us about those?

We’re participating in one study in the housing area that looks at the effect of an offer of representation to defendants (tenants, usually) in summary eviction proceedings in a court of (more or less) general jurisdiction.  In that study, it’s looking as though few folks randomized to receive no offer of representation are able to find any form of assistance elsewhere.

We’re participating in a second housing/eviction study in which “treated” defendants (again, tenants, usually) are given an offer of a full attorney-client relationship, but “control” defendants are referred to a lawyer-for-the-day program that provides representation only on court dates.  Thus, this study is not a lawyer versus no-lawyer study but rather a test of whether we can deliver legal services via a lawyer-for-the-day program without losing too much in the way of effectiveness.  If it turns out that a lawyer-for-the-day program is equally or nearly as effective as full attorney-client relationship, that will be big news.  Lawyer-for-the-day programs are a lot less expensive than portfolios of full attorney-client relationships, and they can serve a lot more people.

We have a study of the effectiveness of another student-based group, this one in the area of Social Security disability programs (SSI and SSDI).  The adjudicatory process is here is administrative as well, but there can be medical evidence and quasi-expert testimony involved.

Finally, we’re just getting started with a study of pro bono assistance in divorce cases.

Q.        Are you looking for other research partners?  If so what kinds of situation would you like to look into, and what should people do to explore this further with you?

Yes, we’re actively looking for research partners.  We want to know in what sort of situations lawyers make a big difference as well as where lawyers don’t really change much.  The only way we’re going to figure that out is with lots of studies in lots of different legal areas, with different adjudicators, in different states, etc.  We also want to show people that this sort of research doesn’t have to be expensive, nor does it require that legal services providers do much that they aren’t doing already.

Anyone interested should please email or call me.  Please!  My contact information is available here:  http://www.law.harvard.edu/faculty/directory/index.html?id=705.

Q.        So, what is your biggest takeaway from the research so far, and what do you hope for the future?

The biggest takeaway is that in some legal areas, with respect to some client bases, and before some adjudicators, an offer of legal representation probably does not make a very big difference.  In this set of circumstances, we think legal services providers working in these areas should either change how they do business (by, for example, attempting to reach a different potential client base) or shift the resources previously devoted to this subject area to another one where legal representation really does make a difference.

Unfortunately, because we’ve done a grand total of three (3!) randomized studies in this country on the effect of representation, we don’t yet know which legal areas, which client bases, and which adjudicators make lawyers more or less critical.  So the biggest takeaway message is:  we need a lot more randomized studies!

Jim, thanks so much for this work and for your help.  I too believe that this is critically important work.

This blog encourages comments.  Jim will try to respond to your thoughts and questions.