LSC has posted the comments they have received on the Draft Strategic Plan.
Overall, the tone is positive. Here to me are the highlights, focusing perhaps more on areas of disagreement, rather than the many repeated statements of support for the process and concepts at the most general level.
- SCALAID’s Comment, while generally supportive of outcome metrics, expresses caution about efficiency metrics, worrying about the need to capture broader impacts beyond the individual case. Certainly fair comment, and appropriate in a general expression of support for metrics.
- NLADAs Comments are more cautions. While supportive of best practices, they oppose the development of an outcome measurement system. They, and others, draw attention to the absence of reference in the plan to the prior Performance Criteria.
- Some field programs, such as Midwest Project Directors, oppose even consideration of a national system of performance metrics. That group goes so far as to oppose any system of monetary rewards and penalties.
- In contrast, the Coalition of Connecticut Programs supports the plan, suggesting only that performance metrics take into account particular factors and circumstances. Alan Ells, former project director, also supports those aspects of the plan. South Central Michigan thinks they should be developed in partnership with the field.
- A number of programs urge that the goal of increasing funding should be a more major focus of the Plan (e.g. Legal Aid of Cincinnati). (Since LSC always does this, I saw no need to be explicit, but others obviously disagree.)
- Wayne Moore offers a number of very specific and detailed ways in which tracking of statistics could improve overall effectiveness and impact.
- Jim Greiner and Becky Sandefur (American Bar Foundation) submit powerful comments on the value of research, as planned in the Plan.
- OIG makes a number of planning and management suggestions, including the appointment of a Chief Information Officer.
- Gerry Singsen submits a thoughtful Comment on the overall climate, risks of over-centralization, and the need to focus on system change.
- Linda Rexler, of the Michigan State Bar Foundation, worries that the Plan is too ambitious.
- Legal Services of New York City filed generally supportive comments.
- My own comments, here, focus on a number of what seem to me the most innovative aspects of the plan, and a number of streamlining suggestions, such as the adoption of a timeline, and the use of the regranting process.
What to make of this?
As a general matter, there is broad support for the Plan, with the main focus of worry, as usual, being the outcome measures.
This is not new issue, but, as time goes by, the arguments against such measures become weaker and weaker, and the possible benefits become stronger. Government and other funders simply expect such measures, and their absence is already weakening the access to justice movement, on the Hill and beyond. As research becomes more and more sophisticated, the possible use of such numbers to make significant improvements is harder and harder to ignore. This debate on this has become totally predictable, and the objections can be overcome — as recognized by those whose only concern was that the measures not be too rigid. It is particularly significant that SCLAID takes a generally positive tone.
I think LSC can treat this as a broad endorsement of the Plan.
As I am sure they have always intended, they should make sure that the detail of the outcome measures retains the crucial benefits of general applicability and comparabilty, while having the flexibility that can take circumstances into account. The numbers are the beginning of the debate about effectiveness and efficiency, not the end, but they must be there to have a real debate. We are way beyond the time when we should be debating whether to have numbers.
Go to it.
Richard Zorza's Access to Justice Blog 
The only way we will have an idea of how to improve our delivery system is if we based our decisions on objective metrics, understand the assumptions made in the process of collection and creation of those metrics, and then factor in our experience. Without a core set of metrics (a dashboard if you want an analogy)–we won’t be able to know if what we are doing is benefiting the communities we serve, nor using the scarce resources we raise effectively. We could start with regional variations–take each federal region, and in aggregrate format, start identifying the top quartile and bottom quartile on some metrics (like the ones Wayne More describes):
1. ratio of non/case handling staff to case handling staff
2. ratio of support staff to attorneys handling cases
3. ratio of attorneys per case, paralegals per case (broken down by NMSI code and solution (w/litigation w/out litigation, settlement, etc–by LSC closure case)
4. ratio of attorneys with less than 5 years experience to those with more than 10 years experience
5. ratio of LEP cases closed to non LEP cases closed
6. ratio of disabled clients served to non-disabled clients served
7. ratio of over 80 clients served to under 80 clients served
4. Some of these metrics by poverty penetration (are there disproportionate share programs–meaning programs that are in such heavy poverty areas where that endemic poverty really affects their practice? This concept is applied in Medicare, where dispropiortionate share hospitals get an additional payment–in recognition that DSH hospitals need more resources due to the populations they serve).
5. Also, create an index of legal resources (including LSC, pro bono, and Iolta, plus law school and law library) and see if LSC groups in high index areas have different metrics than LSC programs in lower index areas–in other words, if you are the only legal non profit in your area and there is no other group doing any poverty law work in your state, how does that affect your practice, staffing, metrics, and ultimately allocation of resources.
6. Then also compare these metrics across certain true or false characteristics, like a) program has a hotline,b) program has a strong CMS system that is accesible by all staff across a coverage area, c) program has substantive area strategic meetings at least once year, d) program has a dedicated grant writing staff, e) program has a full time pro bono coordinator–f) loan repayment program or assistance, g) 401K match and e) training program and opportunities and tuition reimbursement. This type of analysis factoring in these true of false factors this will help us understand how these factors affect the general metrics.
Just gathering these very aggregate metrics across Federal Regions and sharing those–would be an amazing source of information–and would start a very positive dialog that could eventually lead to a better understanding of how legal services delivery varies and what creates that variance.
A Response Regarding the LSC Planning Process
The July 14 Access To Justice blog post summarizes comments on LSC’s draft strategic plan, and urges LSC to move ahead (“Go to it”).
I believe, however, that LSC will be a little more flexible and deliberate, and not just “go to it.” LSC has, after all, published a draft plan, and sought public comment. The next step is not to begin the work, but carefully to craft a final plan. I have some confidence that in doing so, LSC will read and use the comments with care and appreciation.
One example: Strategic plans need to capture the essential elements of the current work and the best, most achievable vision of the future. The draft strategic plan does not mention one of the central roles of LSC: to communicate to the White House and Congress not only the devastating impact on poor people of the shortage of civil legal services, but also to communicate to these lawmakers the importance of significantly higher federal funding for legal services. This omission is likely a drafting oversight, but LSC must specifically claim its central role in fighting for more adequate Congressional funding for the Strategic Plan to be a Strategic Plan.
As to the most controversial aspect of the draft plan – the use of specific numerical measures for various purposes — there is significant room for improvement. I don’t find NLADA’s or others comments to be “predictable.” I think they offer LSC an opportunity to re-consider how to move towards numerical measures in a manner that achieves the most useful results. There are two kinds of challenges, as all of us know who have struggled for decades with numerical measures. First, the comments that focus on the details and pitfalls of measurement must be attended to, because it is quite easy to create systems that produce numerical garbage instead of useful results. Second, it is extremely hard to balance between measuring what is valuable, and valuing what is measureable. There is always a significant danger in creating measurement systems that we start to value the advocacy whose impact is simple and are easy to measure, instead of the advocacy whose impact is complex and deep.
If measurement could happen in a securely-funded laboratory environment (a guarantee that legal aid would be fully funded for the next ten years, during which time there is opportunity for experimentation and measurement?) that would be one thing. But there are some in Congress who would take any shred of information – even research that is so flawed as to be meaningless – and use it to argue for the reduction or elimination of legal aid funding. We live in highly charged times, and there is a lot at stake. Before finalizing its Strategic Plan, we count on LSC to carefully weigh the comments, the opportunities, and the risks, and build a Plan that will lead our justice community with strength and maximum impact into the coming years.