This is a modified version of my post yesterday, Avoiding the Shut Down Effect From Uncertain Research Results in the online Symposium conducted by the Concurring Opinions blog.
That blog is about some research, previously highlighted on this blog, into the impact on offers of legal assistance by Harvard Law students in unemployment cases. The purpose of this post is to explore the implications of the fear that negative research findings will be used to reduce resources for access to justice.
While it would be impossible to rebut the claim that this might happen, that can not be an argument against conducting or reporting research. On the contrary, it has to be an argument for more and better research. Having been involved in access to justice for decades, I am all too aware that it never seems to be the right time to make ourselves vulnerable — so the answer has to be that the right time is always, because that is the only way to gain credibility.
But this does all raise the question as to how research should be structured, analyzed, and particularly, reported in order to minimize the risk of results inconsistent with the ultimate findings of the research, in all their complexity and subtlety.
- Generality of Reporting. The headings, abstract, etc, must be structured to accurately reflect the generality of the research
- The Reporting of Context. Studies should be very careful about describing accurately the context in which treatment is provided.
- Randomization/Observation. Where studies are not randomized, that must be very clearly reported, and selection bias must be loudly proclaimed, not the subject of a footnote.
- Explanation of Statistical Significance. The issues of statistical significance must, to the extent possible, be explained as clearly as possible, in lay terms. The failure to do so, when it occurs, makes both overstatement an unfair critique easier.
- Lay Version. Research should be made available in a lay summary version, without complexities but with the detail and cautions. This will reduce the risk of the results being oversimplified by the media and/or others
- Vigilance as to Over/Under Generalization. The text should not only be accurate as to the level of generality of the research, but should be explicit as to the kinds of generalizations that might erroneously be drawn from the research. (This makes it easier to rebut overdrawn conclusions made by legal or political opponents.)
I would very much appreciate additions to such best practices. Indeed, we might even think about formally promulgating best practices. There are surely many more in the social sciences. Those suggested here, however, are less about avoiding error, and more about taking steps to minimize the risk of error in the reporting or use by others.