I recently posted about some disturbing research strongly suggesting that the results of judicial evaluation surveys reflect cultural biases against women and minorities.

I am glad to be able to report that there is some important work going on to address this problem. Jenny Elek and David Rottman at the National Center for State Courts and Brian L. Cutler at the University of Ontario Institute of Technology have reviewed these surveys against recent research into the use of surveys in work performance evaluation generally, and discovered that the surveys often are inconsistent with recent research into how these surveys can be made more accurate.  For example, they found many instances of double barreled questions such as “Uses common sense and is resourceful in resolving problems that arise during proceedings,” as well as questions that did not refer to direct observations.  Moreover, there were significant problems with sampling and other survey methodologies.  In other words, these are fixable problems.  (The general point is that when a survey is badly designed, people will not be properly focused on the behavior od judges, and will therefore be more likely to rely on their unconscious biases, rather than their knowledge.)

The three will shortly be publishing a paper on their findings, and are also working with a state on constructing and administering surveys that would not suffer from these defects.

This sounds very important, and I hope that states will soon be paying serious attention to this issue.