As state start to think about evaluating how their judges do in cases without lawyers, this might be a useful starting point. It is US Department of Labor Handbook on measuring hearing quality in unemployment hearings (in which the vast majority of claimants, and many employers do not have representation). The link is here.
At least initially, such evaluation might well be a general statewide self-evaluation, rather than any attempt to get a handle on individuals.
By breaking down specific kinds of behavior and hearing attributes which might be assessed, the Handbook might offer a useful beginning.
Here are the areas for which scoring systems are recommended:
For example, here is the scoring system for the administrative judge explanation of the process (#1):
CRITERION 1: PRE-HEARING/PRE-TESTIMONY EXPLANATION.
PURPOSE – At the start of the hearing, the hearing officer should clearly explain the procedures to be followed. The elements shall be covered in the recorded prehearing explanation or opening statement. The explanation must be clearly stated and delivered in an understandable manner.
Good (6): After recording began and before testimony was taken, the hearing officer clearly explained the hearing procedures. This explanation included: (a) the order of testimony, (b) the right to question witnesses, and (c) an opportunity for each of the parties to ask questions about the hearing process or procedures.
Fair (3): The hearing officer allowed an opportunity to ask questions about the hearing process or procedures, but did not explain all of the elements (a) through (c). Unsatisfactory (0): The hearing officer did not explain the procedures or did not allow an opportunity to ask questions about the hearing process or procedures.
While, obviously, this whole issue is very very sensitive indeed, we have to recognize that cases without lawyers provide particular quality challenges. When there are two lawyers in the room, it is a reasonable bet that a) the judge will be aware of the possible consequences of sub-par performance, and b) that serious problems will ultimately be bought to the relevant authority. However, when there are no lawyers, things may be different. (I remember appealing a parking ticket in San Francsico in the 1970s, and hearing the hearing officer, a retired cop, comment to a different appellant that, “I do not believe the officer would have issued a ticket if you had done nothing wrong.” So much for neutral judging.
So, we need some method of quality control, and a system like this, in place in all states through US government standard setting for unemployment hearings, is a good place to start the discussion.