As state start to think about evaluating how their judges do in cases without lawyers, this might be a useful starting point. It is US Department of Labor Handbook on measuring hearing quality in unemployment hearings (in which the vast majority of claimants, and many employers do not have representation). The link is here.
At least initially, such evaluation might well be a general statewide self-evaluation, rather than any attempt to get a handle on individuals.
By breaking down specific kinds of behavior and hearing attributes which might be assessed, the Handbook might offer a useful beginning.
Here are the areas for which scoring systems are recommended:
For example, here is the scoring system for the administrative judge explanation of the process (#1):
CRITERION 1: PRE-HEARING/PRE-TESTIMONY EXPLANATION.
PURPOSE – At the start of the hearing, the hearing officer should clearly explain the procedures to be followed. The elements shall be covered in the recorded prehearing explanation or opening statement. The explanation must be clearly stated and delivered in an understandable manner.
SCORING SEGMENT
Good (6): After recording began and before testimony was taken, the hearing officer clearly explained the hearing procedures. This explanation included: (a) the order of testimony, (b) the right to question witnesses, and (c) an opportunity for each of the parties to ask questions about the hearing process or procedures.
Fair (3): The hearing officer allowed an opportunity to ask questions about the hearing process or procedures, but did not explain all of the elements (a) through (c). Unsatisfactory (0): The hearing officer did not explain the procedures or did not allow an opportunity to ask questions about the hearing process or procedures.
While, obviously, this whole issue is very very sensitive indeed, we have to recognize that cases without lawyers provide particular quality challenges. When there are two lawyers in the room, it is a reasonable bet that a) the judge will be aware of the possible consequences of sub-par performance, and b) that serious problems will ultimately be bought to the relevant authority. However, when there are no lawyers, things may be different. (I remember appealing a parking ticket in San Francsico in the 1970s, and hearing the hearing officer, a retired cop, comment to a different appellant that, “I do not believe the officer would have issued a ticket if you had done nothing wrong.” So much for neutral judging.
So, we need some method of quality control, and a system like this, in place in all states through US government standard setting for unemployment hearings, is a good place to start the discussion.
Richard Zorza's Access to Justice Blog 
Richard, so great to see attention to this DOL Handbook. It really is a remarkable resource. I’m a co-researcher in a study based in a DC unemployment insurance court where this handbook was an important part of our findings about active judging in cases with self-represented litigants. My colleagues in the study are Colleen Shanahan and Alyx Mark. In a forthcoming article, Active Judging and Access to Justice, I discuss the handbook, its role in unemployment appeals, and its influence on active judging, at length. I include significant detail about the DOL quarterly review process, where judges grade one another’s hearings and cases based on the handbook guidelines. Interestingly, I find that all of the judges in the sample are “active” in some way, but there is also a lot of variation in the nature of views and practices across the judges. The old saying, “the devil is in the details,” definitely applies here. The judges have areas of similarity (explanations about basic process at the beginning of a hearing, for example), but in other areas, such as explaining the law and eliciting facts, active judging practices are quite different across judges. I ultimately suggest that consistency in active judging may require more substantial guidance, or perhaps stronger accountability mechanisms, than even the handbook provides. Clearly, there’s much more work to be done to understand the role of guidance and accountability mechanisms when it comes to judges and SRLs. Thanks for drawing attention to this important document.
Link to the article is here: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2911198