mvnForum Homepage

Print at May 20, 2019 9:28:51 PM

Posted by Squashbuckle at Jul 18, 2008 10:23:36 AM
Re: When Judges Go Wrong...
Having run a wide variety of judged events, I think I've run into just about every judging snafu you could imagine. In addition to your issues, here are some common problems and how I solve them:

1. The judging rubric may unintentionally set things up so the best entry doesn't get the highest score. I've run into this several times, especially in art contests. In one case, for instance, I had a category for the piratey details to give what seemed like a deserved bonus for clever touches. In the end, though, an extra few points for an added BK trinket in the background that didn't actually change the mood or skill of the piece bumped some people above others who had much more artistic skill and much more clever entries overall.

If this happens to you, I would strongly suggest throwing out any categories that seem to give bonuses for irrelevant material and retallying the scores.

2. Different judges use different scoring scales. Even when you tell everyone to grade on a scale of 1-100, some people will put all of the quality entries into the 70-100 range, some will put them all in the 90s, etc. This has the effect of valuing some judge's results over others. (Whomever uses the wider range will knock people down further and create the real separation between places.)

There are a couple of ways to solve this: The solution I prefer is to simply take the judges results and turn them into rankings instead of scores. Add up the rankings and whomever has the lowest ranking total wins. You can then use the raw scores to break ties or bump people a place or two in whatever direction as you see fit.

A second solution would be to give the judges a guideline. Pick a few example entries of different quality and score them yourself. Give them, say, an example of a 30, and a 70. Have them score the other entries relative to those. That helps keep everyone on the same scale.

3. Judges interpret the rules differently and disqualify or lower the scores of entries you felt were perfectly within the rules. This is the easiest problem to solve: Take the rule interpretation out of the judges' hands entirely by doing all disqualifications yourself. I do all of my own disqualifications before passing things on to judges. This just means a cursory glance to make sure all materials used were legal, grammar is acceptable, nothing is plagiarized, etc. (I do grammar pass/fail rather than points.) I also take the time in that run through to eliminate the really terribly composed entries. 5 judges don't need to each waste time scoring entries that clearly stand no chance of even an honorable mention.

Every entry passed on to judges in my contests qualify under the rules, so the judges can judge based on skill rather than worrying about whether I wanted an entry thrown out. Also, by eliminating the lousy entries, the judges can make full use of the scoring range to separate contenders.

4. Cultural differences between judges can lead to radically different scores. This is particularly problematic because it can completely trash the score of a great entry if the style they used was too American for your British judges or vice versa. I ran into this problem in Happy Hallmark Holiday when it became clear that British and American Valentine's card traditions are remarkably different.

In that case and others like it, I respond by throwing out the low scores that were clearly affected by cultural differences and using one or two tie-breaker judges to sort out the final order. So, for instance, if the British judges scored entry A very high but the US judges didn't, and the opposite was true for entry B, I throw out the US scores for A and the British scores for B then send the final list (in order) to a tie-breaker judge and see if they think any changes should be made. I of course explain to whomever is breaking the tie what cultural issues there are regarding each entry.

It is important, though, to try to get a diverse set of judges so you can account for these cultural differences. An all-US judging set, for instance, would inadvertently penalize people who create valid and clever entries in a different tradition. I know it's not possible for all cultures to be represented, but I try to get at least 2-3 countries in the mix.

5. Potential judging bias in favor of personal friends, as stated above by Feylind. This one is tricky. Obviously the first key is to get judges you trust to be impartial, but bias can still creep in. One solution if there is suspected bias is to use the tie-breaker judge(s) as I do for the cultural difference bias. You can also toss out, disregard, or lower the weight of anomalous scores. When all is said and done, the event runner is the final judge. You are not married to the numbers judges pass you, so can adjust scores as necessary to keep someone who was ranked 15, 13, and 1 by your three judges from finishing in the top 3. You'll also want to ask your judges about their reasons in a case like that. You may discover a cultural difference is causing the issue, not bias in favor of a friend.

Edit to add: Of course, there is a final and perfect solution to bias for or against friends: Present the entries to judges in an anonymous fashion. This isn't possible for all contests, but for some you can accept entries by email, or at least ask your judges not to look at the entry thread and then compose a judging page for them. If it's images, make your own photobucket album. If it's writing, a simple website can hold all of the entries. This is a lot of extra work and isn't usually necessary, but it does solve that problem perfectly (as long as the judges don't talk to their friends about entries).
Captain of Flocktarts

Avatar by Hedvvig

Puzzle Pirates™ © 2001-2016 Grey Havens, LLC All Rights Reserved.   Terms · Privacy · Affiliates