• Play
  • About
  • News
  • Forums
  • Yppedia
  • Help
Welcome Guest   | Login
  Index  | Recent Threads  | Register  | Search  | Help  | RSS feeds  | View Unanswered Threads  
  Search  


Quick Go »
Thread Status: Normal
Total posts in this thread: 10
[Add To My Favorites] [Watch this Thread] [Post new Thread]
Author
Previous Thread This topic has been viewed 5662 times and has 9 replies Next Thread
Deborah

Member's Avatar


Joined: Dec 7, 2003
Posts: 636
Status: Offline
When Judges Go Wrong... Reply to this Post
Reply with Quote

I'd like some advice for future events, based on some troubles I've run into in previous ones.

I'm sure I'm not the only person who has run an event, only to have the judges do completely wild and wacky stuff...even after they were given explicit / specific instructions.

What have you done in those situations, mates?

Generalized examples:

--A judge, when told to judge the entire field of < 20 entries, judged their favorite 3 and then gave all the other contestants the same exact score (so the scoring went something like 98, 95, 91, 80, 80, 80, 80, 80...)

My response was to throw this judge's 'efforts' out entirely as it completely screwed up the tally to have people who barely satisfied the requirements receive the same score as people who'd obviously done a fantasic job yet weren't this person's "top 3".

--A judge felt the product was outstanding, but the colors chosen were not optimal, and in addition to the judging criteria provided this judge also added coloring/choice of materials to the criteria.

My response in this situation was to include this judge's efforts, even though it wasn't part of the criteria I'd provided. In my opinion it fell under the subjective points each judge could award.

--A judge counted against all entrants who used the British spelling of words like "colour" and "honour".

My response was to remind the judge that those were perfectly legitimate spellings for those words, and that unless the entrant had swapped back and forth between spellings in one entry, it was legitimate and valid and should not affect the score. (The judge agreed and was kind enough to rework their tally.)

--A judge based their scores for particular entries on some of the anti-cheating photos rather than the submitted Final Project photos.

Even though it made some of the detail stand out more, this was specifically against the judging criteria sent out, so I decided to eliminate this judge's responses as there was no way to tell which entries they'd used the Final Photo only on and which they'd used creation / anti-cheat photos on.

-----------------------

In these situations, how would you have handled these judges?

--m3
----------------------------------------
Hermes says, "At any rate, I'm afraid I'll have to cede to your blockade, ladies. Business calls."
Madmarymeade says, "I think this means Rexe-Mary 2, OMs 0?"
Hermes says, "We go down easily."
----------------------------------------
[Edit 1 times, last edit by Deborah at Jul 17, 2008 10:12:52 PM]
[Jul 17, 2008 10:12:20 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Ascotte

Member's Avatar


Joined: Apr 28, 2006
Posts: 1470
Status: Offline
Re: When Judges Go Wrong... Reply to this Post
Reply with Quote

 
--A judge, when told to judge the entire field of < 20 entries, judged their favorite 3 and then gave all the other contestants the same exact score (so the scoring went something like 98, 95, 91, 80, 80, 80, 80, 80...)

My idea would be to keep his top 3 aside if that fitted all other judging rules. It could then be used in case of doubt or in case of an equal score.

 
--A judge felt the product was outstanding, but the colors chosen were not optimal, and in addition to the judging criteria provided this judge also added coloring/choice of materials to the criteria.

Judging is always partly subjective. This just points that out very clearly. It would be best to ask to have a clear look at the other entries too and to judge only by the current rules. That he will include personal preferences when making up his mind is inevitable.

--
 
A judge counted against all entrants who used the British spelling of words like "colour" and "honour".

Correct spelling, but when you do have a writing contest where spelling would be really important you could leave out those who constantly tend to swap between american and british english (using both honor and honour at the same time). Though being too hard on this might not be ideal neither, since that would leave out most non-native english speakers.


 
--A judge based their scores for particular entries on some of the anti-cheating photos rather than the submitted Final Project photos.

same response as you did.


judging is subjective for a large part, so was my respons =p
----------------------------------------
Ascotte,
member of the White Seahorse Club.
Playing on Emerald
 
"East-West, Huntards are best!"

 
tells ye, "You invented forum cades"

[Jul 17, 2008 11:25:00 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
BlackBeth

Member's Avatar


Joined: Feb 28, 2005
Posts: 1858
Status: Offline
Re: When Judges Go Wrong... Reply to this Post
Reply with Quote

Basically, I think judging issues need to be considered on a case by case basis, exactly as you did. There's certainly no one solution that works in all cases.

For the case studies you mentioned, I agree completely with your solutions to all except maybe the last one, because I am not sure if I am understanding the situation correctly.

If the judging instructions stated that judging should be done based on the one final photo, only, and the judge did something else, then yes, I agree about not using those scores. My concern is more with the fact that for most "create an actual object" art events, it's very hard to show all the work and detail in one photo. The object is what's being judged, not the photography skills of the artist. So I tend to think it's reasonable for judges to be allowed to consider in-process photos, which can show creativity in materials and methods, and close-up and side/back view photos, which can together give a clearer picture of what the entry really looks like. So in many cases, I feel like instructing the judges to use only the final photo may not be in the best interests of the competition.

Here's another case study to consider, from one of my first events:
In a writing event, one of my judges gave the top score to an entry by one of her crewmates which I didn't personally feel was deserving of that score. (Nor had any other judge scored it highly.) But judging is always subjective, so it was just my personal opinion that the judge was biased; there was no judging instruction clearly broken. I was therefore reluctant to just throw out that judge's work. I was also reluctant to let it stand because due to the small number of judges, that questionable entry was highly placed based on that one score. In the end, the solution I adopted was to keep the judge's scores, but to hire 2 additional judges. I figured they would either validate the score I questioned, or dilute it. (Since then, I have never run a contest with less than 5 judges; lesson learned.)

Does anyone have a better solution to offer for that one?
----------------------------------------
Feylind

Semi-retired Midnight Cerulean Event Addict
[Jul 18, 2008 7:20:19 AM] Show Printable Version of Post        Send Private Message    FeythePirate [Link]  Go to top 
Aethera21

Member's Avatar


Joined: Aug 22, 2005
Posts: 2831
Status: Offline
Re: When Judges Go Wrong... Reply to this Post
Reply with Quote

In cases like the extra judging criterion and the photos issue, I'd say that could be resolved by telling the judges up front that it's very, very important for them to ask you about those types of things early in the process, rather than assuming something they're going to do is ok. The guy who wanted to judge on color should have PMed you the first day and said "hey, can we add a color category? I think that's important" and leave it up to you instead of making a unilateral decision. Judges have to be expected to abide by whatever criteria the event runner sets out. If the judge is unable to accept any decision you make, you'd need to find yourself another judge.


Things like British vs. American spelling, I'd never have thought of. I guess I assume everyone knows the Brits spell a few words differently than Americans do, but of course that's probably not the case. I think in that instance, you would have to deal with it case-by-case, as you did here...an event runner can't be expected to think of everything in advance. Of course, the next time spelling/grammar is a judging category, you'd probably include that note in your instructions...learn from experience, as it were :)


Finally, having a couple of extra judges means that if a judge is lazy about scoring, or scores things completely wrong, you can drop them without compromising the integrity of the scoring. Alternatively, you could have them re-do their scores if feasible. However I wouldn't have included the guy who scored everyone after 4 at an 80 in the tally of scores.
----------------------------------------
Aethera of Cobalt Cerulean
Princess of Maniacal Menagerie
Blockade staff on any ocean
Avatar by Bunnylaroo
[Jul 18, 2008 10:04:30 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Squashbuckle

Member's Avatar


Joined: Dec 21, 2004
Posts: 6276
Status: Offline
Re: When Judges Go Wrong... Reply to this Post
Reply with Quote

Having run a wide variety of judged events, I think I've run into just about every judging snafu you could imagine. In addition to your issues, here are some common problems and how I solve them:

1. The judging rubric may unintentionally set things up so the best entry doesn't get the highest score. I've run into this several times, especially in art contests. In one case, for instance, I had a category for the piratey details to give what seemed like a deserved bonus for clever touches. In the end, though, an extra few points for an added BK trinket in the background that didn't actually change the mood or skill of the piece bumped some people above others who had much more artistic skill and much more clever entries overall.

If this happens to you, I would strongly suggest throwing out any categories that seem to give bonuses for irrelevant material and retallying the scores.


2. Different judges use different scoring scales. Even when you tell everyone to grade on a scale of 1-100, some people will put all of the quality entries into the 70-100 range, some will put them all in the 90s, etc. This has the effect of valuing some judge's results over others. (Whomever uses the wider range will knock people down further and create the real separation between places.)

There are a couple of ways to solve this: The solution I prefer is to simply take the judges results and turn them into rankings instead of scores. Add up the rankings and whomever has the lowest ranking total wins. You can then use the raw scores to break ties or bump people a place or two in whatever direction as you see fit.

A second solution would be to give the judges a guideline. Pick a few example entries of different quality and score them yourself. Give them, say, an example of a 30, and a 70. Have them score the other entries relative to those. That helps keep everyone on the same scale.


3. Judges interpret the rules differently and disqualify or lower the scores of entries you felt were perfectly within the rules. This is the easiest problem to solve: Take the rule interpretation out of the judges' hands entirely by doing all disqualifications yourself. I do all of my own disqualifications before passing things on to judges. This just means a cursory glance to make sure all materials used were legal, grammar is acceptable, nothing is plagiarized, etc. (I do grammar pass/fail rather than points.) I also take the time in that run through to eliminate the really terribly composed entries. 5 judges don't need to each waste time scoring entries that clearly stand no chance of even an honorable mention.

Every entry passed on to judges in my contests qualify under the rules, so the judges can judge based on skill rather than worrying about whether I wanted an entry thrown out. Also, by eliminating the lousy entries, the judges can make full use of the scoring range to separate contenders.


4. Cultural differences between judges can lead to radically different scores. This is particularly problematic because it can completely trash the score of a great entry if the style they used was too American for your British judges or vice versa. I ran into this problem in Happy Hallmark Holiday when it became clear that British and American Valentine's card traditions are remarkably different.

In that case and others like it, I respond by throwing out the low scores that were clearly affected by cultural differences and using one or two tie-breaker judges to sort out the final order. So, for instance, if the British judges scored entry A very high but the US judges didn't, and the opposite was true for entry B, I throw out the US scores for A and the British scores for B then send the final list (in order) to a tie-breaker judge and see if they think any changes should be made. I of course explain to whomever is breaking the tie what cultural issues there are regarding each entry.

It is important, though, to try to get a diverse set of judges so you can account for these cultural differences. An all-US judging set, for instance, would inadvertently penalize people who create valid and clever entries in a different tradition. I know it's not possible for all cultures to be represented, but I try to get at least 2-3 countries in the mix.


5. Potential judging bias in favor of personal friends, as stated above by Feylind. This one is tricky. Obviously the first key is to get judges you trust to be impartial, but bias can still creep in. One solution if there is suspected bias is to use the tie-breaker judge(s) as I do for the cultural difference bias. You can also toss out, disregard, or lower the weight of anomalous scores. When all is said and done, the event runner is the final judge. You are not married to the numbers judges pass you, so can adjust scores as necessary to keep someone who was ranked 15, 13, and 1 by your three judges from finishing in the top 3. You'll also want to ask your judges about their reasons in a case like that. You may discover a cultural difference is causing the issue, not bias in favor of a friend.

Edit to add: Of course, there is a final and perfect solution to bias for or against friends: Present the entries to judges in an anonymous fashion. This isn't possible for all contests, but for some you can accept entries by email, or at least ask your judges not to look at the entry thread and then compose a judging page for them. If it's images, make your own photobucket album. If it's writing, a simple website can hold all of the entries. This is a lot of extra work and isn't usually necessary, but it does solve that problem perfectly (as long as the judges don't talk to their friends about entries).
----------------------------------------
Polly
Hunter
Captain of Flocktarts

Avatar by Hedvvig
----------------------------------------
[Edit 2 times, last edit by Squashbuckle at Jul 18, 2008 10:30:11 AM]
[Jul 18, 2008 10:23:36 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Aethera21

Member's Avatar


Joined: Aug 22, 2005
Posts: 2831
Status: Offline
Re: When Judges Go Wrong... Reply to this Post
Reply with Quote

Polly wrote: 
Edit to add: Of course, there is a final and perfect solution to bias for or against friends: Present the entries to judges in an anonymous fashion. This isn't possible for all contests, but for some you can accept entries by email, or at least ask your judges not to look at the entry thread and then compose a judging page for them. If it's images, make your own photobucket album. If it's writing, a simple website can hold all of the entries. This is a lot of extra work and isn't usually necessary, but it does solve that problem perfectly (as long as the judges don't talk to their friends about entries).



Back when PTG did the Author's Author Death Match, which I was a judge for, he created a website that allowed for anonymous judging, and it worked really well. I seem to remember him popping up again recently...maybe someone who knows something about websites should bug him for the code.
----------------------------------------
Aethera of Cobalt Cerulean
Princess of Maniacal Menagerie
Blockade staff on any ocean
Avatar by Bunnylaroo
[Jul 18, 2008 11:53:54 AM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
Deborah

Member's Avatar


Joined: Dec 7, 2003
Posts: 636
Status: Offline
Re: When Judges Go Wrong... Reply to this Post
Reply with Quote

I typically try to have several 'backup judges' lined up just in case my original judges go out of town or something crazy like that. I also do any disqualifications before sending things on to the judges -- much easier that way.

Fey, in regards to using just one photo, that was specific to the Pumpkin Carving contest. The single "lit photo" was specifically stated in the rules to be the one that would be judged. In other contests I've run, such as the Woolly contest, I've let entrants submit multiple photos to be used in the final judging. I also created a special Googledoc where judges could go to get links to each (non-disqualified) entrant's work.

Excerpt from the Pumpkin Carving judging PM:
Several of our carvers used a similar pattern--they may have modified it slightly (adding text, for example, or using a mirror image of it)--and this will bring you added joy in your judging, I'm sure. Personally, I give more weight to 'homemade' patterns rather than storebought or commercial ones, but you are entitled to judge as ye see fit. This is a carving contest, after all, not a pattern-making contest.
The contest description states that "judging will be based upon the creativity and cleverness of design, the quality and detail of carving, and the overall effect of the lit pumpkin."
Please remember that it is the LIT photo that "counts", so to speak. If the unlit photo is amazingly detailed, yet none of the details were carved deeply enough to show up on the lit pumpkin, the details don't "count". If the photo of the lit pumpkin is blurry or fuzzed, unfortunately that's the entry photo you are judging, regardless of how awesome the unlit or progress photos look.

Here's an example of judging instructions I sent out for the Woolly contest (most recent, so easiest to find in my outbox). First, I copied the judging criteria and anything pertaining to rules that the contestants saw in the original post. Then I included these instructions:

Note that all of the entrants listed with links on the Googledoc passed the "Objective" criteria portion of the judging. You fine judges have the hard task of judging based on the Subjective criteria. The grey background on some of the links is to help keep the entries separated and does not denote anything special.
Entrants were allowed to submit multiple "finished" photographs, and I have included several links for some of the entrants. Some views show an excellent view overall, while other links showcase fine detail.
To Judge:
Please take your time and look at each entry, and all final entry links provided for each entrant. You will be scoring the entries based on the following breakdown:
20 points for creativity / originality of seascape design
20 points for fine detail and complexity / variety of stitching or gluing
20 points for overall appeal of the completed woolly (does it appeal to you, engage your imagination, is it something you would say, "Wow, that's nifty!" about?)
(note that each entrant listed with links on the g'doc page already received 40 points for the Objective criteria)
When you have completed your judging, please either reply to this pm, or send me a PM with the subject "Woolly Judging". Please list the entrants and the breakdown of their score. (example: Madmarymeade 15 / 15 / 4)


I was a "wonky judge" for someone during a writing contest. It is a personal pet peeve of mine to see misused homophones, mismatched tenses, commas used in place of semicolons (and vice versa), apostrophes scattered like birdshot, and misspellings that a spellcheck program would have picked up on. I graded very harshly against those things and pretty much booched up the score curve so much that the contest runner asked me if I was judging in favor of certain entrants. Oops. So, it's not always favoritism or bias for a particular person or entry that motivates a judge, heh. *grin*
----------------------------------------
Hermes says, "At any rate, I'm afraid I'll have to cede to your blockade, ladies. Business calls."
Madmarymeade says, "I think this means Rexe-Mary 2, OMs 0?"
Hermes says, "We go down easily."
[Jul 18, 2008 1:25:24 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
BlackBeth

Member's Avatar


Joined: Feb 28, 2005
Posts: 1858
Status: Offline
Re: When Judges Go Wrong... Reply to this Post
Reply with Quote

 
I was a "wonky judge" for someone during a writing contest. It is a personal pet peeve of mine to see misused homophones, mismatched tenses, commas used in place of semicolons (and vice versa), apostrophes scattered like birdshot, and misspellings that a spellcheck program would have picked up on. I graded very harshly against those things and pretty much booched up the score curve so much that the contest runner asked me if I was judging in favor of certain entrants. Oops. So, it's not always favoritism or bias for a particular person or entry that motivates a judge, heh. *grin*

Lol. In that case, I personally think maybe the other judges were the ones booching the curve, not you. Spelling and grammar errors are very jarring to the reader and detract horribly from even the best plot. Proofreading is a very basic part of writing, and it's reasonable to expect it. A few errors always slip through (picking them out of published novels is always fun) but I'd be horrified if I ever ran a writing contest and the winning entry was chocked full of obvious spelling/grammar/punctuation errors.
----------------------------------------
Feylind

Semi-retired Midnight Cerulean Event Addict
[Jul 18, 2008 2:12:13 PM] Show Printable Version of Post        Send Private Message    FeythePirate [Link]  Go to top 
sweetnessc

Member's Avatar


Joined: Nov 10, 2004
Posts: 16105
Status: Offline
Re: When Judges Go Wrong... Reply to this Post
Reply with Quote

No, you're wrong, it was definitely m3 booching the curve. *grins*
----------------------------------------
My friends, love is better than anger. Hope is better than fear. Optimism is better than despair. So let us be loving, hopeful and optimistic. And we'll change the world. ~ Jack Layton

Sublime is shame.
[Jul 18, 2008 4:12:58 PM] Show Printable Version of Post        Send Private Message [Link]  Go to top 
BlackBeth

Member's Avatar


Joined: Feb 28, 2005
Posts: 1858
Status: Offline
Re: When Judges Go Wrong... Reply to this Post
Reply with Quote

 
No, you're wrong, it was definitely m3 booching the curve. *grins*

Yeah, yeah. It amazes me sometimes how differently different judges see things. During Round 1 of "Space Pirates", I asked the judges to provide me an ordered list of the what they thought were the 20 best entries. When I combined the scores after the first two judges had reported in.. I had 39 different entries on the list.

After that, no amount of judging variation will ever surprise me again.
----------------------------------------
Feylind

Semi-retired Midnight Cerulean Event Addict
[Jul 18, 2008 4:37:18 PM] Show Printable Version of Post        Send Private Message    FeythePirate [Link]  Go to top 
[Show Printable Version of Thread] [Post new Thread]

Puzzle Pirates™ © 2001-2016 Grey Havens, LLC All Rights Reserved.   Terms · Privacy · Affiliates