« 2015 Independent Games Festival opens call for submissions | Main | IGF China 2014 reveals Main Competition and Student finalists »

IGF statement re: judging process

You may have recently seen some odd claims of collusion between independent developers/funders & both the Independent Games Festival and Indiecade, most recently with untrue allegations that select independent developers have worked to 'fix' both award events.

The IGF (and its parent company, which has looked into these claims) would like to assure all entrants and gamers that we take the idea of conflict of interest seriously in selecting our judging and jurying pools. Worries about the Independent Games Festival are nothing new (thanks to its perceived 'star-making' qualities), but for those who are unaware, we'll explain again how the process works.

The explanation below is the process we've had in place since 2011, and, in the interest of transparency, it will be officially added to our FAQ in the coming days:

Each year, we invite hundreds of game developers both big and small, educators, critics and press to be the initial IGF 'large' judging pool. (Think of the judges as an Oscars-style cross-section of the game community.) If any individual was directly involved in the creation of any game entered in the festival that year, they recuse themselves from judging the festival. As explained in this 2014 blog post, any interested party that works in the games industry can request an invitation to be a judge. If you are selected, you'll receive an email from us when our judging process has begun.

All submitted games are assigned to a random set of judges (nearly 375) from our 'large' judging pool. It's possible to see how the random assigning works, and third parties have verified that it's truly random - based on the game platforms the judge owns. Judges are given around 4-6 weeks to play and recommend any game they believe should be a finalist in any category.

Over time, games are randomly assigned to even more judges, to ensure a breadth of opinion. Judges can then communicate with one another to recommend games, schedule multiplayer sessions, and raise any technical issues directly with a game's developer, as well as debate the positives and negatives of specific titles. There is plenty of positive (and negative) debate on specific titles. After judges finish recommending their assigned games, they are free to explore the rest of the entered games, and vote for any additional game they believe deserves a recommendation.

At the end of the 'large' judging period, we then enter a jurying phase. There's is a different jury for each award, made up of subject matter experts - except the Audience Award, which has public votes. Here, for example, are all members that made up 2012's juries. They are given access to the total number of recommendations each game received for their particular category. They also receive another metric that they look at: a percentage that represents how many recommendations each game got from assigned judges. This helps to bring less well-known titles to light and guards against judges 'swarming' games they might have heard of.

Each jury, made up of 7-15 members, takes another 4-6 weeks to play and discuss the games in private. At the end of this discussion, each individual juror puts forward their list of nominations for the specific award they are judging - as many or as few as they like (often called 'approval voting').

These nominations may include the games that received many recommendations from the judges, but often includes titles with less recommendations as well. (Think of the juries as film festival panels, such as the Cannes Palme D'Or). The jury nominations are then tallied, and the games which received the most nominations are then decided as the finalists and honorable mentions.

At that point, each individual IGF jury then takes an additional few weeks to discuss those finalists, and casts another vote on which games they believe should be the winner in each category. Those votes are tallied by hand, and the game that received the most votes is declared the winner at the IGF Awards every March.

Our biggest issue tends to be the growing number of entries, something we hope our new and more open judge recruitment will help alleviate. Thanks, as always, to the larger community - the hundreds or even thousands - involved for helping us make the awards happen.