06 November 2012

Change Scoring Algorithm for Playoffs?

A player recently wrote the following to us:

I would like to share some thoughts about Playoff rankings. It is not about my own results, just some thoughts of how to make it more fair. Don't you think it is not too fair, for example, to answer 100% for 35 mins and finally lose to one having 80% correctness and a much shorter time (15 min)?

I know you the same system for the daily quiz as in the playoff, and a player can sometimes win with some errors if they are quick enough. But there is no time limit in the daily quiz, so it's OK there.

Regarding playoff, it is all quite different. Existing situation rewards very quick answers. Even if you make a lot of mistakes, you will probably be among the best. Conversely, if you used almost all the time available, even 100% correctness doesn't guarantee a first place showing.

Why not rank playoff with a very simple algorithm, like that shown below (unweighted_score is the score based only on % correct, independent of time taken to submit the answer):

select player_name, unweighted_score, time
from playoff_results
order by
unweighted_score desc, time asc

I see many advantages in this approach. Simplicity, fair, tendency to thoughtful answers. It will stimulate thinking and not guessing...but time still plays an important role. Do you see any limitations?

I would very much like to know what you think, especially players who have participated in a number of playoffs over the past few years. First, though, some background:

The quarterly playoff has changed quite a bit from its original form back in 2010. Originally, the playoff consisted of 10 quizzes to be completed in 10 minutes. The idea was that you should have no or very little time to write code to verify answers and certainly not much time to look up answers or consult with others.

This didn't work out so well. First problem is that it turned out to be very hard for me, Steven, to judge the amount of time needed for a player to read and think through a quiz. So many, many players ran out of time.

The second problem was that coming up with ten new and tough quizzes for the playoff was, well, a challenge.

The third problem had to do with a very hard to track down bug that causes errors when players did run out of time.The site was supposed to very smoothly and elegantly notify you that you had run out of time, and save all answers you'd entered. Ah, software...instead, players would encounter a variety of errors and in some cases, we lost answers.

We were never able to reproduce the error. So we took the following steps instead (to address both of the problems mentioned above):

1. Cut down on number of quizzes from 10 to 5. Ah, what a relief!

2. Extend the amount of time in which players can complete the competition. It is now up to 35 minutes.

3. Remove the countdown clock from the page. We theorized that the Javascript needed for the clock was somehow interfering with other processing on the page and leading to the errors.

And how did this work out for us? Pretty well. We haven't seen those errors since, though that may also be because fewer players are actually running out of time. But this also means that players have a lot more time to research their answers, as opposed to relying solely on their own knowledge and experience to answer the quizzes.

I am not terribly comfortable with the current situation in the playoffs; it's hard not to want to give the 1st place prize to the person with 100% correct answers. Yet I am also reluctant to assign prizes by unweighted score, using submission time only to break ties, at least if we give players 30 minutes to submit their answers.

So...what do you think?