We have been spending lots of time and exhausting many brain cells as we fine-tune our algorithms for scoring and ranking to ensure players trust the PL/SQL Challenge and so that everyone competes on a level playing field. We have also gotten many ideas from players, some of whom have suggested that we not include the time it takes to answer a quiz in the scoring and rankings of players. Others urge us to modify our scoring algorithm to diminish the effect of "time to answer."
You will find below an explanation of why we need to use timing. I then offer some guiding principles for us as we make changes to rules, and finally I announce a new rule regarding the treatment of very fast answers.
I hope you find this helpful and that it will enable you to enjoy the PL/SQL Challenge even more than you have to date.
The Importance of Timing in Our Scoring Algorithm
We certainly can adjust the role that timing plays in ranking players (several of you have offered specific recommendations), but timing
must be incorporated into the algorithm. Without it, there can
be no ranking. If rank is determined solely by correctness, then what you will see in the rankings are lots of ties. There simply aren't that many ways for experienced PL/SQL developers to answer these quizzes. With dozens or more people tied at every score level, the rankings would become meaningless.
The amount of time it takes a player to complete the quiz aces, in essence, as the "tie breaker" for people who achieve a similar level of correctness. This means that we must do everything possible to make it difficult to "game" the system through timing. Unfortunately, given the nature of the quiz, that players take it without any supervision, we have no way of knowing for sure if someone is cheating.
That is, it is certainly possible that
no one is cheating (any longer, at any rate; two people
did confess their cheating to us). It is possible that
everyone is cheating. There is no way for us, the organizers of the PL/SQL Challenge, to
know, for sure.
All we can do is analyze player activity and identify patterns of play that can be explained most simply by an act of cheating. That is, you could come up with more complicated explanations for how a certain level of performance is possible, but the simplest, most obvious explanation is: cheating.
We need, therefore, to set up rules that discourage players from following those patterns - ranging from rules making certain patterns illegal (if this is possible) to disincentives (such as the disqualification of answers under certain circumstances) to outright expulsion (likely for repeated violations of the rules).
Principles to Guide Our Rules
The rules for PL/SQL Challenge scoring and ranking must:
1. Make sense to our players. They should be intuitive, and obvious. They should feel right and fair.
2. Be easy to administer. If they require a large amount of time to monitor or enforce (daily manual analysis, responding to requests for exemptions, etc.), we will be overwhelmed and unable to apply the rules properly.
3. Emphasize correctness over timing. We don't want to lose sight of the main objective for most players: to improve
and to demonstrate their expertise in the PL/SQL language.
4. Address very fast answers. This is a pattern that can most simply be explained through cheating (multiple accounts). It does
not mean that players with very low timings are cheating. Our rules will explicitly discourage players from answering more rapidly than we believe is possible without some kind of manipulation of the rules.
5. Offer forgiveness. Many of our players are very serious about the PL/SQL Challenge and their rankings. They realize that missing a day can have a big impact. They also agonize over dumb mistakes that cost them valuable points. We don't want the PL/SQL Challenge to be stressful game, and we don't want players to be discouraged about their prospects if they miss a quiz or have a bad day. So we will introduce rules soon that will soften the blow of missed quizzes or other rank-damaging occurrences.
Immediate Changes to Quizzes
We are going to take the following steps with all future quizzes to make it easier for all players to compete on a level playing field.
1. Design the question and choices so that they must be read fully in order to have a reasonable chance at getting the answer right. You will no longer take a quiz that can be answered with nothing more than a glance at the question and choices.
2. Determine and record the minimum number of seconds that
we believe a player would need to actually read and answer the question. This number will
not be made public. Of course, this is to some extent a subjective determination on our part. We will, however, be conservative in setting this value; if any of you saw the number, you would immediately agree that it is sensible and acceptable as a lower limit. You will, however, have to trust us.
3.
We will also take the quiz and record the number of seconds it takes to do so. This will be the "expert's timing"; we figure that since Steven wrote almost all of these questions
and knows PL/SQL pretty well that he should be able to answer these questions just about as fast as anyone else. This number will also
not be made public.
4.
We will randomly enforce the minimum number of seconds. On randomly-selected days of the month, the score of all players whose timing is less than the minimum will be set to 0. You will not know on which days this occurs until after rankings are performed.
We hope that these changes will establish a powerful disincentive to answer very quickly. You don't know when a very fast answer might result in a disqualification and you don't know what the minimum timing is for a given quiz. So you should take your time, read the question fully, then submit your answer.
This additional timing information also gives us the opportunity to analyze patterns of player behavior. We will be able, for example, to identify players who
routinely answer quizzes faster than we believe is humanly possible.
We hope that these principles and rules make sense. We will be updating the Rules page of the website very soon. Please reply to this blog or send an email to quizmaster@plsqlchallenge.com, if you have questions or suggestions.
Regards,
Steven Feuerstein
