06 August 2010

An update regarding very fast answers

I received this email today from a devoted player:

"I continue to wonder how fast some people manage to submit. Like "Neha Jain", only three seconds to check three questions before hitting the submit button. How's that possible even when realizing that of course no question text could possibly be read. I've used to period-type "Week" report to get to the conclusion above. The same holds for the next conclusion: Well, Neha just answered one question this week. But what about Alen Oblak, nr 23 Overall Rank. Just 37 seconds to answer all four questions (almost without error) this week? What can we learn from Alen ? I was getting curious because every day on 'Past Quizzes' the maximum score of the day is mentioned, which is always higher than reasonably possible for anyone not just hitting some buttons very quick at random."

At first, I hesitated to include the names this player mentions in his email, but that's the beauty of the way I am running the PL/SQL Challenge (not to sound boastful :-) ). We are open about this data so everyone can see - and wonder the same things.

When we upgraded to 1.4, I published a change in the rules that sent out a rather severe warning to people who submit very fast answers. I plan within the next week or so to apply these rules (and a modification of them, which will be published in the Rules section this weekend with the ugprade to 1.5).

Here's the bottom line: you are about to see significant changes in the rankings of players who seem to be able to consistently play the quiz dramatically faster than even I (a PL/SQL expert and author of the quizzes) am able to simply read my own questions.

As I have mentioned in earlier blog postings, I suppose it is theoretically possible for someone who is an incredible speed reader; amazing PL/SQL expert; and very lucky individual to answer much faster than I can manage, but that behavior cannot be distinguished from those who cheat. So all I can do, to preserve the credibility of the PL/SQL Challenge, is discourage players from following that pattern.

So, my friends, just a little bit more patience and then those rankings will start to make a bit more sense.

In fact, again for the sake of transparency, you will find below my plans for adjusting timings, scores and rankings for VFAs (very fast answerers). Feel free to let me know what you think.

Planned Algorithm

1. I record the time it takes me to quickly answer each quiz. I consider this a reasonable minimum (RM), since it is actually an aggressive score, as I know PL/SQL pretty well and I also wrote the quiz (making it hard to avoid rapidly identifying the correct and incorrect answers).

2. Every two weeks (10 quizzes), I identify all players whose total time is less than mine (call them TFP, too fast players). I choose every two weeks because (a) I don't want to compare and adjust/punish on a daily basis. I need to allow that some people will know a feature better than me and answer more quickly and (b) 10 quizzes should be enough to reveal a pattern. Over two weeks, I would expect my RM to be around 300 seconds or 30 seconds per quiz on average.

3. For each TFP, if the delta is within 10 seconds, I do nothing; that's close enough and nothing to worry about.

4. If TFP is more than 10 secs faster, then for every 10 seconds faster (or portion thereof), I add 60 seconds to the time for each quiz answered faster than I did on each day.

So if my RM is 300 seconds and a person answers in 280, and 5 of the scores are less than mine, then their overall time with increase by 5 minutes or so (5 x 1 x 60 = 300). This should be enough to cause a dramatic decrease in their ranking and be a substantial incentive to SLOOOOW down or stop cheating. If they only had a couple of too fast answers and the others are reasonable, then the impact on their is lessened.

Your thoughts?

SF

5 comments:

  1. Here's my idea for an algorithm that is less subjective (not based solely on a single individual's opinion of the correct "cut-off time", but is also adjusted on actual time taken by each individual player vs. other players). Any numbers/percentages given below should be adjusted to best fit the algorithm to an analysis of the actual existing data.

    After the completion of the day's quiz, start by taking a set of FPs (Fast Players) by taking the fastest 200 players or fastest 10% of players for the day (The sample taken should be at least twice as large as the number of suspected TFPs). This could also be done by taking all players whose time was less than say 45 or 60 seconds, or perhaps your reasonable Minimum (RM) + 5 or 10 seconds, as beyond that time you will have players who are doing more than reading & answering the questions based on their existing knowledge (reading documentation, testing code blocks, etc.)

    Now, determine the average (mean) time taken by all identified FPs for the day, take take a percentage of this (say 50%) to the nearest second, then add a 2 second grace period to the minimum non-cheating time for the day.

    Anything under the non-cheating time should be penalized by seconds as you have suggested, and anyone 50% or more under the adjusted minimum time should be scored as 0.

    The averaging and grace period should help to separate the likely cheaters from "honest" TFPs, as questions that are able to be answered much more quickly (and therefore more players are likely to answer quickly) will penalize less players because the adjusted minimum time will be lowered.

    ReplyDelete
  2. Hello,
    If you could make every single player see a different answer set for a particular question, it would be very very hard to cheat. However, this causes you to work far more harder on the questions' background.
    For example, for a question, let me see 3 options that are totally incorrect; other player can see 5 options: 1 correct, 4 incorrect; the other may see 6 options: 4 correct, 2 incorrect.
    Ideally, every player must see a different answer set for a question but this is too difficult to do, so you can prepare 5 or 6 different answer sets and every player may get 1 out of possible ones. This may reduce the possibility of cheaters' cooperation.

    ReplyDelete
  3. Bora,

    That is a fascinating idea - and, I must admit, well beyond our ability/bandwidth to implement right now. Perhaps someday....

    SF

    ReplyDelete
  4. One issue with Bora's suggestion is that, someone offered 5 options in the quiz can get a better score than someone with only 4 options since each option earns you points if you successfully say whether it is right or wrong.

    Personally, I'd let them get through to the quarterly challenge (or a 'Fastest Gun' challenge at the same time) and see if they can reproduce their results when there's no chance of cheating the timing algorithm.
    If they are the Usain Bolt's of the PL/SQL world, then so be it. If they are the Ben Johnson's, then that'll show them up.

    I would like to see, on the 'previous quiz' page, the percentage of people who got any choice right/wrong. It would be good to see where people's mis-conceptions lie.

    ReplyDelete
  5. "I would like to see, on the 'previous quiz' page, the percentage of people who got any choice right/wrong. It would be good to see where people's mis-conceptions lie."

    +1 from me

    ReplyDelete