26 June 2010

New rankings now in place

You will notice big changes in the rankings between last week and this week.

Over the weekend, we put into place a modification to the algorithm by which we score and rank players.

Before going into the details of the change, please note that on the Rules page we say:

"We reserve the right to change the scoring and ranking algorithms at any time; such changes may be applied retroactively in the current quarter. We guarantee, however, that the algorithm will always be applied equally to all players."

We don't make such changes lightly, but in this case it was required - and the result is a much more accurate reflection of the way players compete on the PL/SQL Challenge.

Some background: in the process of analyzing quiz answer data, we noticed that many players were visiting the quiz page more than once in a day. That is, you log in, press Take the Quiz button and then "I'm Ready! Start the Quiz Now" button, at which point you see the quiz for that day. But these players were not submitting their answer at this time. Instead, they were coming back later in the day to go through the same steps and then press Submit Answer.

Of course, in many cases, this is happening because a player might lose a connection or have to "get back to work!" (so sayeth the unenlightened manager). But in several other cases, we have players who routinely "double visit" the quiz page - which can offer a significant advantage, and might even be considered a form of cheating.

Putting aside the issue of cheating for a moment, we decided that we can and should adjust our algorithm to take this behavior into account by changing the way we calculate the number of seconds used to answer the question. Previously, the number of seconds was computed as the amount of time between pressing the "Start the Quiz Now" button and the "Submit Answer" button, in that same session (this will be referred to as TQ below).

From now on, the number of seconds will be computed as follows:

FV = Time of first visit to quiz page
LV = Time of last visit to quiz page

Number of seconds = (LV - FV) + TQ

The downside of this change is that if you visit the quiz page and then lose your connection and cannot return for a long time (or some other similarly innocent scenario), you will receive a greatly-reduced score. If this happens you can contact us via the Feedback button and request "forgiveness" for that gap in time. We will likely make the adjustment if it happens only very rarely.

The upside is that the score now much more accurately reflects the time that a player takes to figure out the correct answer to a quiz.

We hope you agree with these changes; if you have any questions of concerns, please reply to this blog or provide feedback through the PL/SQL Challenge site.

9 comments:

  1. My coworkers and I had noticed that over the past several weeks a number of players had apparently dramatically improved their performance on the quizzes. Our suspicion had been that they were using multiple accounts or were getting the quiz questions from someone else in advance of visiting the quiz page. We had not even considered the “disconnection” mechanism to account for their “improved” abilities. In my opinion this method is most definitely cheating. The new scoring algorithm seems to have corrected the rankings of most of the players that we thought showed unbelievable advancement)

    I do have one issue with the new algorithm: using the “back” button after submitting one’s responses, if recorded as an access to the quiz page, would negatively affect the player’s score. Such an impact would be unfair since there is no mechanism by which the player could change their responses. This concern can be addressed by scoring the time to answer as the time of submission minus the time of first visit to the quiz page.

    ReplyDelete
  2. This sounds completely fair to me.

    I didn't realize the prior method was using (last visit + submit time); I had always assumed it was (first visit + submit time).

    Just to be safe, FV and LV would always be the same if you only visit once, resulting in (0 + TQ), correct? I had just imagined a ((0 - FV) + TQ) resulting in a negative adjustment to TQ. :)

    ReplyDelete
  3. John, (JHall62)

    That is a very good point, and that makes more sense. We shouldn't count visits that occur after you have submitted your answer. We will adjust our algorithm and re-score.

    Richard, yes, there are never negative adjustments.

    Many thanks, SF

    ReplyDelete
  4. I absolutely don't get the new system. Why visiting this page is considered something like cheating? The page describing rules hmmm describes the rules. And, BTW, it is this page the user is redirected to when he enters user name + password and clicks the button next to the input box - IMO it's page design f*ck up. I don't expect this button being there.

    PS. Another rant from me on the same topic: I really don't want to even press login button, I want "remember me" option.

    ReplyDelete
  5. We're not talking about the first step of the page -- we're talking about the next step where you see the question and the options. That's cheating when you see the quiz, think about it for a while, potentially research it, come back it and then answer it, trying to beat the timing algorithm.

    Hope this helps.

    Cheers

    FinnIt's cheating to view the question, go away and think about it for a while

    ReplyDelete
  6. Oh, I see. I didn't understand the message correctly, my mistake. Now I agree that it was a bug in the algorithm, and it indeed allowed to cheat.

    ReplyDelete
  7. I'm 100% behind the improved scoring algorithm even though I {was) one of the cheaters.

    ReplyDelete
  8. Frankly I have not read in much detail how the new scoring system is working but I am sure SF and Finn must have worked hard to design the new algorithm, so I am quite sure it is good for all of us :)

    But I have one suggestion regarding "time" part in calculating the score. It would be great (IMO) if the clock starts after a minute (or may be less) once you hit "Start Quiz" button. That way everybody will get some time to concentrate and think over the question.

    Many times it happens that I lose concentration in a hurry of submitting answer to score more and end up in selecting wrong answers. Actually I just took today's quiz and after submitting answer as usual I tried checking the correct answer and I found I selected "totally" wrong answer even though I knew the correct answer. So in frustration I thought of this idea and posting this comment immediately. Feeling so sad about it :(

    As said I haven't gone much in details of new scoring system and am not sure if it is feasible to incorporate this in the new algorithm but guess many players will agree with this idea.

    ReplyDelete
  9. Ramesh, I appreciate your concern and your idea, but I really don't see how that could work. For many people 1 minute is way more than they need to answer most of these questions.

    SF

    ReplyDelete