17 June 2011

Change in Percent Corrrect May Be Noticed

A player wrote to me this morning:
My lifetime correctness has been around 75% for the last several weeks. Suddenly, when I check it this morning, it's dropped to 67.9%. Something is definitely odd, since I've been over 90% correct this week.
So what is going on? We finally fixed a bug in the way that % correct was being calculated for those "at most one right" questions.

A couple of months ago, we introduced a new "flavor" of question: when the choices are presented are clearly mutually exclusive, the scoring proceeds differently. Either you pick the correct answer and are 100% correct, or you picked the wrong answer and you get a score of 0.

This is different from other questions in which more than choice might be correct, and you get credit for both choosing the right answer(s) and not choosing the incorrect answer(s).

But with "at most one correct" questions, it's more of an all or nothing deal. Our calculation of % correct statistics did not, however, "keep up" with this alternative question style. We were still, say, showing 50% correct or 33% correct, even though you got a score of 0.

Which caused confusion for players and just didn't make sense.

So now you are recorded as either 100% or 0% correct for such questions, which is why some of you will see a reduction in your overall correctness numbers.

Cheers, Steven

10 comments:

  1. It appears that along with the change in calculation for those 'only one correct answer' questions, the ranking % changed from being 'in the top 10%' to being 'better than 90% of all respondents'. It was a little confusing to see that my score was above the median, but my percentage was below (or is that above?) 50%.

    ReplyDelete
  2. Hello Steven,
    Ok, this explains what we saw, though, I am not so sure that the previous calculation was definitely a bug...

    1. First of all, the change does not influence the total scoring, and that is already a good point.

    2. Second, if the time used to answer a single choice quiz IS taken into consideration for the total timing, then I think that at least the correctness percentage should reflect this
    "investment" as well, if the score does not.

    I think the issue is worth a reconsideration.

    Maybe the older proposal of going for a different score calculation method for these quizzes instead of the "all or nothing" score could be an alternative.

    Thanks & Best Regards,
    Iudith

    ReplyDelete
  3. It is true that I also changed the calculation of percentile, which I believe is what Anonymous refers to above. We had previously been presenting percentile in an inverted fashion, at least according to Wikipedia: "The 20th percentile is the value (or score) below which 20 percent of the observations may be found. "

    http://en.wikipedia.org/wiki/Percentile

    So the higher the percentile, the better you did.

    Regarding Judith's comment that % correct should reflect timing: weighted score reflects timing. % correct reflects just that: how correct you were. If only one answer is correct and you don't choose it, you are not correct at all.

    Have a fine weekend!
    SF

    ReplyDelete
  4. Hello Steven,

    Maybe my comment regarding the timing was not very clear.
    What I meant to say is that if a player spent "n" seconds answering a single choice quiz and being wrong, then, if ALL those "n" seconds are added to his total playing time, then I think that the correctness percentage also should reflect all the correct choices (that is, in this case, all the incorrect choices not chosen).

    Thus, a player is not penalized twice for the same mistake, both by having a zero score and
    the percentage diminished.

    Since what decides is the overall score, it does not have such a great importance,
    but I think it is more correct.

    Regarding the inverting of the percentile,
    I personally think that the previous one was
    "more speaking" than the new non-inverted one,
    because it put the emphasis on where you rank rather than on which percent of players you leave behind.
    Again, just a matter of personal taste ...

    Thanks & Best Regards,
    Iudith

    ReplyDelete
  5. Hello Steven,

    my point is the following:

    1. When I am able to eliminate some incorrect checkboxes, but not all of them, then I am rated by 0%, the same way like I am completely incorrect (which is not in fact true). If you feel this is OK, then I would like to propose "Two Choice - One Right" questions instead of "Multiple Choice - One Right".
    I have nothing against getting 0 points when I am incorrect, but I am quite against getting 0 percentage when I was not completely incorrect. I think correctness should by checkbox-based, and not question-based, no matter what kind of question it is.

    2. Unfortunately the recalculation of the percentage was done with the retroactive effect. By my opinion, "Don't touch the past, affect the future" is a good politics in this case, otherwise many uttered successes of your players has been invalidated by a single line in the new algorithm.

    All the best,
    voltrik

    ReplyDelete
  6. Voltrik,

    The weighted scores and rankings were not affected at all by this change. The only thing that has changed is that your % correct is different.

    I made this change because I got persistent complaints/questions about how you can get a score of 0 while being 33% correct.

    I am not at all certain I can come up with one approach that will satisfy everyone, but I will give it some thought.

    SF

    ReplyDelete
  7. Hello All,

    Maybe you remember that in the past I suggested to apply a different scoring calculation
    for the quizzes in which the wording of the quiz itself "autodefines" it as "single choice"
    ( like "What will be the output displayed ..." ).

    The idea was mainly to avoid cases in which players mark several choices as correct or do not mark any one, in an attempt to achieve maximum correctness percent possible.

    Maybe another idea could be to just simply enforce choosing one and only one choice for such a quiz, but keeping the score calculation the same as for any other "regular" quiz.

    Maybe a "half step back" from the current approach, but one that probably many players will agree with.

    Thanks & Best Regards,
    Iudith

    ReplyDelete
  8. My chief complaint with the all-or-nothing scoring is that of late there seems to have been a preponderance of such quizzes. I understand that it is more difficult to create quizzes with multiple correct choices, especially in contrast to the "what will be the output" class of question. The multiple answer questions seem to more fairly reflect the skill and knowledge of the players. All-or-nothing scoring disproportionately penalizes players for minor misreading of quizzes. An incorrect response to even one advanced level single-answer quiz would likely lower one’s ranking by several positions and possibly eliminate one from participation in the quarterly championship despite consistently demonstrating superior PL/SQL knowledge.

    ReplyDelete
  9. Generally, we do not set advanced quizzes to "one choice only" even if the choices are mutually exclusive. We figure that for advanced quizzes, the effort required to sort out the code and choices warrants partial credit.

    Hopefully I have not violated this rule. I will certainly watch carefully going forward.

    ReplyDelete
  10. I think its a pity that the current scoring method makes it impossible to penalize you more for answers that are more wrong than others. All wrongs and all rights are equal, but in reality some answers even though strictly speaking they are wrong, are still not as wrong as others. Maybe its time to score less absolute and more relative instead.
    This could be modelled by mapping answers with points and calculating the final result as the percentage of points achieved to the maximum points achivable. This way the "better" wrong answer gets you still a higher score than a "worse" wrong answer.

    ReplyDelete