07 November 2010

You feedback needed: how best to show "you got right/wrong" info for past quiz?

In response to player requests, in the 1.7 upgrade of the PL/SQL Challenge, we now display for each quiz choice detailed information about how you did compared to other players.

The text will look like this:

You got it right
76% got it wrong.

or

You got it wrong.
42% got it right.

In other words, we display the % of people who answered differently from you. I thought this would be a nice way to highlight your relative performance.

One player, at least, does not like this, writing: "While I was going through the results, I noticed that the statistics on how well the others are doing (23% of all players got it ....) sometimes says 'wrong' and sometimes 'right'. It would be more consequent if they always say 'right' (or 'wrong' if you prefer that option). Now I must do calculations in my head (arghhh...) after reading the text and interpret what was the sentence to compare how well the others did on the answer and to see how stupid I was in selecting the wrong answer compared to the others."

Does anyone else have any strong (or otherwise) opinions about this?

10 comments:

  1. I'm most interested in the proportion who got it wrong as an indicator about whether this was a common error/misconception/misunderstanding.

    ReplyDelete
  2. I was frustrated when I realised it wasn't consistent, so if I could have a preference I would report the percentage as the amount who got it right.

    ReplyDelete
  3. It should be consistent, otherwise the numbers mean different things for each answer - always show either "% got it right" or "% got it wrong" - preferably the former.

    ReplyDelete
  4. But it IS consistent, Jeff - perhaps I should have been more clear about that. If you got it wrong, I show you how many people got it right. If you got it right, I show you how many people got it wrong. Very consistent! :-)

    ReplyDelete
  5. Your algorithm is consistent in that it always works the same way for a given set of data/circumstances, but the way that information is transferred to our already-frazzled minds is inconsistent unless we either got everything right or everything wrong.

    Have you taken quizzes yourself that gave its answers in such a fashion? If not, then I'd have to say you've fallen into the trap of "thinking like a developer" instead of "thinking like a user".

    To me, this might be like having a driving rule that says:

    From midnight to noon, green=go and red=stop,
    but from noon to midnight, green=stop and red=go.

    Such a rule would be consistent, but it'd force users to have to think every time they saw a traffic light.

    You're doing the same thing here: you consistently apply your algorithm, but the reader has to think about the answer.

    You might say, "But we want our readers to think." Yes, we do -- but not about the stupid percent of correctness! Let them ponder the questions and the answers, and make it as easy as possible to understand everything else.

    Eschew obfuscation! ( Keep it simple! )

    ReplyDelete
  6. I see - in that case, I don't mind how it is now, but neither would I complain if it was changed to either proposal :)

    ReplyDelete
  7. It involves an unnecessary "if", so it is sub-optimal :)
    Just always show "X% got it right", does not matter how I got it.

    ReplyDelete
  8. OK, I think the message is clear enough. I will always show the % that got it right, regardless of what you did.

    ReplyDelete
  9. Steven,
    why not have a poll on the homepage of plsqlchallenge.com?
    I think, 5 commenters out of some 1500 players this week aren't that representative. I like the feedback as it is, by the way.

    ReplyDelete
  10. Uwe, great minds think alike. I plan to roll out Polls on the Challenge within the next couple of months.

    Consider: a poll is just another kind of quiz!

    You are right, the comments are not representative. I think that I might add a player preference for the display of this information....

    Thanks, SF

    ReplyDelete