"This is a prime example of a quiz that should not be "at most one" correct choice. More than one concept is being tested; however, if a player is weak in any part then the player receives no credit. The results of the quiz would have been more indicative of players' relative knowledge had each exception scenario been presented as an independent selection."
And we agreed.
But another player then objected to this change, saying: "This is unfair towards the players who consumed extra time to try to answer correctly, for avoiding scoring zero points."
We have set up a poll on the PL/SQL Challenge website to find out how you feel about it. Please take the poll and/or add your thoughts to this posting.
Cheers,
Steven Feuerstein
Hello Steven, All,
ReplyDeleteThere are several issues here.
First of all, I completely agree that everybody should make the maximum effort for answering correctly each and every quiz, and this is what I myself and probably most players definitely do.
Together with this, we cannot ignore the fact
that on changing the quiz type from "one choice" to "multiple choice", this was in fact a free gift afforded to those who (probably) DID NOT invest the maximum time to answer correctly, but rather chose MUCH FASTER one of the (unfortunately) wrong answers, thus receiving POST-FACTUM credit for 50% correctness and a shorter time than many others for 100% correctness.
Such a change exactly encourages fast answering as having anyway the 50% correctness guaranted,
and NOT the effort to answer correctly.
The issue of single choice quizzes was widely discussed and probably will still be.
I was among the first ones who considered the current scoring style for these quizzes as not being exactly the most proper one.
But when it comes to this quiz, the fact that it tests several issues or pieces of knowledge DOES NOT make it yet unsuitable for being a single choice quiz.
Let's just think back one week ...
If we change the quiz type for this last one, then why not also for the 8-th september quiz
about the implicit conversions ?
In that one, each and every conversion was relevant per se, so why not present each one also as a separate choice, allowing everybody's knowledge to be rewarded closer to the reality ?
Instead, no matter how close you were to the correct answer, you received zero points because it was a one choice quiz.
I wonder why at all is the quiz type information published when taking the quiz ?
It can just simply be kept secret and disclosed only with the results.
But, facts are facts.
Regardless of each and every player's playing politics and style, it is a player's full right
to act differently when he knows from the start that this is a one choice quiz, and, as such,
the quiz type CANNOT be changed after having it played, this is unacceptable !
Again, regarding issues like:
a. how is decided to make a quiz single choice
b. other scoring possibilities besides the current "all or nothing"
a.s.o, a.s.o., a.s.o we can continue the debate, and we surely will.
But changing the quiz type AFTER having it played ... is simply not a choice that can be considered, rules can not be changed while playing !
Thanks & Best Regards,
Iudith
My feedback for the quiz of September 13 was not a request for a scoring change, but was instead intended to be an admonition against too easily accepting "only one correct" quiz submissions. In the case of this specific quiz, each of the four exception scenarios should have been presented as an independent choice; however, as structured, the candidate answers are mutually exclusive and changing the scoring approach is inappropriate.
ReplyDeletePerhaps the review process should evaluate "only one correct" quiz submissions based on the the number of independent (though possibly related) concepts covered by the proposed candidate answers and reject those that could be presented easily as multiple choice. Even if only one correct answer is offered, authors should strive to structure quizzes such that multiple answers could be correct. All-or-nothing scoring distorts the relationship between knowledge and ranking since an incorrect answer caused by a minor misread of one part of the quiz is indistinguishable from a total lack of understanding of the quiz's topic. It seems likely that players for whom English is not their first language may be disproportionately disadvantaged by "only one correct" quizzes.
The only case for which I can foresee a need to change a quiz from "only on correct" to true multiple choice is when it was categorized incorrectly because only one of the choices happens to be correct although the quiz is not inherently single choice.
The comment by the player that changing the scoring "is unfair towards the players who consumed extra time to try to answer correctly, for avoiding scoring zero points" is an irrelevant argument. Players should attempt to answer each quiz to the best of their abilities regardless of the scoring rules.
Hello All,
ReplyDeleteTo continue this issue, yesterday's quiz just offered us one more case where making a quiz
"Multiple choice" simply offered free gift points to those that were less than 100% correct.
Though Advanced quizzes are "by tradition" never defined as "One choice", this specific quiz tested a very well defined feature, and not answering it correctly clearly means not mastering that specific knowledge piece at all.
By the way, I cannot understand how can one achieve 75% correctness on such a quiz,
except by marking 2 choices as correct, though knowing that they are mutually exclusive,
which also means ultimately a way of benefiting from the "PRIOR knowledge" that the quiz is multiple choice, and thus, there is a "chance" to increase score by marking more than one choice as correct.
On the other hand, the implicit conversions quiz of 8-th september, just like the PLSQL_WARNINGS quiz in the last Play-Off, were clearly quizzes where one could be closer or further from the right answer, and thus, making them One Choice and scoring ANY incorrect answer with zero points was an unfair punishment for those who were almost entirely correct, by recognizing, let's say, 7 out of the 8 implicit conversions, a.s.o.
So, I only agree with John Hall that reviewers should exercise "a higher severity" before accepting a quiz as "single choice".
This does not mean rejecting such a quiz at all, it can simply be made multiple choice
or even redesigned, together with the author, for still making it suitable to a "One single choice", a.s.o.
Developing a good quiz is not a simple task and care and creativeness should always be exercised.
I generally agree with the statement that only those quizzes should be defined as "One single choice" where ALL the incorrect choices are completely wrong and do not prove any knowledge that deserves to be scored.
As such:
a. The implicit conversion quiz of september 8-th WAS NOT one of these.
b. The quiz of september 13-th is maybe "on the edge" of being one of these or not,
though, in the form in which it was presented, it clearly WAS a single choice quiz.
c. The quiz of september 14-th IS one of these, though, as being an Advanced one,
it was not defined as single choice.
But again, regardless of any criteria by which a quiz is defined or not as single choice, its type CANNOT BE CHANGED after starting playing it.
Sorry, but John's remark above is not correct:
The playing time IS STRONGLY RELEVANT in such a change, because turning a quiz "on the fly" from
single choice to multiple choice comes to reward 50% correctness, and NOT the 100% correctness, exactly the opposite of what is expected.
Again, there is a place to think over a different algorithm of scoring for single choice quizzes, something along the lines of defining a "weight" for each choice, based on its "degree of correctness" or "closeness to the correct choice", instead of the current all or nothing scoring.
Of course, taking care to make a quiz single choice ONLY for cases where it definitely tests
one very specific feature is also a good option, but, as said several times, such a decision can ONLY be taken BEFORE PLAYING THE QUIZ.
Thanks & Best Regards,
Iudith
Thanks, Iudith and John, for your clearly deeply considered comments.
ReplyDeleteAfter giving this some thought, and soliciting feedback from my reviewers, I have come to the conclusion that it is time to acknowledge that I erred.
I should not have responded so quickly to the initial criticism; I should have consulted with my reviewers. And I should generally be very reluctant to make a change like this after the quiz has been taken.
In other words, my actions have served (once again) as an excellent opportunity to demonstrate my humility:
I was wrong and I will correct it.
The quiz will be changed back to "one choice correct only."
The scores will be adjusted for rankings tomorrow (16 September).
I will strive in the future to act with greater deliberation, and also take your many excellent thoughts into account for future quizzes.
Many thanks, Steven
Hello Steven,
ReplyDeleteThanks a lot for this really very impressive
feedback.
"Marking points" when Steven is the goal-keeper on the other side is not a simple and every day issue :):) and makes you really feel that your endeavors to contribute to something useful and to keep it improving are not in vain, which is an important feeling of achievement.
You should really not consider humility, each one of us is subject to erring sometimes,
it just happens to some of us more frequently than to others.
We are all here to collaborate in a constructive way, and, as I said many times, I feel very proud if I am allowed to consider myself one of the PL/SQL Challenge family members.
Regarding the technical issue itself:
I am not aware how long ahead of playing time are the quizzes chosen to be played.
If a long enough prior timetable is available, let's say for the quizzes to be played in the upcoming month, then maybe a specific "quota" of One Choice quizzes can be fixed for that month, and not allow it to be exceeded.
I see that most players feel not very comfortable with the One choice quizzes, myself included sometimes, so I feel like having "to share some of the guilt", because I think I was among the first ones who suggested a different (some kind of weight-based) scoring, for quizzes where not all the choices "weigh the same" in terms of correctness. At that moment there were no One choice quizzes played yet.
Yesterday's quiz (september 15-th) was maybe one good example of a quiz that had one single correct choice, but the wrong choices were perhaps "not equally wrong".
How lucky for the vast majority of players that it was NOT defined as a single choice :) :)
As Steven has said with so many occasions, we developers, like all other human beings and maybe somewhat even more, like to succeed and like to be appreciated when we do, but together with this we are all here for trying to learn more, and, unfortunately sometimes we learn more exactly when we make mistakes...
and I cannot imagine somebody being a competitor for whom the mistakes and the rules that govern the competition are completely indifferent ...
Thanks a lot & Best Regards,
Iudith
Hi Steven,
ReplyDeleteWhile I agree with the above, that the rules should not be changed after the fact, I would like to comment once again on the one answer all or nothing questions. I would like to see them disappear as I don't see how you can have 2 different types of questions in the quiz. Either they should all be like that or none. None is better for those of us that are not the masters of PLSQL, as they give us a little bit more leanience when we answer incorrectly.
For me it is totally normal that all answers have the same weight, even though the question obviously only has one possible answer.
When I took my drivers licence a long time ago back in Iceland, the test was multiple choice, the same way as the plsql challenge is. Some of the questions were obviously with only one possible answer. Yet if you answered that incorrectly, you would only loose 2 of 4 points (or just 1 of four, if you marked the 2 answers that you felt were the most correct).
I ask you therefore to abolish the one answer all or nothing questions and change them back to the way they were before. Have all the questions abide to the same rules.
Regards,
Ingimundur K. Gudmundsson