We will be instituting a new process for the upcoming Q1 2012 championship playoff.
All or some of the players who qualify to participate in the playoff will be required to take three qualifier quizzes.
If the performance on these qualifier quizzes is substantially worse than the player's performance during the quarter (% correct and/or time required to answer the quiz), then the following steps will be taken:
1. That player will be ineligible to participate in the playoff.
2. That player's status will be set to non-competitive.
3. Any awards from the previous quarter will be reversed.
We believe that this qualifier step will strengthen the integrity of the playoff and ensure the best possible results for all players.
Warm regards,
Steven Feuerstein
The PL/SQL Challenge (www.plsqlchallenge.com) offers a daily quiz on the PL/SQL language, through which thousands of Oracle technologists demonstrate and deepen their knowledge of PL/SQL. This blog contains posts by the PL/SQL Challenge founder, Steven Feuerstein, as well as comments from players.
I take it point #3 implies that the qualifier is intended solely to weed out cheats?
ReplyDeleteWell, yes, and not just point #3. If you cannot verify your performance from the quarter, then your performance is suspect, and you should lose the "benefits" of playing in that quarter. Make sense?
ReplyDeleteThat quarter, or the previous quarter?
ReplyDelete(btw - while I agree this measure is necessary, it is sad it has come to this - however the upside is, we get more quizzes :) )
> Does it make sense?
ReplyDeleteYes and no. Of course it's likely that someone has been cheating when the performance is substantially worse. On the other hand it might have hit me in the second play off I participated.
Because of the nature of the available times to play I had to do it during working hours. Just before the play off started a co-worker entered the room and loudly discussed a request for the next release. I could not concentrate on the questions and of course I could not say "Get out, I want to play a challenge" ;-)
That's nothing I might be able to proof, nevertheless I would consider it unfair if I get excluded from playing competitivly.
Yet if I know that this might happen I could try to leave from work earlier and play it at home.
Regards
Marcus
Hi,
ReplyDeleteI like this idea quite much, as I am always surprised on how quickly some people are answering. I would though suggest that you would consider allowing people to explain themselves and in some cases regain competeitive status.
Regards,
Ingimundur K. Guðmundsson
Hi,
ReplyDeleteI understand the why (I've seen people with answering times I can't even read half of the question :) and understand the benefits.
But as Marcus says it's sometimes hard to find place and time where nobody or nothing will interrupt you. But with some manual verification this surely can work.
regards,
Thierry.
Hello All,
ReplyDeleteOh poor souls ... only I know what logistics I am managing to avoid situations where anybody could disturb me while playing :)
This is why I always play late in the evening, when hopefully
I can be alone.
This is valid not only for the Playoff, but for each daily quiz
as well.
Back to the issue itself:
I understand that the additional qualification quizzes will have to be answered in a "Playoff-like" manner (at a single fixed time), otherwise the same "cheating method" will work for those quizzes just the same as "it works" for the daily ones ...
Thierry above is very correct: no one can even read the quiz completely, while others already have answered it ...
I often wonder how is the "Expert time" issue working for detecting too fast answers, because I am pretty sure that on each and every quiz at least the 10-20 leading times are much faster than the expert time set ...
It's not easy to play the "police man", but following after the daily results list makes some very specific cases extremely visible even to a simple eye ...
Thanks & Best Regards,
Iudith
Thanks for your excellent comments, all. Some responses:
ReplyDeleteJeff - a disqualified player's status is set to non-competitive, and that lasts into the new quarter. So you lose your awards from the last quarter and you will not be ranked in the current quarter.
Marcus - I can understand your concern, but you will just need to plan a quiet time to take the qualifier (and not all players will).
Ingimundur - certainly, I will listen to players if they have a very sad story to tell about why they failed the qualifiers. But it will be an uphill struggle to get reinstated.
Iudith - these quizzes will not be played at a specific time, but instead can be taken over and must be finished within a three day period. Each player will potentially have different quizzes. I don't think it will be possible for someone who is maintaining multiple accounts to use them for these qualifier quizzes.
Finally, a comment about expert seconds: as you can see in the rankings table, several players maintain a very high accuracy with very fast times - and I don't apply my adjustment algorithm. That's because my expert seconds timing is very aggressive.
I have to accept (and truly do) that there are people "out there" who know as much or more than I do about PL/SQL and related technologies. I would hate to penalize a person who is not only knowledgeable but very strategic about how they take the quiz, and are able to achieve a high ranking.
Thanks again,
Steven
Hello Steven, All,
ReplyDeleteYou are very generous in accepting that other people may "know as much or more" than you do about PL/SQL, which may indeed happen for one specific quiz or another.
But this still does not mean that you (or anybody else) will be able to answer each and every quiz before reading it at all ...
Systematically seeing an answer time of 10-12 seconds,
while merely displaying the web page usually takes up at least that much is anyway suspect ...
It rather looks like "a good training" was already done for that quiz ...
Thanks a lot & Best Regards,
Iudith
>>> But it will be an uphill struggle to get reinstated.
ReplyDeleteyikes! So, one bad day and you lose competitive status "FOREVER"?
I've had runs of perfect weeks followed by down-spikes of 50-0%.
Also, I tend to play faster during the quarter and slower during the playoffs (I've played 3 playoffs though, so my sampling may not be reliable.) If there is a chance for permanent censure (albeit with "uphill" appeal.) I'll be playing slowly to make sure my answers are perfect. I'm nowhere near the lightning-speed of some (generally 2 minutes or more), so maybe the filters wouldn't even apply to me.
What if you qualify but don't play the qualifiers? I've missed 4 quizzes already this quarter, I imagine I might miss a qualifier quiz too. Would absence be held against someone?
Hello All,
ReplyDeleteIt looks like exactly those players for whom such a qualification would have never come into being are starting to worry, and probably without a basis ...
I am sure that the purpose is not to punish anyone for one bad day or another, this can definitely happen to any human and it does happen to most of the "normal" players ...
I think that the purpose is "to detect" those very few and very specific ones who play all the time "at lightening speed" as Sean said, and also "at lightening correctness" for a whole quarter ... and exactly in the one-time Playoff they are suddenly much sloooooower than usual and much less correct,
which is not very easy to explain.
Thanks a lot & Best Regards,
Iudith
I support this move.
ReplyDeleteI think the integrity of the entire quiz will be improved, not just the play-offs.
I'm not sure we can fully appreciate the pressure Steven must face with some of these potential cheating players. I'm confident Steven will already have had practice at distinguishing between those answering quickly on a regular basis, compared with the occasion when the moons align and we know the question thoroughly; have a quiet room to concentrate; it's an advanced quiz; and you don't miss something vitally important - and bang, you're in the top percentile that day with a good sub 30s answer ;-)
From his previous comment it sounds as though he's thought this out well.
No doubt this also happens on many other websites. I look forward to seeing how it affects rankings, out of sheer curiosity.
Steven, will you maintain a quota of players for each quarter play-off? ie - will replacements be sought if some people are eliminated?
Scott
Here's how I am currently seeing these qualifiers work:
ReplyDelete1. If you finish out the quarter playing competitively and qualify for the playoff, then you MUST take the qualifier quizzes - or voluntarily switch to non-competitive status. We will send emails notifying you of the qualifiers, and they will appear in the Take a Quiz table, so I don't think they will be missed.
2. If one or more players are disqualified, then I must recalc the rankings, then run the algorithm again to "fill in the gaps" for the playoff. So, yes, disqualified players will be replaced.
Cheers, SF
oops, I somehow missed this key clause that had me worried above.
ReplyDelete"and that lasts into the new quarter."
I fully agree with the plan to have qualifier quizzes to help ensure integrity of the playoffs. My only concern was the permanency of errors. Since that concern was based on a misreading I have no reservations about this idea at all.
Playing devil's advocate, as implementing the qualifer as stated starts down a very slippery slope.
ReplyDelete1. The statistics majors could speak to this more, but is a three question qualifier statisticly significant, especially with the potential of banishment hanging on the outcome?
2. Have you looked at the outcomes of the previous quarterly playoffs and compared that with your % correct value for the qualifier and what constitues questionable behavior? In all but the Q4 2011 playoff, over half the contestants are below 66% correct (many below 50%), vs a typical 85+ to more likely 90+% correct for the quarter. This represents a relatively large drop for half the field (25% - 40% or more) for a relatively larger sample size (5 quiz playoff vs 3 quiz qualifier). This includes past playoff champions and top-5 finishers with nearly 50% drops for the playoff vs quarter. Will half the field potentially be removed via the qualifier?
3. Why have a separate qualifier determine eligibility? Why not just add 3-4 invidividuals to the playoffs and then apply your eligibility test to the playoff results. If an individual qualifier skips the playoff (scheduling related) you could deem them ineligible and/or give them the opportunity to take the 3 question test to verify their eligibility.
4. What is the purpose of the qualifier and identifying "cheats"? (rankings during the quarter?, missed playoff participation?, hurt feelings?, etc) If the concern is cheating during the quarter (and any monetary awards given out), should all awards be given out at the conclusion of that quarter's playoff, as some of the awards cannot be retroactively taken back? If it is missed playoff opportunity, why not hold back the results for a day and the day after the formal playoffs, give everyone else an opportunity to take the quizzes in the playoff format (it could be scored as a separate question to keep results separate from actual playoff)?
just some thoughts
Hello All,
ReplyDeleteI think that the entire issue of catching the "cheats" was too far "deviated" in the only direction of qualifying or not in the Playoff.
The Playoff was maybe only a good opportunity where the issue became more visible, and, especially the last Playoff,
with its more reasonable playing time.
Regarding the above remarks of Anonymous:
There is much truth in what he stated about the low percentages of correctness for most of the players in the previous 3 playoffs (quarters Q1 to Q3 of 2011).
Let's not forget that the Playoff scenario was significantly changed in those 3 quarters, when introducing the 5 long quizzes instead of the shorter 10 quizzes before.
The answer time being the same, 20 minutes, but the quizzes much more complex, the result was that many players could not finish all the quizzes, often the last one was entirely missed
and this obviously lowered the percentage.
In the last quarter, with a more reasonable playing time, this
was not so much of a problem any more and the percentages became
much closer to those in the quarter.
Exactly this fact rendered more visible the cases of very high
scoring in the quarter (mainly due to the "lightening speed" answers) versus the low scoring in the Playoff (due to low correctness, in spite of the much lower speed/longer time).
I'm not sure I understand completely the alternatives proposed by Anonymous above, but, maybe indeed, raising the number of Playoff participants could eventually be considered.
As I remember, in the last several Playoffs many of those who qualified finally did not participate, for one reason or another, so there were much less participants in the Playoff than the targeted 45 (as per the qualification rules).
Thanks & Best Regards,
Iudith
If you finish out the quarter playing competitively and qualify for the playoff, then you MUST take the qualifier quizzes - or voluntarily switch to non-competitive status.
ReplyDeleteThat's hard, because it means that you cannot not participate without loosing your competitive status. And there might be valid reasons to do so besides cheating. E.g. when I am on holiday with my family then there is a rule: no internet, no computer. Usualy we are wandering somewhere in the mountains or on a sailing trip. Far away from the next WLAN and surely without a computer.
It is a good thing to step back sometimes and look at what is really important but it should not be a reason to loose your status.
Don't get me wrong, I can see the problem and I don't know a better solution, but there might be good reasons to not take part.
Regards
Marcus
Marcus: First, not all players will be required to take qualifier quizzes. Second, if you cannot participate in the playoff, you can still take the qualifier quizzes, right? If your qualifier quizzes are scheduled when you cannot take them, I can change that.
ReplyDeleteAnonymous: I agree that 3 quizzes are not sufficient to establish a statistical certainty. That is not what I plan to do. I believe I have a way for a player to validate their past performance that does not rely on statistics - and I don't feel that I can explain the algorithm without undermining its effectiveness. I am sorry about the lack of transparency in this case.
The point of the qualifiers is to maintain the highest level of legitimacy of the rankings and awards of the PL/SQL Challenge. I hope to further discourage players (if any are actually doing this) from using two accounts to boost their rankings.
Adding to the number of players in the playoff does nothing to potentially REMOVE those players who should not be there.
And of course no one is required to participate in the playoffs. As Iudith noted, a number of players don't actually play in the playoffs each quarter; most of these come from the correctness and wildcard qualifiers, not from the ranking group. Likely they feel they will not do well; I am not sure.
SF
Hello All,
ReplyDeleteI completely agree with Marcus in that no one should be forced to lose his competitive status for not participating in the Playoff for any other reason, but definitely ONLY for cheating, which cannot be restricted to the Playoff alone.
I completely understand the reason behind trying by all means to avoid eventual "cheaters" to qualify in the Playoff, but I don't see the Playoff as the central point in avoiding cheating,
anyway not more central than the daily competition itself.
Cheating is severe with or without the Playoff and should be punished, but without the risk of punishing the completely
non guilty ones.
... and ... yes, I loved the description of Marcus of a "holiday without a computer" :) :) ... aka the good old days :) :) :)
... though, as an old-comer, who has never in his life touched any other gadget except for a desktop computer -- yes, in the 21-th century ! --, I am extremely preoccupied of how "to organize myself" for being able to enjoy a holiday without missing even a single quiz :) :) even in the middle of a lake !
Addiction is addiction and you "don't belong to the party"
if you are not 100% "at duty" !
But yes, I agree, Marcus is the normal type, not me :) :)
Thanks a lot & Best Regards,
Iudith