Since the early days of the PL/SQL Challenge, we have been aware of the possibility of cheating by players. In response, we developed a set of tools to analyze player data to identify patterns of answer submissions that could reflect one or another form of cheating (such as a single person having multiple accounts, or multiple people colluding to achieve high scores).
We have in the past applied these tools to identify players who answer too quickly, and made adjustments to their scores accordingly. When we apply the adjustment, we notify the players that:
It is certainly quite possible that a person answering so quickly (and correctly) is cheating by taking the quiz under a different account, figuring out the correct answers, and then submitting the answer under their "real" account. It is also possible that they knew that subject very well or simply guessed. Rather than automatically reject such answers, we analyze player patterns over a longer period of time, usually no less than two weeks. We then make adjustments to the timings and ranks of those players who answer, cumulatively, in so short an amount of time that there is no way to distinguish their pattern of play from someone who would be cheating.
Today, it is my distinct non-privilege to announce that I have removed from the website three players who I believe have colluded over many quarters to take a whole bunch of the top prizes in the quarterly PL/SQL Championships. The players are (with their championship history under their names):
Frank Schrader
Q3 2010:12th, Q4 2010:30th, Q1 2011:1st, Q2 2011:5th, Q3 2011:1st, Q4
2011:1st, Q1 2012:13th, Q2 2012:3rd, Q3 2012:2nd, Q4 2012:4th, Q1
2013:2nd, Q2 2013:22nd
Dieter Kowalski
Q1 2012:21st, Q2 2012:8th, Q4 2012:16th, Q1 2013:24th, Q2 2013:16th
Konrad Schmidt (kowido)
Q1 2011:26th, Q2 2011:9th, Q3 2011:17th, Q4 2011:21st, Q1 2012:4th, Q2
2012:1st, Q3 2012:9th, Q4 2012:2nd, Q1 2013:1st, Q2 2013:15th
As you can see, two of our consistent champions dropped dramatically in ranking when in the Q2 2013 championship everyone played at the same time.
After I notified all three players of my determination and asked for a response, all three "people" immediately stopped playing. Only Frank even replied to my request and he protested his innocence. Unfortunately, it was not a convincing protest. The bottom line is that there is no way to distinguish their pattern of play from someone who would be cheating.
Frank and Konrad cumulatively won $2800 in prizes on the PL/SQL Challenge site. This is, of course, a source of extreme frustration for me....but what can you do?
Well, here's what we can do: all rankings have been recalculated after these players' removal. We therefore have in many cases new winners for the championships. I am sorry that we will not be able to issue prizes to these new winners, but at least you will be able to print off and display your certificate of achievement - and these new achievements become a part of your profile and history on the PL/SQL Challenge website.
And - do I really need to say this? - let this be a warning to other players who are cheating or contemplating such: we will continue to enhance and apply our tools to root our cheaters, remove them from the site, and announce their crimes to the world.
We may also make our tools available on the website itself, so that any players can run the analysis and see if they can uncover cases of potential cheating to report to us.
The PL/SQL Challenge (www.plsqlchallenge.com) offers a daily quiz on the PL/SQL language, through which thousands of Oracle technologists demonstrate and deepen their knowledge of PL/SQL. This blog contains posts by the PL/SQL Challenge founder, Steven Feuerstein, as well as comments from players.
I'm also from GERMANY. Frank was a kind of model for me. Too bad...I can't understand such a behaviour. Some, not playing collegues, responsed me in the past to do the same. But that's not the sense of this quiz for me. I use it or my further education and I LOVE it AND I'm very, very thankfully to Steven and the other developers of this quiz!
ReplyDeletest_guitar
Steven,
ReplyDeleteYou have made a brave but necessary move to keep PL/SQL Challenge fair. I support your decision 100%.
Thanks for all the work you're doing for PL/SQL developers community.
Dalibor
I wonder about my ranking in the cheaters' top. This championship was by far the most unpleasant for me - after a 9-hour workday I was not able to read, remember and analyze the complicated pieces of code. I also dropped quite a lot when compared to previous championships. I have never won a monetary prize in these events, but I think that my "behaviour" qualifies as cheater by these standards?
ReplyDeleteI came second in one of those. This news doesn't really cheer me up though.
ReplyDeleteIt takes significant effort to play the contest competitively on a daily basis. You really can't afford to miss even a day. I guess some people got a little too obsessive about it.
Steven,
ReplyDeletesad post, and feeling ashamed, because all three of them are Germans.
Anyway, keep the good work and thanks for all the great quizzes here !
Matthias Rogel
Great work!!! ... It will be better if we learn something by playing the quiz rather than earn something by the way of cheating...
ReplyDeleteslimkask - it wasn't the performance on one championship that led me to this action. It was a long pattern, and relationships between the activity of several players. So, no, your single rankings drop does NOT raise a red flag.
ReplyDeleteMatthias - if it's any consolation, I don't think there were three Germans. Maybe just one one or two. :-(
Steven,
Deletegiven, that, as the above links
https://www.xing.com/profiles/Frank_Schrader17
https://www.xing.com/profiles/Dieter_Kowalski
https://www.xing.com/profiles/Konrad_Schmidt19
show,
Frank Schrader works in Braunschweig ( Brunswick ),
Dieter Kowalski works in Wolfsburg,
and Konrad Schmidt works in Halle/Saale,
and due to their names it seems quite likely they all three are Germans.
For those not familiar with German geography,
http://tinyurl.com/oo68xwq
shows the location of these three towns in Northern Germany
Matthias
Sorry, should have been more clear. I am not certain that there are actually three distinct living and breathing humans. Maybe just two, maybe in actuality just one. One person could pretend to be two people playing the championship at the same time. And while those locations are different, the IP addresses when quizzes were played tended to show something else.
DeleteSteven, have you asked them to return the prizes? They would have an opportunity to show at least a fragment of a moral character.
ReplyDeleteRadoslav, I did in fact make that request, but to no avail.
ReplyDeleteIt's kind of upsetting. I hope Frank was a cheater, because I did not see anything wrong with his performance, some other players that have vanished after the announcement of the "No Cheating policy" were obviously cheaters.
ReplyDeleteIt make me feel vulnerable as well as when I am going on holidays I don’t have anywhere to test my answers so I answer quickly just by knowing enough of the subject or guessing. Does it mean I might be qualified as a cheater?
Dear Steven, dear fellow players,
ReplyDeletea sad day for the PL/SQL challenge indeed. I'm kind of stunned - I'd never have thought that cheating on this scale was possible (since obviously, cheating in the playoffs requires collaboration of at least two playoff participants). Even now, I'm hesitant to believe this, although I guess Steven must have conclusive proof.
Regarding the possibility of making the tools for checking available: I'm of two minds about this. Although I believe in open-sourcing algorithms and software (more eyes=fewer bugs), you would also have to publish all raw results from the daily challenge. This might allow potential cheaters to spot bugs in the detection algorithm or to "fine-tune" their cheating to stay below the radar.
Kind regards,
Frank
Anna, as I stated earlier, it is not a matter of just a single answer or performance on a single championship. It is a matter of analyzing a long-term pattern of behavior that involves at least one other player, and sometimes more. So, no, if you guess and answer quickly on a quiz, you will not be flagged as a cheater. If you start answering really quickly all the time with 100% correctness, and then in the playoff you blast through all obstacles to a 100% score in 5 minutes, well, then, yes, you would come under suspicion.
ReplyDeleteDoes that clarify things?
Frank, I wasn't planning on releasing the algorithms. Just letting players APPLY the algorithms to their choices of players. I would keep the data and code to myself.
ReplyDeleteI love this site too much for testing my database, pl/sql knowledge and learned so many new things, really Great. I understand Steve's frustration.
ReplyDeleteI looked for similar site for java too, but found nothing so far. Steve might inspire someone to start it.
Hello Steven,
ReplyDeletetoo bad for them! It's their loss. My main prize is to learn with all of you and challenge myself everyday. And what a journey has been so far. As an active player, i don't give a damn to where their prizes are. Just keep on forward!
Here's a suggestion: i know you want plsqlchallenge to be free but how about charging a symbolic amount (1$/year ?). Those kind of players would stay away, or am i being too naive?
Best regards,
Bruno Martins
Hi,
Deletecheaters are really bad and shows no real valuation for creating the mass of quizzes and administration of the website.
I think, charging a symbolic amount could some of them get out of the challenge, but this could victimize mass of players, who are not cheating.
My personal motivation for playing is to learn about new or unknown features, so for me prizes are not really necessary. The prizes are a nice gift for rewarding good knowledge, but they can be a wrong incentive for cheaters.
Steve, I don't know if I remember correct: was not planned a change in the prices to reduce the costs of website? Probably, the prices could be reduced to maximal 10$ for the best or better, no distribution of any prices.
Without monetary prices it is only played for honour, which is sufficient for me. This will not stop all cheaters, but I think, most of them will not invest much effort without monetary incentive.
Best regards,
Tobias
Bruno, I like your idea and have entertained it in the past, in various forms, but I tend to think that MANY players would stop playing to avoid giving their credit card info for something that is free, and something that is not really necessary for them.
ReplyDeleteBut perhaps I will put up a poll about this.....
Hi Steven.
DeleteIn your poll on payments it's not very clear per what we are voting to pay. Is it per Quiz? Per Week? Per Month? Per Quarter? etc or it's once off? Because my answer is different depending on the above.
Best Regards
Anna
Here is what former Chess World Champion Garry Kasparov thinks about cheaters:
ReplyDeletehttps://twitter.com/Kasparov63/status/387388445779185665
I don't know if he refers to this case here.
Matthias,
DeleteI think he talks about the chess players who cheat during the game by consulting a computer, using their cell with or without help of an accomplice.
Hamid Talebian
Thanks, I will fix that and "re-issue" the poll.
ReplyDeleteHello Steven, All,
ReplyDeleteI fully agree with Gary Myers's post above, in that it takes a high degree of commitment
and not a small effort to play daily, along several years, and sacrificing a lot for not missing any quiz.
My personal feeling was always that the Championship Playoff ranking in the Top3 was usually celebrated with "a little bit too much drums", and this even without any relation to the money prize
that accompanied it up no now.
If we look back at the first few Playoffs played, then along all these years the Playoffs have become
more and more difficult with the time ... so, if ranking on a low place in a Playoff or even in
several ones would always mean cheating, then most of the Playoff participants, including probably ALL the winners up to now could easily be categorizable as cheaters, exactly as Siim Kask said.
When I said "too much drums", what I wanted to say was that, again, in my opinion,
the Playoff is maybe the kind of competition which reflects in the lowest degree the long-time commitment and devotion of a player to the PL/SQL Challenge ... at the other end, in my opinion, stands the "Lifetime ranking", which goes almost unnoticed when compared to the Playoff ...
Maybe this "too shining" color of the Playoff became kind of "naturally attractive" for cheaters ...
I am confident, however that, as Steven said, the algorithms implemented are strong and reliable enough for not producing "false positives" ... this would be too sad for any Playoff participant
that just happens to rank low, something that, again, may happen at any time, to anyone of us .
Thanks a lot & Best Regards,
Iudith Mentzel
Please please please do not think that a low ranking in a Championship would raise a flag for cheating. That is NOT how our algorithm works, not at all.
ReplyDeleteGreat point about lifetime achievement vs. one Championship, Iudith. I will give that some thought.
Generally, I think the site could be MUCH better at showcasing players in various ways. I have thought about:
Player of the Year
Reviewer of the Year
and so on.
Hi Steven,
ReplyDeleteI really appreciate your effort in removing cheaters from this forum and I assume that all genuine players play ONLY to test and enhance their knowledge. So probably you can consider doing away with cash (Amazon card) prizes and award only Books. I am sure this will never be a motivation for a cheater to play the quizzes and at the same time this sort of prize could be the most precious one for the genuine learners. Probably you may ask the winner to select a book from a list of books (including books on other technologies like Java, Unix) just in case he/she already won an Oracle book and now may want to learn some other technology as well. In this way we can ensure the prizes are actually helping winners to learn even more.
And obviously, I Love Pl SQl Challenge :-)
Thanks,
Shilpa
Yes, we are done with cash prizes. I will look into "broadening" the books that can be chosen as prizes.
ReplyDeleteYou can not only select books at O'Reilly that are on the page that is mentioned in the "winner" emails. I have often written an email to O'Reilly because the book I wanted to recveive wasn't in the list of possible books. In the section titled "Before you request your Ebook please be sure to do the following:" there is one important item in the list:
ReplyDelete"Fill out the form below to submit your request. If the book you are interested in is not listed below please email orders@oreilly.com."
So just forward your winning message with a request for the book you want to recveive to "orders@oreilly.com".
Kind regards,
Niels Hecker
And you never got any complaint from them? They just got you your book? OReilly is so nice....but I am not sure if I want to put that in the message to players - because OReilly has NOT told me that officially. Hate to abuse their generosity.
ReplyDeleteAh, but I stand corrected on one thing: you are NOT restricted just to Oracle books. We SAID that but it wasn't true - there is a Loooooong list of books to choose from.
Hello Steven, All,
ReplyDeleteI just want to mention that once I wrote to O'Reilly and asked about a book that was not in the
list of books that accompanied the e-mail, but was available while browsing the O'Reilly shop site
( as in the "normal" way of shopping on a web site ).
The answer that I received, however, was that "the book I asked about is *not* included in this promotion",
so, from here I understood that we have to choose only from the list of books that accompanies the
e-mail we receive.
The list of books is indeed very long, though, in comparison with other topics,
the number of Oracle-related titles is not very large.
Thanks a lot & Best Regards,
Iudith
Hello Steven,
ReplyDeleteno, I never got any complaint from them. They were rather friendly - the only problem is that it can last a little bit longer until you get the ebook or any reply at all from them.
Kind regards, Niels
Steven,
ReplyDeleteYou deserve a price!
As an alternative you can select 10 charity organisations, the winner can choose what organisation gets the money, his price is that he or she is mentioned in the hall of fame with the price and organisation mentioned.
Regards,
Dik Pater.
The Netherlands.