Starting 2 July 2010, we will be sending reminders to take the daily quiz via email to any player who has indicated in their profile that they want to receive a daily report on the results of yesterday's quiz and who has not yet taken the quiz at the time the emails are sent out.
The results from sending reminders so far is dramatic: for the second day this week, we have over 900 players and today we may well hit 1000 for the first time ever. Go, players, go!
By mid-July, you will be able to specify that you want to receive the daily report on yesteday's quiz and not the reminder. Until then, if you do not want to receive this reminder, you will need to turn off the daily report as well through your profile page. We hope this will not cause any problems for you. If it does, I am certain we will hear from you. :-)
Please add no-reply@plsqlchallenge.com to your whitelist to avoid having the reminder email treated as spam.
In the reminder that went out on 2 July, we did not filter out those players who already took the quiz. We apologize for that oversight.
The PL/SQL Challenge (www.plsqlchallenge.com) offers a daily quiz on the PL/SQL language, through which thousands of Oracle technologists demonstrate and deepen their knowledge of PL/SQL. This blog contains posts by the PL/SQL Challenge founder, Steven Feuerstein, as well as comments from players.
02 July 2010
Players speak out on ranking algorithm and concerns about cheating
We have received a number of emails from players who have expressed concern about how we rank and also about how quickly some players play the quiz. In the interests of transparency, I offer a sampling of such comments below, and then respond, sharing with you some of the challenges we face in coming up with a fair and accurate algorithm for ranking.
"The scoring algorithm is supposedly geared towards answering questions correctly, however in the rankings, there are a goodly number of people answering wrong but quickly above people who answer slower but correctly. This seems counter intuitive to me; should I just guess quickly the answers in order to improve my ranking? Seems like that would work better than spending time making sure I've got the right answer, yet that's supposedly the opposite of what you're trying to achieve with the quiz!"
"See ranking of quiz taken on 01-Jul-2010, some users have given answer within 6 - 20 seconds. Can this be possible? As I had told you that a user needs some minimum time to read question and then its choice and then we have to answer the question, it will obvious that it will take more to read 01-Jul-2010 quiz. We must find some way to avoid this."
"I still have doubt that still competitors may use two ids and from one they see the question and than they submit right answer of that question using another id and time they take is even less than the time to even read the question. So, i request you that, rather to give ranking with more points based on as much less time, you put one minimum time for each questions. So if user is submitting bellow that time limit, algorithm can consider them as invalid competitors or it can be treated separately."
So to sum up:
1. It would be nice to not reward people for "guessing" (answer quickly, worrying more about elapsed time than correctness)
2. Very fast times to answer a question must indicate that cheating is going on.
3. Set a minimum time for the answer to a question, and ignore answers that are submitted in an amount of time that seems to surely indicate cheating.
4. We need to find a way to stop cheating through multiple accounts.
These are all very worthwhile ideas, and I must admit that we have considered all of these and more in the last few months. It is very hard, we have found, to run a quiz like this over the Internet and completely avoid the possibility of some form of cheating. Beyond that, we also find it difficult to come up with rigid formulas, such as a minimum time to answer, that treats all players fairly and isn't easily circumvented.
Consider the idea of a minimum time. This was actually in my original plan as a way to avoid cheating. We would establish, say, 10 seconds as the minimum answer time. The score for anyone who answers in a time below that will be set to 0. And we wouldn't publicize this fact, because then anyone cheating would simply wait until 10 seconds had elapsed and then press Submit.
But how effective or fair would a minimum time be? Consider:
* We feel strongly that we should be open and honest with you about our scoring algorithm. If we hide key aspects of our rules, then you will not understand the ranking and you will feel manipulated by us. Yet if players know there is a minimum time, it can be circumvented easily by ensuring that one always answers in ten seconds or a little more. Can we perform further statistical analysis to identify patterns that might imply cheating (example: player ABC always answers in 10 seconds. How likely is that?) - but I have found myself very reluctant to accuse someone of cheating (and take unilateral action throwing out all the time they spent on the Challenge) based on patterns and inferences from those patterns.
* Sometimes a question generally can be answered very quickly or a person happens to know that topic very well and can answer quickly - do we then penalize them for this? Generally, I am less certain now than when the PL/SQL Challenge started that a very fast answer time must mean that the player is cheating. It's a big world out there filled with lots of PL/SQL developers with many different kinds of brains. :-)
* The minimum time really should be determined based on the difficulty level of the question and even the specific question (length of question text and each multiple choice, etc.). That then becomes very complicated for us to manage and measure.
Regarding the concern about the impact of fast answer times overwhelming correctness: this makes me uncomfortable, too. But what to do about it? If we reduce the weighting of score by time, then those who double visit (first to get the question, next to submit the answer after taking their time studying, discussing, whatever) can cheat their way to a high score, not concerned about how much they are taking to get to the answer. Having said that, we can certainly take a look at the impact of changing how much the timing impacts the score, to perhaps find a better balance. The bottom line, however, is that (I believe, in any case) that there will always be the possibility of a higher rank with lower correctness.
So what are we doing about all this? How can we ensure that everyone playing has a reasonably high level of confidence that they are being treated fairly and that the PL/SQL Challenge is worth their time to play?
1. We continue to evaluate and apply changes to the scoring and ranking algorithm based on our analysis of player activity and your suggestions. So please do keep submitting your ideas.
2. The playoff (in which everyone plays simultaneously and you really won't have the time to leverage multiple accounts) should help distinguish those who have played fairly and know PL/SQL well, from those who have played quickly but do not have a deep, solid knowledge of the language.
3. Consequently, we have decided not give out prizes or recognition simply for ranking in the top 10 or 25 at the end of the quarter. We will instead award prizes for top ranking once the results of the championship playoff are in.
4. We communicate with any players about whose patterns of play we have concerns. Some of these players have confessed to cheating and we have wiped their answer history clean. Others have explained in great length the strategies they use to play the Challenge, educating us on the wide diversity of the way that human brains can apply themselves to tasks.
5. We have broadened the rules for participation in the playoff to ensure that a player who ha a high level of correctness still has a chance to play, even if their overall time has reduced their ranking. We also offer a wild card pathway into the playoff to avoid discouraging those players who cannot play every day. See the Rules page for details.
Is it still possible that someone who is cheating will push out of the playoff a player who deserves to be there? Yes. Is it still possible for a person who is cheating to win a weekly or monthly prize? Yes,
Will we get better at minimizing the chances of any of this happening in the future? I hope and believe so - and I am convinced that you can help us in this matter, so please do not hesitate to reply to this blog entry or provide feedback through the PL/SQL Challenge website with your own ideas.
"The scoring algorithm is supposedly geared towards answering questions correctly, however in the rankings, there are a goodly number of people answering wrong but quickly above people who answer slower but correctly. This seems counter intuitive to me; should I just guess quickly the answers in order to improve my ranking? Seems like that would work better than spending time making sure I've got the right answer, yet that's supposedly the opposite of what you're trying to achieve with the quiz!"
"See ranking of quiz taken on 01-Jul-2010, some users have given answer within 6 - 20 seconds. Can this be possible? As I had told you that a user needs some minimum time to read question and then its choice and then we have to answer the question, it will obvious that it will take more to read 01-Jul-2010 quiz. We must find some way to avoid this."
"I still have doubt that still competitors may use two ids and from one they see the question and than they submit right answer of that question using another id and time they take is even less than the time to even read the question. So, i request you that, rather to give ranking with more points based on as much less time, you put one minimum time for each questions. So if user is submitting bellow that time limit, algorithm can consider them as invalid competitors or it can be treated separately."
So to sum up:
1. It would be nice to not reward people for "guessing" (answer quickly, worrying more about elapsed time than correctness)
2. Very fast times to answer a question must indicate that cheating is going on.
3. Set a minimum time for the answer to a question, and ignore answers that are submitted in an amount of time that seems to surely indicate cheating.
4. We need to find a way to stop cheating through multiple accounts.
These are all very worthwhile ideas, and I must admit that we have considered all of these and more in the last few months. It is very hard, we have found, to run a quiz like this over the Internet and completely avoid the possibility of some form of cheating. Beyond that, we also find it difficult to come up with rigid formulas, such as a minimum time to answer, that treats all players fairly and isn't easily circumvented.
Consider the idea of a minimum time. This was actually in my original plan as a way to avoid cheating. We would establish, say, 10 seconds as the minimum answer time. The score for anyone who answers in a time below that will be set to 0. And we wouldn't publicize this fact, because then anyone cheating would simply wait until 10 seconds had elapsed and then press Submit.
But how effective or fair would a minimum time be? Consider:
* We feel strongly that we should be open and honest with you about our scoring algorithm. If we hide key aspects of our rules, then you will not understand the ranking and you will feel manipulated by us. Yet if players know there is a minimum time, it can be circumvented easily by ensuring that one always answers in ten seconds or a little more. Can we perform further statistical analysis to identify patterns that might imply cheating (example: player ABC always answers in 10 seconds. How likely is that?) - but I have found myself very reluctant to accuse someone of cheating (and take unilateral action throwing out all the time they spent on the Challenge) based on patterns and inferences from those patterns.
* Sometimes a question generally can be answered very quickly or a person happens to know that topic very well and can answer quickly - do we then penalize them for this? Generally, I am less certain now than when the PL/SQL Challenge started that a very fast answer time must mean that the player is cheating. It's a big world out there filled with lots of PL/SQL developers with many different kinds of brains. :-)
* The minimum time really should be determined based on the difficulty level of the question and even the specific question (length of question text and each multiple choice, etc.). That then becomes very complicated for us to manage and measure.
Regarding the concern about the impact of fast answer times overwhelming correctness: this makes me uncomfortable, too. But what to do about it? If we reduce the weighting of score by time, then those who double visit (first to get the question, next to submit the answer after taking their time studying, discussing, whatever) can cheat their way to a high score, not concerned about how much they are taking to get to the answer. Having said that, we can certainly take a look at the impact of changing how much the timing impacts the score, to perhaps find a better balance. The bottom line, however, is that (I believe, in any case) that there will always be the possibility of a higher rank with lower correctness.
So what are we doing about all this? How can we ensure that everyone playing has a reasonably high level of confidence that they are being treated fairly and that the PL/SQL Challenge is worth their time to play?
1. We continue to evaluate and apply changes to the scoring and ranking algorithm based on our analysis of player activity and your suggestions. So please do keep submitting your ideas.
2. The playoff (in which everyone plays simultaneously and you really won't have the time to leverage multiple accounts) should help distinguish those who have played fairly and know PL/SQL well, from those who have played quickly but do not have a deep, solid knowledge of the language.
3. Consequently, we have decided not give out prizes or recognition simply for ranking in the top 10 or 25 at the end of the quarter. We will instead award prizes for top ranking once the results of the championship playoff are in.
4. We communicate with any players about whose patterns of play we have concerns. Some of these players have confessed to cheating and we have wiped their answer history clean. Others have explained in great length the strategies they use to play the Challenge, educating us on the wide diversity of the way that human brains can apply themselves to tasks.
5. We have broadened the rules for participation in the playoff to ensure that a player who ha a high level of correctness still has a chance to play, even if their overall time has reduced their ranking. We also offer a wild card pathway into the playoff to avoid discouraging those players who cannot play every day. See the Rules page for details.
Is it still possible that someone who is cheating will push out of the playoff a player who deserves to be there? Yes. Is it still possible for a person who is cheating to win a weekly or monthly prize? Yes,
Will we get better at minimizing the chances of any of this happening in the future? I hope and believe so - and I am convinced that you can help us in this matter, so please do not hesitate to reply to this blog entry or provide feedback through the PL/SQL Challenge website with your own ideas.
01 July 2010
Alex Nuijten wins US$500 in first PL/SQL Challenge Live event
On 28 June 2010, over 50 attendees of the ODTUG Kaleidoscope conference participated in the first-ever PL/SQL Challenge Live event. Players answered ten questions, with a maximum of 60 seconds to to answer each question. Alex scored highest with 30 points. Michael Brunstedt came in second place with 24 points and She Rah took third place with 21 points (after winning a tie-breaker contest with two other 21 pointers, Kim Berg Hansen and Shashi. Alex won the first prize of US$500 (provided by ODTUG and the PL/SQL Challenge), while the second and third place winners won $100 American Express cards, courtesy of Quest Software.
Congratulations to all the winners, but especially to Alex, who works at AMIS,writes a fascinating blog, and
was very happy to win this prize, as you can see....
This is the first in a series of PL/SQL Challenge Live events that will be taking place in the months to come. We will announce details as they are confirmed.
Congratulations to all the winners, but especially to Alex, who works at AMIS,writes a fascinating blog, and
was very happy to win this prize, as you can see....
This is the first in a series of PL/SQL Challenge Live events that will be taking place in the months to come. We will announce details as they are confirmed.
Subscribe to:
Posts (Atom)