Daily reminders now going out to players

Starting 2 July 2010, we will be sending reminders to take the daily quiz via email to any player who has indicated in their profile that they want to receive a daily report on the results of yesterday's quiz and who has not yet taken the quiz at the time the emails are sent out.

The results from sending reminders so far is dramatic: for the second day this week, we have over 900 players and today we may well hit 1000 for the first time ever. Go, players, go!

By mid-July, you will be able to specify that you want to receive the daily report on yesteday's quiz and not the reminder. Until then, if you do not want to receive this reminder, you will need to turn off the daily report as well through your profile page. We hope this will not cause any problems for you. If it does, I am certain we will hear from you. :-)

Please add no-reply@plsqlchallenge.com to your whitelist to avoid having the reminder email treated as spam.
In the reminder that went out on 2 July, we did not filter out those players who already took the quiz. We apologize for that oversight.

Players speak out on ranking algorithm and concerns about cheating

We have received a number of emails from players who have expressed concern about how we rank and also about how quickly some players play the quiz. In the interests of transparency, I offer a sampling of such comments below, and then respond, sharing with you some of the challenges we face in coming up with a fair and accurate algorithm for ranking.

"The scoring algorithm is supposedly geared towards answering questions correctly, however in the rankings, there are a goodly number of people answering wrong but quickly above people who answer slower but correctly. This seems counter intuitive to me; should I just guess quickly the answers in order to improve my ranking? Seems like that would work better than spending time making sure I've got the right answer, yet that's supposedly the opposite of what you're trying to achieve with the quiz!"

"See ranking of quiz taken on 01-Jul-2010, some users have given answer within 6 - 20 seconds. Can this be possible? As I had told you that a user needs some minimum time to read question and then its choice and then we have to answer the question, it will obvious that it will take more to read 01-Jul-2010 quiz. We must find some way to avoid this."

"I still have doubt that still competitors may use two ids and from one they see the question and than they submit right answer of that question using another id and time they take is even less than the time to even read the question. So, i request you that, rather to give ranking with more points based on as much less time, you put one minimum time for each questions. So if user is submitting bellow that time limit, algorithm can consider them as invalid competitors or it can be treated separately."

So to sum up:

1. It would be nice to not reward people for "guessing" (answer quickly, worrying more about elapsed time than correctness)

2. Very fast times to answer a question must indicate that cheating is going on.

3. Set a minimum time for the answer to a question, and ignore answers that are submitted in an amount of time that seems to surely indicate cheating.

4. We need to find a way to stop cheating through multiple accounts.

These are all very worthwhile ideas, and I must admit that we have considered all of these and more in the last few months. It is very hard, we have found, to run a quiz like this over the Internet and completely avoid the possibility of some form of cheating. Beyond that, we also find it difficult to come up with rigid formulas, such as a minimum time to answer, that treats all players fairly and isn't easily circumvented.

Consider the idea of a minimum time. This was actually in my original plan as a way to avoid cheating. We would establish, say, 10 seconds as the minimum answer time. The score for anyone who answers in a time below that will be set to 0. And we wouldn't publicize this fact, because then anyone cheating would simply wait until 10 seconds had elapsed and then press Submit.

But how effective or fair would a minimum time be? Consider:

* We feel strongly that we should be open and honest with you about our scoring algorithm. If we hide key aspects of our rules, then you will not understand the ranking and you will feel manipulated by us. Yet if players know there is a minimum time, it can be circumvented easily by ensuring that one always answers in ten seconds or a little more. Can we perform further statistical analysis to identify patterns that might imply cheating (example: player ABC always answers in 10 seconds. How likely is that?) - but I have found myself very reluctant to accuse someone of cheating (and take unilateral action throwing out all the time they spent on the Challenge) based on patterns and inferences from those patterns.

* Sometimes a question generally  can be answered very quickly or a person happens to know that topic very well and can answer quickly - do we then penalize them for this? Generally, I am less certain now than when the PL/SQL Challenge started that a very fast answer time must mean that the player is cheating. It's a big world out there filled with lots of PL/SQL developers with many different kinds of brains. :-)

* The minimum time really should be determined based on the difficulty level of the question and even the specific question (length of question text and each multiple choice, etc.). That then becomes very complicated for us to manage and measure.

Regarding the concern about the impact of fast answer times overwhelming correctness: this makes me uncomfortable, too. But what to do about it? If we reduce the weighting of score by time, then those who double visit (first to get the question, next to submit the answer after taking their time studying, discussing, whatever) can cheat their way to a high score, not concerned about how much they are taking to get to the answer. Having said that, we can certainly take a look at the impact of changing how much the timing impacts the score, to perhaps find a better balance. The bottom line, however, is that (I believe, in any case) that there will always be the possibility of a higher rank with lower correctness.

So what are we doing about all this? How can we ensure that everyone playing has a reasonably high level of confidence that they are being treated fairly and that the PL/SQL Challenge is worth their time to play?

1. We continue to evaluate and apply changes to the scoring and ranking algorithm based on our analysis of player activity and your suggestions. So please do keep submitting your ideas.

2. The playoff (in which everyone plays simultaneously and you really won't have the time to leverage multiple accounts) should help distinguish those who have played fairly and know PL/SQL well, from those who have played quickly but do not have a deep, solid knowledge of the language.

3. Consequently, we have decided not give out prizes or recognition simply for ranking in the top 10 or 25 at the end of the quarter. We will instead award prizes for top ranking once the results of the championship playoff are in.

4. We communicate with any players about whose patterns of play we have concerns. Some of these players have confessed to cheating and we have wiped their answer history clean. Others have explained in great length the strategies they use to play the Challenge, educating us on the wide diversity of the way that human brains can apply themselves to tasks.

5. We have broadened the rules for participation in the playoff to ensure that a player who ha a high level of correctness still has a chance to play, even if their overall time has reduced their ranking. We also offer a wild card pathway into the playoff to avoid discouraging those players who cannot play every day. See the Rules page for details.

Is it still possible that someone who is cheating will push out of the playoff a player who deserves to be there? Yes. Is it still possible for a person who is cheating to win a weekly or monthly prize? Yes,

Will we get better at minimizing the chances of any of this happening in the future? I hope and believe so - and I am convinced that you can help us in this matter, so please do not hesitate to reply to this blog entry or provide feedback through the PL/SQL Challenge website with your own ideas.

Alex Nuijten wins US$500 in first PL/SQL Challenge Live event

On 28 June 2010, over 50 attendees of the ODTUG Kaleidoscope conference participated in the first-ever PL/SQL Challenge Live event. Players answered ten questions, with a maximum of 60 seconds to to answer each question. Alex scored highest with 30 points. Michael Brunstedt came in second place with 24 points and She Rah took third place with 21 points (after winning a tie-breaker contest with two other 21 pointers, Kim Berg Hansen and Shashi. Alex won the first prize of US$500 (provided by ODTUG and the PL/SQL Challenge), while the second and third place winners won $100 American Express cards, courtesy of Quest Software.

Congratulations to all the winners, but especially to Alex, who works at AMIS,writes a fascinating blog, and
was very happy to win this prize, as you can see....

This is the first in a series of PL/SQL Challenge Live events that will be taking place in the months to come. We will announce details as they are confirmed.

New rankings now in place

You will notice big changes in the rankings between last week and this week.

Over the weekend, we put into place a modification to the algorithm by which we score and rank players.

Before going into the details of the change, please note that on the Rules page we say:

"We reserve the right to change the scoring and ranking algorithms at any time; such changes may be applied retroactively in the current quarter. We guarantee, however, that the algorithm will always be applied equally to all players."

We don't make such changes lightly, but in this case it was required - and the result is a much more accurate reflection of the way players compete on the PL/SQL Challenge.

Some background: in the process of analyzing quiz answer data, we noticed that many players were visiting the quiz page more than once in a day. That is, you log in, press Take the Quiz button and then "I'm Ready! Start the Quiz Now" button, at which point you see the quiz for that day. But these players were not submitting their answer at this time. Instead, they were coming back later in the day to go through the same steps and then press Submit Answer.

Of course, in many cases, this is happening because a player might lose a connection or have to "get back to work!" (so sayeth the unenlightened manager). But in several other cases, we have players who routinely "double visit" the quiz page - which can offer a significant advantage, and might even be considered a form of cheating.

Putting aside the issue of cheating for a moment, we decided that we can and should adjust our algorithm to take this behavior into account by changing the way we calculate the number of seconds used to answer the question. Previously, the number of seconds was computed as the amount of time between pressing the "Start the Quiz Now" button and the "Submit Answer" button, in that same session (this will be referred to as TQ below).

From now on, the number of seconds will be computed as follows:

FV = Time of first visit to quiz page
LV = Time of last visit to quiz page

Number of seconds = (LV - FV) + TQ

The downside of this change is that if you visit the quiz page and then lose your connection and cannot return for a long time (or some other similarly innocent scenario), you will receive a greatly-reduced score. If this happens you can contact us via the Feedback button and request "forgiveness" for that gap in time. We will likely make the adjustment if it happens only very rarely.

The upside is that the score now much more accurately reflects the time that a player takes to figure out the correct answer to a quiz.

We hope you agree with these changes; if you have any questions of concerns, please reply to this blog or provide feedback through the PL/SQL Challenge site.

Advance notice of upcoming change in scoring/ranking

We are just coming up to the end of the first quarter of the PL/SQL Challenge. Now that we have gathered lots of data from thousands of quiz answers over three months, we have started doing intensive analysis of that data, looking for various patterns.

One consequence of our analysis is that we are changing the algorithm we use in scoring and thus ranking. We will publish a full explanation on Monday, but we wanted to give you advance notice of three things:

1. The #1 ranked player for the week as of Friday may not be the #1 ranked player for this week come Monday, so that prize will not be awarded until early next week.

2. You can expect to see major changes in the ranking on Monday, so don't be shocked. Or, rather, feel free to be shocked, but at least not surprised.

3. Looking further ahead, we plan to hold the championship playoff in the middle of July, not immediately after the end of June. We need time to, quite honestly, finish implementing the playoff process and also to identify all participants in the playoff.

Thanks so much for your patience as smooth out the functioning and accuracy of the PL/SQL Challenge. We know that your rankings on the website are important to you, and we are doing everything we can to ensure that your ranking accurately reflects the effort you put into answering those quizzes!

Warm regards,
Steven Feuerstein and Finn Ellebaek Nielsen
The PL/SQL Challenge
http://www.plsqlchallenge.com

Questions raised about 21 June Quiz - what do you think?(409)

The question was: "I declare a nested table or varray of numbers. I then want to assign the number 10 to index value 1, as in:

my_collection (1) := 10;

"Which of the statements correctly describe a step I must take after declaring my collection and before I can successfully execute the above statement?" And we at PL/SQL Challenge claimed that the only correct answer was: "You must initialize the collection by assigning it a value returned by a call to a function provided by Oracle that has the same name as the type from which the collection was declared." Two players wrote to question this viewpoint as follows: 1. "Hi, I disagree with your answer to yesterdays quiz. You've checked the "You must initialize...." answer as correct. I think it's not a complete answer, because it lacks the the use of "extend". I can bypass using extend by using a dummy value in the constructor. But that is IMO such a bad practice that I don't consider it a correct way to do it." 2. "Do we must initialize nested table? Yes. Do we must do it by calling default constructor? No. We can call another function returning initialized collection.We can construct a collection using bulk collect. So we indeed do not must call default constructor.Although it's a typical way we have more choices.That is why i consider your answer literally incorrect." Here is our answer to these objections: 1. We never said that calling a constructor function was the only thing you had to do before you could execute the assignment. It is, certainly, "a step I must take." The player is right; a call to the EXTEND will almost always be needed as well. But that doesn't make the choice incorrect. 2. The player is right. You could call your own function to initialize the collection; you could also avoid the need to call a constructor by populating the collection with a call to BULK COLLECT. Having said that, please refer back to this assumption which appears before you take each quiz: "The session and the environment in which the quiz code executes has enabled output from DBMS_OUTPUT, and can reference only those datatypes, programs and database objects defined in the context of the quiz or are available in a default installation of the Oracle instance." This means that unless we explicitly say that we have created in the context of this question another such function, it does not exist and you cannot assume its use in the answer. As far as BULK COLLECT goes, in general, that will allow you to avoid the need to call a constructor, but in this particular case (performing a scalar assignment), that does not apply. So we believe that while these are interesting and informed questions to raise about the 21 June quiz, all your answers were scored correctly. What do you think?

PL/SQL Challenge server was down from 15:45 to 16:30 UTC

We were notified that the server hosting the PL/SQL Challenge was experiencing technical difficulties, starting at 15:45. The issues were resolved and the site back up at 16:30 UTC.

If the PL/SQL Challenge is unavailable for at least four hours, then the results of the quiz for this day will not be used to compute rankings.

Since this outage was shorter than that, today's quiz results will be used in ranking.

Our apologies for any inconvenience,
Steven and Finn