10 July 2010

Release 1.4 now live: playoff support, image rendering of quizzes, email reminders and more

PL/SQL Challenge has now been upgraded to version 1.4. The main focus of this release has been to implement the new logic and pages needed for the championship playoff. We will send out an email on Monday with all the details regarding the playoff, including the list of participants and instructions.

Other changed/improved/fixed features are:
  • All quizzes are now rendered as images on the "Take Quiz" and "Past Quizzes" pages. We're aware that this is an issue for visually impaired players using screen readers and we regret to take this drastic measure, but unfortunately it's necessary in order to prevent copy/paste of the quizzes into a PL/SQL editor. Please contact us immediately if this is an issue for you or someone you know.
  • Reminder emails will now be sent automatically and can be configured to your liking from the Profile page.
  • Players can now submit their own Statistics queries: download a zip file with instructions and have at it!
  • PL/SQL Challenge twitter account now available for following. We will use this account to send out notifications to players.
  • Text and bugs on several pages modified and fixed, including the display of the Best Week in the profile summary. 

Mistake confirmed in 9 July 2010 quiz(465)

Sadly, the quiz for 9 July had a mistake in it. In the question text, the header of the function looks like this:
FUNCTION nothing_lower (collection_in IN DBMS_SQL.NUMBER_TABLE)
   RETURN PLS_INTEGER 
 
but all the answers had this END statement:
END lowest_index;
Consequently, none of those function bodies would compile and thus none of the answers were correct. Now, of course, this was not my intention, and I expect that many people who took the quiz focused on the issue at hand: the method used to get the lowest defined index value in a collection. Putting aside my careless error (which I will explain below - it is a classic programmer mistake), one of the choices was supposed to be correct:



IS
BEGIN
   RETURN collection_in.FIRST();
END lowest_index; 
We will, therefore, (a) fix all the code so that this mistake is erased from the repository, (b) change all answers on that choice to correct, and (c) re-score and re-rank. So if you marked that choice as incorrect, your score will increase, and so may your rank. This change will not be reflected in your automatic email reporting on today's quiz.



What made this mistake especially infuriating for me (Steven) is that I had just finished cleaning up the question and choices. It all looked good. And then I decided to change the name of the function in the question from lowest_index to nothing_lower. Did I go back and test all my code after making "one little change"? NO! And it blew up the quiz.



I get frustrated when this happens, and I am sure that you do, too. You play the PL/SQL Challenge to test and improve your knowledge of PL/SQL; not to be on the lookout for trivial mistakes or what seem to be "trick" questions. I feat that if we keep publishing quizzes with mistakes, eventually you will give up and spend your time elsewhere.



So...I am going to institute some changes regarding the review and approval process for quizzes. First and most important, an excellent Oracle technologist with whom I have worked over the years has volunteered to review the quizzes. Wow, am I happy! Having another pair of eyes is so important when writing, well, anything, but most definitely when writing code. This step alone should greatly decrease the chance of a mistake sneaking out to you.



Second, I am going to start following a strict checklist for every quiz. Here are the current items on the list:

  • Ran all code, verified that there are no unexpected errors or behavior that differs from what we claim in the question.
  • Each choice is a self-contained assertion. Avoid ambiguity with question and "correct".
  • Is not a "trivia" question, but actually tests knowledge of the language.
  • Does not contain double negatives or complicated text that non-native English language would struggle over.
  • Does not contain "absolute" words like "never", "always", "none", "impossible", etc. They almost always (!) lead to trouble, because so many of you are excellent at coming up with "outlying" scenarios that can provide an exception to almost any hard-and-fast rule.
In other words, among things I hereby promise to you that I will never publish a quiz before running every single piece of code that appears in the question. These steps do not guarantee that we won't make mistakes in the future, but I intend to turn them into extremely rare occurrences. If you have any other suggestions for how I can avoid mistakes or if you know of someone who might want (and be qualified) to review quizzes (the downside being you then cannot play the daily quiz), please let me know. Please post a reply on this blog or send an email to steven@stevenfeuerstein.com.

08 July 2010

Linguistic confusion on the 7 July 2010 quiz(1248)

Several people have complained about the way this question was phrased. First, to recap, we showed some code and then asked: "Which of these choices contain exception handlers that will cause a compilation error in the block when [handlers] is replaced with the code in that choice?" We then give these instructions (for every quiz):
Check the box next to any choice you believe is correct. If you believe that none of the choices are correct, simply press the Submit Answer button.
One player wrote: About yesterday's quiz, the indications in the text asked to indicate the wrong exceptions, but the column header on the answer table asked for "Correct". This puzzled me, and in doubt, I assumed that the table header was the right thing to do, and I did not check any answer, because none of them was actually "Correct". Today I see that my choice was the wrong one, but I think that the indications in your quiz pages are ambiguous: you cannot set the header "Correct" to indicate a "non-Correct" syntactical construct! And another: Hello, First of all thanks a lot for the idea of PL/SQL quiz. It really inspires me take part in it. Let me comment the quiz published on 7th July (yesterday). I think the question was very confusing and I understood its meaning in absolutely opposite way. I am not English native speaker (I reckon most of people here are not) and that's why I didn't catch the gist of that quiz. Could you please avoid such tricky combinations in future. We are just Oracle specialists and not English language experts. :-) Another person reported that we scored his answer entirely wrong, but I am pretty sure he experienced similar confusion. So, first of all, our apologies for causing this confusion. There can, in essence, be several layers of "correct" that you have to deal with when answering a question:
  • Is the code correct or does it have errors?
  • Is the choice a correct answer to the question?
Having said that, the question clearly asks you to pick all choices that cause a compilation failure. So a choice is correct if it would cause a compilation failure. Of course, I can make all the arguments I'd like to show how, ahem, correct I am - but that doesn't matter if our players are getting confused. We avoid using the words "correct" or "incorrect" in our questions to avoid "overlap" with the usage of "correct" for the choices. That is not, apparently, always enough. We hope to minimize this confusion in the future by changing our instructions to:
Check the box next to any choice you believe is a correct answer to the question above. If you believe that none of the choices satisfies the question, simply press the Submit Answer button.
Now we explicitly ask you to determine correction in relation to the question. Please let us know if you think this helps. We plan on implementing this change in the 1.4 release this weekend.

07 July 2010

Plans for the next week here at PL/SQL Challenge Headquarters

The first full quarter (Q2) of the PL/SQL Challenge ended on 30 June. We have started the next quarter (Q3), but we still need to hold the championship playoff for Q2. We plan to hold this playoff in about one week.

As you are probably aware, the PL/SQL Challenge website is very new; we are still building out the functionality of the website and, in fact, the playoff functionality is not yet on the website.

So this weekend (and we have to upgrade over the weekend, when you are not playing - a very powerful reason, by the way, to not extend the daily quiz seven days a week), we will be upgrading the PL/SQL Challenge to version 1.4. The main focus of this release is to implement the new logic and pages needed for the championship playoff. There will also be other significant new features, such as automated reminders (with the expected opt-out checkboxes on the profile page).

Once we complete the upgrade this weekend and are confident that we can hold the playoff next week, we will do the following:
  1. Identify all players who are elgible to participate in the playoff.
  2. Decide on the date and time for the playoff.
  3. Publish all of this information on Monday, 10 July, along with an explanation of the playoff process.
We will have players from all over the world participating in the playoff, and the playoff must be held at the same time for everyone. It is, therefore, quite possible that some of you will have to play at a very inconvenient time for you. If you cannot play at the published date and time, you cannot participate in the playoff. We apologize for this, but we don't see any way around it.

Finally, I would like to set expectations appropriately. We are moving very quickly with the PL/SQL Challenge initiative and, contrary to what you might believe, we do not have a big team of full-time developers on this project (yet :-) ). So it is possible that even though we test thoroughly this weekend, when the playoff starts next week we run into a problem and all players are not able to complete the playoff.

That would be very unfortunate, but it will not be the end of the world. If we do encounter such problems, we will simply re-schedule the playoff. Have no fear, your answers and your ranking from Q2 will not change.

So....wish us the best for this weekend and look forward to some big announcements on Monday!

Players point out workable solution for 6 July 2010 quiz(462)

In the 6 July quiz, we asked you, essentially, which call to DBMS_OUTPUT.PUT_LINE would allow to display the value returned by this cursor:
SELECT SUM (share_price) 
FROM nasdaq_listings
WHERE profits > 0 AND sector = 'INTERNET';


None of the choices were correct. But about a dozen players pointed out the following: if you do not provide an alias for an expression in the SELECT list, you can still reference the value returned by the query by putting the expression itself inside double quotes, as in:



DBMS_OUTPUT.PUT_LINE (few_and_far_in_between."SUM(SHARE_PRICE)"
I will add this alternative to the answer text for this question. Several players also commented on this technique and the issue of best practices. Greg: "When a test involves a very poor coding practice, as in today's (July 6th) quiz which is centered on an unnamed expression without an alias in a cursor, shouldn't we get points for being unknowledgeable about the subject? I mean, since we always use best practices and all ... :)" Michal: "Hi guys, I unchecked everything in today's quiz. I believe that you can't reference such a column expression by neither of those names. But my point is that I'll never ever need this information. Why? Because not giving an expression a name is a bad practice I do not follow. I would always name it so there is no doubt about how to reference it (of course, I haven't done it from the beginning, it is a result of experience). If you want to teach by showing a bad example and stating tomorrow that it is bad and should not be done then it's probably OK. Although my strictly subjective opinion is that this type of "educational service" should not take this approach. If you're not planning this then in my opinion in this case the challenge fails it's main purpose - to raise the skill level of developers." Here's our feeling regarding best practices: first, we are still struggling with a way to ask best practice-oriented questions in an unambiguous way. "What is the best solution?" is a very different question from "What is a valid use of a feature?" Second, the point of the quiz is not to promote best practices. It is to test your knowledge of the language. I plan to offer other resources to developers for best practice coding. Michal is right, though. When we have code that includes a best practice, our answer should point this out. What is your opinion on all this?

06 July 2010

Mistake confirmed for quiz on 5 July 2010(461)

We have confirmed a mistake in the quiz offered on 5 July 2010.

While the code itself was correct, the comments attached to the names of columns in logtab were not correct. Specifically, I wrote this:

, created_on /* column type is VARCHAR2(30) */
, created_by) /* column type is DATE */

when I should have written:

, created_on /* column type is DATE */
, created_by) /* column type is VARCHAR2(30) */

Ironically (and believe you me, very irritatingly), I did not simply include the CREATE TABLE DDL for logtab, because I wanted to make it harder for a player to copy/paste the code and simply run it to get the answer. The table was already created in my schema, so when I compiled this package and tested the choices, everything was fine. Argh!

If you created the table according to these comments, then any attempt to insert a row into the table would raise an Oracle error (unless your Oracle schema name can be converted implicitly to a date - and what's the chance of that?). So the following choice, which we scored as "correct", is actually incorrect: "Replace NULL with -1403. Execute the block and a row will be inserted into logtab; then the change will be committed." (I will refer to this choice below as BADCHOICE) There was another choice that was scored as correct, and it remains correct, because under that scenario, no insert was ever attempted: "Run the block unchanged. No new row will be inserted into logtab." Many, many players wrote to us, having noticed this discrepancy between comment and actual usage. I congratulate all of you on your close reading of our quizzes. We certainly can't "sneak" anything by you! I also apologize to everyone. It is hard enough when you are presented with a long, complicated quiz. To then discover a mistake inside of it, well, that can be, and I am sure was to many, very frustrating. So this is what we did to fix our mistake:
  • Fix the question so that it no longer has this problem.It is vitally important that our database of questions and answers does not have mistakes.
  • Everyone (321 players) who did not mark BADCHOICE as correct now have their answer changed to "correct".
  • The scores of those same people are increased to reflect this change.
  • Rankings are refreshed.

Player "MarcJT" was the first to report this mistake. Nice work, Marc, and you win your choice of an O''Reilly e-book.

05 July 2010

Guiding Principles for the PL/SQL Challenge and Changes in Scoring/Ranking

We have been spending lots of time and exhausting many brain cells as we fine-tune our algorithms for scoring and ranking to ensure players trust the PL/SQL Challenge and so that everyone competes on a level playing field. We have also gotten many ideas from players, some of whom have suggested that we not include the time it takes to answer a quiz in the scoring and rankings of players. Others urge us to modify our scoring algorithm to diminish the effect of "time to answer."

You will find below an explanation of why we need to use timing. I then offer some guiding principles for us as we make changes to rules, and finally I announce a new rule regarding the treatment of very fast answers.

I hope you find this helpful and that it will enable you to enjoy the PL/SQL Challenge even more than you have to date.

The Importance of Timing in Our Scoring Algorithm
 

We certainly can adjust the role that timing plays in ranking players (several of you have offered specific recommendations), but timing must be incorporated into the algorithm. Without it, there can be no ranking. If rank is determined solely by correctness, then what you will see in the rankings are lots of ties. There simply aren't that many ways for experienced PL/SQL developers to answer these quizzes. With dozens or more people tied at every score level, the rankings would become meaningless.

The amount of time it takes a player to complete the quiz aces, in essence, as the "tie breaker" for people who achieve a similar level of correctness. This means that we must do everything possible to make it difficult to "game" the system through timing. Unfortunately, given the nature of the quiz, that players take it without any supervision, we have no way of knowing for sure if someone is cheating.

That is, it is certainly possible that no one is cheating (any longer, at any rate; two people did confess their cheating to us). It is possible that everyone is cheating. There is no way for us, the organizers of the PL/SQL Challenge, to know, for sure.

All we can do is analyze player activity and identify patterns of play that can be explained most simply by an act of cheating. That is, you could come up with more complicated explanations for how a certain level of performance is possible, but the simplest, most obvious explanation is: cheating.

We need, therefore, to set up rules that discourage players from following those patterns - ranging from rules making certain patterns illegal (if this is possible) to disincentives (such as the disqualification of answers under certain circumstances) to outright expulsion (likely for repeated violations of the rules).

Principles to Guide Our Rules

The rules for PL/SQL Challenge scoring and ranking must:

1. Make sense to our players. They should be intuitive, and obvious. They should feel right and fair.

2. Be easy to administer. If they require a large amount of time to monitor or enforce (daily manual analysis, responding to requests for exemptions, etc.), we will be overwhelmed and unable to apply the rules properly.

3. Emphasize correctness over timing. We don't want to lose sight of the main objective for most players: to improve and to demonstrate their expertise in the PL/SQL language.

4. Address very fast answers. This is a pattern that can most simply be explained through cheating (multiple accounts). It does not mean that players with very low timings are cheating. Our rules will explicitly discourage players from answering more rapidly than we believe is possible without some kind of manipulation of the rules.

5. Offer forgiveness. Many of our players are very serious about the PL/SQL Challenge and their rankings. They realize that missing a day can have a big impact. They also agonize over dumb mistakes that cost them valuable points. We don't want the PL/SQL Challenge to be stressful game, and we don't want players to be discouraged about their prospects if they miss a quiz or have a bad day. So we will introduce rules soon that will soften the blow of missed quizzes or other rank-damaging occurrences.

Immediate Changes to Quizzes

We are going to take the following steps with all future quizzes to make it easier for all players to compete on a level playing field.

1. Design the question and choices so that they must be read fully in order to have a reasonable chance at getting the answer right. You will no longer take a quiz that can be answered with nothing more than a glance at the question and choices.

2. Determine and record the minimum number of seconds that we believe a player would need to actually read and answer the question. This number will not be made public. Of course, this is to some extent a subjective determination on our part. We will, however, be conservative in setting this value; if any of you saw the number, you would immediately agree that it is sensible and acceptable as a lower limit. You will, however, have to trust us.

3. We will also take the quiz and record the number of seconds it takes to do so. This will be the "expert's timing"; we figure that since Steven wrote almost all of these questions and knows PL/SQL pretty well that he should be able to answer these questions just about as fast as anyone else. This number will also not be made public.

4. We will randomly enforce the minimum number of seconds. On randomly-selected days of the month, the score of all players whose timing is less than the minimum will be set to 0. You will not know on which days this occurs until after rankings are performed.

We hope that these changes will establish a powerful disincentive to answer very quickly. You don't know when a very fast answer might result in a disqualification and you don't know what the minimum timing is for a given quiz. So you should take your time, read the question fully, then submit your answer.

This additional timing information also gives us the opportunity to analyze patterns of player behavior. We will be able, for example, to identify players who routinely answer quizzes faster than we believe is humanly possible.

We hope that these principles and rules make sense. We will be updating the Rules page of the website very soon. Please reply to this blog or send an email to quizmaster@plsqlchallenge.com, if you have questions or suggestions.

Regards,
Steven Feuerstein

Player requests more flexibility regarding "weekday" definition

We just received this email from a player:

"Hello All, The initiative is very nice, instructive and entertaining. But there is a small problem with those of us for which week-end means Friday+Saturday, rather than Saturday+Sunday (in Israel for example). And, unfortunately, we cannot always access the computer on week-ends, due to various reasons. So, a most welcome enhancement would be to allow for a few days more for completing each quiz, instead of "closing" it the same day. I'll be grateful if you could consider favourably such an enhancement, just to open the contest for "a wider world" ! Thanks & Best Regards and Many congratulations for the initiative!"

Sigh. I can definitely see how this could cause some problems for players in Israel. I wonder if this is an issue in other countries? Please reply to this blog if you have a similar difficulty. The number of players from Israel does seem low, given the amount and sophistication of Oracle development in that country.

As for what we can do about it: I am really not sure. Allowing multiple days to answer would require many changes in our website and backend. And then it would no longer be a daily quiz.

We are looking at ways to build some "forgiveness" into the system, so you could miss some quizzes and not be penalized quite as badly. We are also considering offering different levels of play (daily quiz, 3 times a week, etc.). If you really could not answer five times a week routinely, you might consider playing at a different level (different "league"). But that is all still in the future.

What do our other players think about this?

Regards, SF

How You Can Be a PL/SQL Challenge Winner

We are updating the Rules page on PL/SQL Challenge today with the following information.

There are many ways to win a prize in the PL/SQL Challenge.

Weekly Winners

Two prizes are awarded each week:
  • Correctness: To qualify, you must have played at least four days in the week and your percentage of correctness must be at least as high as the #1 ranked player that week. The winner is then picked randomly from all qualified players. The prize is a US$50 Amazon.com gift card.
  • Raffle: For every day you played that week, you are entered into a random drawing to win an O'Reilly ebook. In other words, the more you play, the higher the chance you have of winning. We choose a single winner.

Monthly Winners

Four prizes are awarded each month:
  • Correctness: To qualify, you must have played at least sixteen (16) days in the month and your percentage of correctness must be at least as high as the #1 ranked player that month. The winner is then picked randomly from all qualified players. The prize is a US$100 Amazon.com gift card.
  • Raffle: For every day you played that month, you are entered into a random drawing to win an O'Reilly ebook. In other words, the more you play, the higher the chance you have of winning. We choose three winners.

Quarterly Winners

Each quarter, we hold a championship playoff. For more details on the playoff process, please refer to Quarterly Championships in Rules. Here are the prizes you can win in the championship:
  • First prize, highest ranking in championship: US$1000 or another prize of similar value.
  • Second prize, second-highest ranking in championship: US$500 or another prize of similar value.
  • Third prize, third-highest ranking in championship: full set of O'Reilly Oracle ebooks or another prize of similar value.
  • Place in the top ten by ranking (after the first three): prize is an O'Reilly ebook.

Other Prizes

If you are the first person to identify a mistake in a quiz that affects the ability of players to answer correctly (typos, for example, may not apply), then you will win an O'Reilly ebook.

04 July 2010

Change in awarding of weekly and monthly rank-based prizes

We currently award prizes based on ranking with weekly, monthly and quarterly frequency. The quarterly championship has already moved away from awarding prizes based solely on ranking.

We will now do the same for weekly and monthly awards. Instead of awarding a prize based strictly on ranking, we will award the prize by randomly selecting a winner from all those players whose percentage of correctness is at least as high as the #1 ranked player.This approach encourages correctness, rather than speed, which is something a number of players have been urging.

This change in policy will be applied retroactively to June 2010 and week 26 (June 28 - July 2). All winners will be posted and notified by 5 July 2010.