A player wrote the following after seeing the answer to the quiz:
Please pass on my thanks to Iudith Mentzel, this trick question caught me! I didn't even notice that the table name was the same name as the schema, so totally missed it :) in fact, I didn't even look at the table name. I learned something knew, so it was worth it.I agree with the player: Iudith deserves congratulations; but I disagree that this is in any way a "trick question." First, congratulations: this quiz is an excellent example of using the quiz to uncover quirky behavior in the Oracle database and warn developers of a hard-to-diagnose problem.
I was staring at the answers for quite some time before answering, because I couldn't work out if it was an Advanced or a Beginner question. I didn't have access to a database at the time to try it out. In the end, I decided it must be a Beginner question because it was dealing in simple concepts (functions called via SQL vs. PL/SQL), and answered accordingly.
Personally, I feel this question is an argument in favor of bringing back the "Beginner/Intermediate/Advanced" classification that we used to be able to see when answering a question.
Alternatively, the question should have drawn attention to the fact that the table had the same name as the schema - then it would no longer be a trick question, but a test of one's knowledge about name resolution in SQL vs PL/SQL.
I have run into this scenario several times as I supported the Quest Code Tester tool. A user would complain about the product not working properly and we'd lose way too much time diagnosing the problem before checking the name of the schema - same as the name of a package and BAM!
As for the suggestion that we should have "drawn attention to the fact that the table had the same name as the schema" - I would argue that we did just that. The question starts off with:
"My DBA has created a schema named PLCH in which I am doing my development."
How often does a quiz specifically name a schema? Very rarely. So that should, in fact, have been something of a tip-off regarding the goings-on in the quiz.
I continue to resist the idea of showing the difficulty level of the quiz. In the real world, when you are confronted by a bug or trying to understand some code, you don't get a hint regarding the difficulty or complexity of the problem. The problem is the problem and the code is the code. We dive in and we analyze. So I don't see why we'd provide this information for the quiz.
Your thoughts?
Cheers, SF
I was indeed wondering why the schema was mentioned, but had not the correct idea after it. Stupid me. Don't show the complexity of a question. Real life does not work that way. Sometimes the worst problems (with big consequences) are easily solved and some time the little problems take forever (but then, why bother?)
ReplyDeleteOK by me not to show difficulty level - I agree with your arguments there, Steven.
ReplyDeleteBut I failed this question along with the others because I did not realize that the topic was name resolution. Similar thing has happened to me before and I sort of wonder if question submitters know the question Topic is not shown when the quiz is taken? (Probably they do and I am just not smart enough to deduce the topic on my own :-)
Whether players complain about it I believe is mostly a question of whether you play for the competition or for educational purposes. Competitive players might feel it to be a trick question since the topic was not crystal clear. "Educational" players won't care so much they failed but rather be happy they learned something new.
Like you say, Steven, if you think of the quiz as "you've been called in as the expert to solve an issue" - naturally you wouldn't get a hint as to "topic" of the problem :-)
So the challenge can be "steered" in a more competitive or a more educational direction by the phrasing of the questions. I foresee a pattern of Steven trying to steer towards education and competitive players pulling him in the other direction. Personally I like "a bit of both" :-)
I can see both sides.
ReplyDeleteWhen I did this quiz, I was looking for the trick - it was too straightforward otherwise.
I didn't spot it, but I was right - there was a "trick".
Is it a good thing? Hmmm. Again pluses and minuses.
Do we all have time to go through the wording of the quiz with a fine tooth comb everyday? Not really.
I'm sure that different people have different motivations for playing the PL/SQL Challenge. I only recently started doing so, having been introduced to it by colleagues at my new job.
ReplyDeletePersonally, I do it from a competitive perspective, but for me the competition is either with myself, or between myself and the question setter.
If I get "tricked", as with this question, I can berate myself for not paying sufficient attention, or congratulate the question setter on their deviousness, but I would never feel cheated or that the question was in some way "unfair" because of that.
I'm very much of the opinion that as little extraneous information as possible ought to be provided, whether it be the topic or difficulty level. Maybe even the name of the setter ought to be anonymous until after you have taken the quiz, because I am sure, much like with crosswords or other puzzles, that you get used to the "style" of a particular setter, and this may give a hint as to what to watch out for also.
A player who cannot post to the blog from work sent me the following comments and asked me to post for him. I hope you all have a fine weekend!
ReplyDeleteMy opinion on the topic of today's blog:
1. A reasonable amount of trickiness in quiz is fine.
2. Not showing difficulty level is okay, but it is important that level is set more carefully and consistently. (I will try to explain this below)
3. I partially disagree with your statement that "in real world... , you don't get hint...", I think you have to use whatever hints you have available to try to troubleshoot the issue. Clues can be taken from past issues history, code developer's expertise level, user's or other developers hints regarding issue, or background of others who troubleshooted issue before you. (I know, this cannot be sole argument to show difficulty level on quizzes)
Anyway, I wanted to know whether you agree with below statement or not?
"A nice quiz should be authored in such a way that if a player knows the concept that is being tested in the quiz, then normally one should be able to score higher than those who don't know about that concept."
If you agree with above statement, then I suggest following:
1. Quiz should be made difficult for players to find answer by simply typing and running the quiz. (I mentioned this few times in the past. In this sense, April 12th question was good. I got it wrong but I would have figured out the answer if I knew it was ADVANCED level quiz, it took me several minutes to test and figure out correct answer. It is funny, I did lots of research and found that most people who have been doing well in PL/SQL Challenge blew that quiz. Most of the top scorer for the quiz were guessers, I think.I don't think that is right!?)
2. Quiz should not reward players who take chances or 'type and test' every quiz all the time. (Yes, doing so occasionally is okay, but that strategy should not work all the time). Recent quizzes and its difficulty level has not been consistent. For example:
Beginner Level quiz on April 8th - 53% chose correct answers, Median Weighted Score - 50, Median Time - 776 sec.
Intermediate Level quiz on April 5th - 80% chose correct answers, Median Weighted Score - 448, Median Time - 155 sec.
Do you see the problem here? There are several other quizzes where scores does not justify its difficulty level recently)
3. Difficulty level should be based on how hard it is to answer the quiz correctly, not based on the difficulty level of concept quiz is covering. (If most beginner level developers can find answer to ADVANCED level topic (that they have not heard of or ever used) in 60 seconds by typing code, then that should not qualify as ADVANCED level quiz)
I appreciate your and your team's effort to help PL/SQL Developers. I wish PL/SQL Challenger do even better in future. This has become part of my daily routine, sometimes I feel like I am getting addicted to PL/SQL Challenge
Your reader raises a good point. In the early days you set the difficulty level of the question probably based on "gut feel", but now that you don't show the difficulty anyway until after the quiz has been finished, you could defer setting of the difficulty level until after the quiz is over. That way, you could set the difficulty level according to the percentage of players who got it right.
ReplyDeletePerhaps abolish the concept of 'difficulty' entirely ? The difficulty feeds into scoring so that someone who gets an Advanced question right gets more points than someone who gets a basic question right. But equally someone who gets a basic question wrong doesn't lose too many points (relatively to winners) which doesn't feel right.
ReplyDeleteHello all,
ReplyDeleteYes, the points raised above by both the reader who wrote to you and Jeff Kemp in his comment do have something right.
Maybe it is a good idea to display the difficulty level only after the quiz is closed.
I would not go that far as to determine the difficulty level only based on the number of players who answered it correctly, but rather somehow try to use some coefficient in the scoring calculation based on the number of correct answers, rather than using a fix coefficient.
Of course, the idea of being consistent in defining the difficulty level is always a welcome one, though, most of the unconsistencies seem to have occurred between the Beginner and the Intermediate level,
rather than for the Advanced level.
Ok ... for really very advanced players it would be difficult to admit that there are still Advanced quizes,
that is probably why for the most difficult quizes we see however survey answers saying that "the quiz was too easy".
Also, I would also not think of totally abolishing the concept of difficulty, because this would mean that all the quizes are the same, which is of course not the case.
Maybe even considering more than 3 levels will allow for a finer categorization and less disagreements regarding the level.
The idea of hiding the author's name until after answering the quiz does also look as a very good proposal.
For those who closely follow after the ranking, the name of a top-ranked player as author can be inhibiting from the beginning, before even reading the quiz ...
The credit for being an author remains anyway, even if publishing the author's name is delayed until answering.
Happy Easter & Passover for everybody !
Iudith Mentzel
Thanks for all your excellent ideas and feedback. For now, I am going to leave things as they are - in part because we are focused on getting out V2 with many, many changes.
ReplyDeleteBut I will definitely add these to our list of ideas to consider as we move forward.
Cheers, SF
One other comment I'd like to respond to:
ReplyDelete"Quiz should not reward players who take chances or 'type and test' every quiz all the time."
I don't agree here. Fundamentally, the PL/SQL Challenge is about learning PL/SQL. One of the best ways to do that is exactly what is described above: WRITE CODE.
If a player is willing and able to spend the time to work through the code (type it) to run it and see what happens, then I am very pleased to give them a high score.
And even if I didn't want to, they will do it anyway - and I don't see a way to stop them or to construct quizzes that can NOT be answered by writing code and verifying answers before submitting them.