09 October 2010

Criteria for setting difficulty level of quizzes

I received this email yesterday:

Hi Steven, Why was the question on "08th Oct 2010" on an advanced level? I ask because everyone who works with Oracle object types feels this question is more like a beginner question (just initializing and assigning). If you don't mind, can you please share as how the questions were categorized as Beginner, Intermediate and Advanced?

I certainly don't mind. One of the aspects of the PL/SQL Challenge that I like best is its transparency.We make available all sorts of data about the quizzes and the answers people have submitted. This makes it possible for you to give me feedback, help me improve the PL/SQL Challenge, and see how other players are doing.

So, first, thanks for asking about this specific quiz. It should not have been marked as an Advanced question. As to how it got that way, I have generally been assigning a difficulty level of at least Intermediate for any areas of functionality that are rarely used, such as object types.

Your email has me re-thinking this approach, though. It seems like what I should do is rank the difficulty level within the context of the functionality area, at least for a feature as broad as object types. There are beginner questions on object types and, you are right - this is one of them.

So here's what I am going to do:

1. Change the difficulty level on this question to Beginner (everyone's scores will be decreased, but your rankings will not be affected).

2. Be more discriminating about how I set that difficulty level. It may be true that few developers use object types, but it is a part of the core PL/SQL language and has been available for years. There is no reason that we should not all be expected to have at least a rudimentary awareness of it.

Many thanks and please keep that feedback rolling in,
Steven

6 comments:

  1. Helo steven, Thanks for considering my email :-)
    I really wrote that not for rescoring but to know as how the dificulty level is set and more importantly 'I' am afraid of answering that question quickly. I really had a doubt that if we answer it quickly(a question which was marked advanced) it will it be considered as cheating?
    Any ways the response I got is more than what I expected. Thanks a lot again steven.

    ReplyDelete
  2. "everyone's scores will be decreased, but your rankings will not be affected"

    If person A is behind person B by 45 points and person A got everything right and scored 50 and person B got it all wrong and scored 0, person A overtakes person B in rankings. If the scores are reduced and person A only gets 40, they stay behind.

    Even if doesn't affect rankings immediately, if it changes the gaps between players, it will in the longer run.

    PS. It's player's self-ranking of their PL/SQL level that interests me. There's a fair few "intermediate" in the top ten/twenty etc.

    ReplyDelete
  3. Jay,

    I changed the difficulty level because I want to make sure that all my quizzes are as correct and useful as possible. They will be used again in the future in a variety of ways.

    As for a concern about a fast answer being considered cheating, no single fast answer will be interpreted as cheating. Instead, every two weeks or so I look at the accumulated time spent answering. If your time over 10 quizzes is markedly faster than mine, I take action.

    Gary,

    You are right; gaps can narrow. I suppose that theoretically that could have an impact on ranking at the end of the quarter, in a very tight "race." I guess we will all just have to live with that.

    The self-ranking is interesting, because there are all sorts of psychological aspects to it. A person might know PL/SQL very well, but also because of that realize how much there is to know, and so don't want to call themselves advanced.

    In 1.7 (out in a few weeks), you will be able to note in your profile the number of years you've been working with PL/SQL. That will be another interesting stat to use in reports at some point.

    Cheers, SF

    ReplyDelete
  4. I rank myself intermediate in PL/SQL - even though I have managed to reach a high rank last quarter.

    If I consider Oracle SQL only, I would call myself advanced. It's practically daily I code SQL statements utilizing many features of the SQL language. I can understand most SQL statements Tom Kyte write :-)

    I do not consider myself advanced in PL/SQL because I do not use it extensively apart from a means to extend SQL. I have written one application in PL/SQL 9 years ago (that's where I learned PL/SQL), but I hardly ever have the need of coding PL/SQL where I have to consider using features like bulk processing etc.

    So I know there must be many many people out there who use PL/SQL daily and must be more advanced than me. I'm intermediate since I can become much more advanced if I start having projects where I can utilize PL/SQL to the limits.

    That I can reach a high ranking is more because of a sense of logic that helps me a lot - not because I have a lot of PL/SQL knowledge stored in my brain :-) I can score low on a beginner question if the answers are based more on knowing correct syntax. But I can score high on an advanced question if it's more based on being able to understand the logic of a procedure :-)

    I have considered changing my self-ranking to advanced (since I obviously am capable of reaching high rank) but in my own mind I still have a lot of ground to cover before I can think of myself as an advanced PL/SQL coder :-)

    ReplyDelete
  5. I also am an "Intermediate" PL/SQL coder. I have used it for over a decade. I use it almost every workday. I am usually one of the more proficient coders in my workplaces.

    However, I seldom have need to code what I consider to be advanced features: bulk operations, object types, cursor variables, and the like. Usually I grab a book to check my syntax and glance at the samples first. I feel a true "PL/SQL guru" would not need such crutches most of the time. So I do not consider myself advanced.

    Still, I did manage to make the top twenty last week -- though it was top ten before the re-scoring.

    ReplyDelete
  6. Congrats on your top ranking!

    You rarely use FORALL or BULK COLLECT? I am surprised...do you work with large volumes of data? It certainly will have a big impact on performance....

    SF

    ReplyDelete