A player recently wrote the following to us:
I would like to share some thoughts about Playoff rankings. It is not
about my own results, just some thoughts of how to make it more fair. Don't
you think it is not too fair, for example, to answer 100% for 35 mins and finally
lose to one having 80% correctness and a much shorter time (15 min)?
I know you the same system for the daily quiz as in the playoff, and a player can sometimes win with some errors if they are quick enough. But there is no time limit in the daily quiz, so it's OK there.
Regarding playoff, it is all quite different. Existing situation rewards very quick answers. Even if you make a lot of mistakes, you will
probably be among the best. Conversely, if you used almost all the time available, even
100% correctness doesn't guarantee a first place showing.
Why not rank playoff
with a very simple algorithm, like that shown below (unweighted_score is the score based only on % correct, independent of time taken to submit the answer):
select player_name, unweighted_score, time
from
playoff_results
order by unweighted_score desc, time asc
I see many advantages in
this approach. Simplicity, fair, tendency to thoughtful answers. It will
stimulate thinking and not guessing...but time still
plays an important role. Do you see any limitations?
I would very much like to know what you think, especially players who have participated in a number of playoffs over the past few years. First, though, some background:
The quarterly playoff has changed quite a bit from its original form back in 2010. Originally, the playoff consisted of 10 quizzes to be completed in 10 minutes. The idea was that you should have no or very little time to write code to verify answers and certainly not much time to look up answers or consult with others.
This didn't work out so well. First problem is that it turned out to be very hard for me, Steven, to judge the amount of time needed for a player to read and think through a quiz. So many, many players ran out of time.
The second problem was that coming up with ten new and tough quizzes for the playoff was, well, a challenge.
The third problem had to do with a very hard to track down bug that causes errors when players did run out of time.The site was supposed to very smoothly and elegantly notify you that you had run out of time, and save all answers you'd entered. Ah, software...instead, players would encounter a variety of errors and in some cases, we lost answers.
We were never able to reproduce the error. So we took the following steps instead (to address both of the problems mentioned above):
1. Cut down on number of quizzes from 10 to 5. Ah, what a relief!
2. Extend the amount of time in which players can complete the competition. It is now up to 35 minutes.
3. Remove the countdown clock from the page. We theorized that the Javascript needed for the clock was somehow interfering with other processing on the page and leading to the errors.
And how did this work out for us? Pretty well. We haven't seen those errors since, though that may also be because fewer players are actually running out of time. But this also means that players have a lot more time to research their answers, as opposed to relying solely on their own knowledge and experience to answer the quizzes.
I am not terribly comfortable with the current situation in the playoffs; it's hard not to want to give the 1st place prize to the person with 100% correct answers. Yet I am also reluctant to assign prizes by unweighted score, using submission time only to break ties, at least if we give players 30 minutes to submit their answers.
So...what do you think?
The PL/SQL Challenge (www.plsqlchallenge.com) offers a daily quiz on the PL/SQL language, through which thousands of Oracle technologists demonstrate and deepen their knowledge of PL/SQL. This blog contains posts by the PL/SQL Challenge founder, Steven Feuerstein, as well as comments from players.
I suggest changing the time adjustment factor and recalculate all of the playoffs. See who filters to the top and who to the bottom.
ReplyDeleteWhen you come up with a value (whatever that value may be) go with that value.
Alternatively to to blindly looping in incremental trials, arrange the results how you want. Then see which sql generates those results. Then apply that sql to another playoff and see if it returns similarly palatable results.
or- better yet. Post some sample data for multiple playoffs (use real data if you want) and preferred results.
And see who can come up with an algorithm/factor/ordering/etc that best fits.
You may see the pattern to my suggestions - get results YOU want.
While I appreciate being asked for input, a large part of the value in a playoff certificate is that it is signed by Steven Feuerstein.
If you're not comfortable putting your name on the results, then I'm not comfortable publishing my results.
"Not terribly comfortable" is not the same thing as not accepting the results or feeling they are illegitimate.
ReplyDeleteWe may come up with a better algorithm, but certainly the one we've been using for 2.5 years has overall served us very well. I stand by the results and I hope you will publish your certificate with pride. Certainly, no past results will be chagned.
Text-based communication can be tough to parse sometimes. Sean, are you being sarcastic above ("get results YOU want")?
I definitely do not want to jigger the algorithm to get to a particular set of results. I want to use an algorithm that is fair and sensible.
Thanks, SF
Hello Steven, All,
ReplyDeleteWell ... the poster of this suggestion definitely does have a lot from my older ideas regarding the Playoffs ...
There are indeed many aspects that can be discussed related to the Playoffs.
If I am allowed a personal opinion, I would state from the start that, in spite of the fact that the current Playoff formula does allow moving back and forth among the quizzes, this possibility
is probably lost for most players, because probably no one takes the time to first visit ALL the 5 quiz pages, just for "getting a flavor" of the topics and decide in which order he/she
will work on them ... such a visit would also "eat up" time ...
And, then, what happens is that if you are "lucky enough" to master better the topic of the FIRST one or two quizzes, you final score will be much higher than the score of a player who
masters better the topics of the LAST one or two quizzes ...
As I suggested once in the past, an introductory page with the 5 titles/topics of the 5 quizzes ( a one row link for each one ) would be extremely welcome.
Second, as far as I understand, there is no "individual scoring" for each quiz separately, because there is no separate timing for each quiz, and this means that if you spend a very long time
on one of the quizzes, this will probably lower your overall score also for the quizzes on which you were really fast.
All in all, regarding this aspect, I can say that I was "much more fond of" the previous Playoff formula, that of 10 quizzes, with limited time for each one and even without the possibility
of backward navigation.
Of course, it is clear for me that 10 quizzes require more work from your and the reviewers side first of all ( and I really know what are you talking about ! ) but, let's face it, a quiz that should be answered within a time limit of 2 minutes is by its very nature supposed to be much more focused and, in my humble opinion, better reflect the "bare knowledge" of a player,
in addition to the fact that it allows covering 10 topics instead of 5 only, so "larger knowledge" has an additional chance to get materialized.
( a long thread as usual ... will continue ... )
( continued )
ReplyDeleteRegarding the technical issue, as far as I understand, the problem of receiving errors when hitting the time limit was already solved by automatically registering (saving) the current choices made by the player on the current page on which the time-out happens, exactly in the same way as the choices are saved when navigating from one page to the next.
I don't see why is this supposed to be different for the 5 quiz Playoff than for the 10 quiz one, the mechanism should be the same (and for the 10 quiz playoff applied for each quiz when its
individual time expires and the automatic navigation to the next quiz is triggered).
Regarding the calculation algorithm proposed by the player who wrote to you, I definitely see the advantage (and justice!) of attempting to reduce the influence of the "time" factor
in comparison with the "correctness" factor.
Using a simple ORDER BY just puts the "correctness" before the "time", so, as you said, the time only serves as a tie-breaker.
Maybe the "golden way" could be achieved by calculating a "global (scalar) result" ( kind of a well-thought MAP method ) based on BOTH the correctness and the timing, some form of "weighted average" of the two factors, but by still affording to the "correctness" a much higher "weight" than to the "timing" ...
and the relative proportion between the two can indeed be open to the players' debate.
Another idea, maybe somewhat much "further shooting" would be to have 2 types of Playoffs:
a) A Quarterly Playoff, that would be more similar to the previous 10 Quiz Playoff formula (shorter and more focused quizzes)
and
b) A Yearly Playoff, more similar to the current 5 Quiz Playoff formula, the qualification for the Yearly Playoff being of course based on the playing results for the entire year,
with the addition of some kinds of "bonus points" for the qualification(s) in the Quarterly Playoffs of that year and even for the results in those Playoffs, a.s.o.
Couldn't this make life easier for everybody ?
I dare bet that once this topic (re)open, many ideas from the players will continue to arrive.
Last but not least, in the spirit of the suggestion in this post, maybe here is the place to congratulate Frank Schrader
for his 100% correctness in the last Playoff,
as far as I can remember, the first time when somebody achieves such a result, KUDOS :) :)
Thanks a lot & Best Regards,
Iudith Mentzel
Actually no, I wasn't being sarcastic.
ReplyDeleteI agree the results of previous playoffs are still legitimate, all players played by the same rules and win or lose, the scores came out according to those rules. So it's fair and I'm not trying to besmirch previous scores.
I also agree it does seem odd when a perfect score doesn't win. So, something ought to change.
As already mentioned, your name has weight. When you publish the results but include an asterisk (effectively) on 1st and 2nd place, I think it does diminish the shine a little. Again, I'm not trying to say the higher score with lower accuracy doesn't deserve it; but I don't think 1st place should require an explanation.
Also, if you do decide to change the rules, I assume you'll test any new algorithm on existing data and see if it produces logical results or not. If it doesn't then you'll reject it. Which basically means any suggestion will have to pass the "satisfy Steven" test anyway - and that's not bad.
So, I'm not suggesting you actually change previous results; but you can use that data as a test bed to see "what if" those same scores happen in the next playoff.
Would those results, when ranked in some new way, be more palatable?
For instance, change the scoring that the fastest perfect score is the winner, after that, score 2nd place on down by normal weight and timing. If nobody gets a perfect score, then simply score as normal.
Run that on previous results, how do things sort out? Do they make sense or does that simply make 2nd and 3rd place look odd instead of 1st and 2nd?
Alternatively, instead of changing the timing scores as I mentioned previously, change the correctness scores. Same effect but different math.
If you want to ameliorate your own bias: post data, let us generate some sample code and then we vote on most desirable results out of some set of submissions you select.
My thoughts on the issue:
ReplyDeleteOn the daily quiz the time factor in the weighted is (in my opinion) not really so much about rewarding fast players - it is more to discourage using a long time reading documentation, typing the quiz, and so on, in order to get high score. How "big" the factor is (how high time is weighted) must be set at some suitable value that seems fair to most players - some would benefit from a high factor, some would benefit from a low factor. I believe the factor for the daily quiz is about right as it is - pretty much as fair as you can get, much fairer than the US election system ;-)
For the playoffs there is a time limit anyway, so discouraging taking your time is not as important. I think it did not matter as much when you had 10 quizzes in 10 minutes - most players probably used between 8 and 10 minutes and time factor would not change ranking much. It is after we now have fairly long time that the time factor matters more - finishing in 15 minutes versus 30 minutes makes weighted score much in favor of the 15 minute player, even if he has more errors.
The previous playoff I was in, I had the feeling that the 5 quizzes were really tough needing all that time to think. This playoff my feeling was that I could answer in much shorter time (not that I answered much correctly, but I just had to guess more where I felt taking my time to think wouldn't work anyway, I could not deduce as much but had more to rely on knowledge - which wasn't there ;-)
My suggestion would be to use unweighted score for playoffs - since there is a time limit anyway, you don't need time factor to discourage slowness. But instead cut the time limit to 20 minutes. The playoff "time factor" should more be a matter of whether you are able to finish in time.
I think the playoff is most fairly judging knowledge if most players in the playoff finish "just in time" - perhaps by having to rush a guess or two if they don't know. And a few players may finish early, which might prove bad for them if they didn't stop to think it through. And a few players might only answer 3-4 quizzes within the time limit.
That would IMHO encourage using the time limit in the playoff to try and optimize your score in correctnes, and thereby discourage both fast and slow ;-)
From Niels Hecker:
ReplyDeleteI find the actual scoring algorithm good. When I read the comment above I would think that the player with the 80% in 15 minutes has more knowledge in his brain and deserves a higher ranking than the player with 100% in 35 minutes which looks more like he has tried some of the choices in the database.
If the time wouldn't be calculated into the scoring everybody could use the complete 35 minutes to type in the choices to a tool and executes them in a database and so the ranking wouldn't rely on the knowledge but only on the speed of your typing or your ocr-software (
Kind regards, Niels
This comment has been removed by a blog administrator.
ReplyDeleteI think that the scores should still be weighted. If one player manages to get 90-95% in half the time of a player who got 100% I would rate the first as "better". I can't decide how low the quicker player's correctness % would need to be before I thought the second better, but it would probably be in the 80s.
ReplyDeleteI think that the weighting could be reduced however, but only if the total time limit was brought down as well. 35 minutes is quite a long time and gives a lot of scope to type code in a database or look things up. If the time was reduced to 15-20 minutes then it would be harder to do this.
This would still benefit a player who answers significantly faster. For me this shows greater skill.
I think the current algorithm does a reasonable job of rewarding better knowledge/ability. Any scoring algorithm will have some level of "unfairness" (produce some results that just don't feel right), even if it becomes impossible to get 2nd place for 100% correct. However, so long as the scoring algorithm is understood in advance, each player has the option of modifying their behaviour to "suit" the algorithm.
ReplyDeleteCertainly from my perspective, reducing the time available would appear to be the simplest option to reduce the chance of a 2nd place 100%. Having said that, if the time weighting on the playoff scoring is adjusted to make time less significant (reasonable given an overall time limit), that would probably produce a better end result, although it would probably take some fiddling to get it look right with the existing playoff results for sample data.
Hello All,
ReplyDeleteAnswering 90-95% correct in half of the time is indeed a remarkable result, but, I still don't think it is more valuable than the 100%.
Ultimately, we play for becoming better developers for the REAL daily life.
Yes, speed is important, but only if doubled by correctness. If in any of our competitions a 90-95% correctness is remarkable and really worth to be rewarded, in real life a 1% mistake
can be extremely significant.
Maybe far from this debate, but I cannot avoid telling you about a real life case that we experienced recently:
Users complained about slowness of some certain operations, involving a few very specific SELECT statements, which were running satisfactorily fast in 2 of 3 production databases, but excessively slow in the 3-rd database.
After 2-3 days of checking Network, EDI traffic, System issues, tracing, a.s.o, they concluded that the problem was solved and everybody was happy ... This was about 2 months ago ...
In parallel with their checks, I took one of those SELECT statements and spent a few hours with it, checking data statistics and understanding the business problem (not being one of my own applications) and then I knew where the real problem was residing, of course, without having any one asking
for my opinion ... But ... problem declared solved ( and "prizes awarded", to be "in tone" with our debate ).
Now, after two months, the exactly same complaint arose again, from the same place, in the same context ... and, this time, the answer of the DBA team which checked the issue was that
"Oracle has different statistics in the different databases (which is true, of course) and therefore it uses different executions plans for the same statement (also true)
and ... the statement is a complicated SELECT, involving understanding much application logic (also true) and in summary the DBA team cannot do anything more about it", issue closed.
If only the invested time matters (and it does matter indeed !), then probably the final decision would be to leave the issue as it is, because no one would allow any one else to start and spend a lot of time ( most probably unknown from the start ) to try and solve the problem ...
Yes ... but the problem is STILL THERE and it still continues to annoy the users, to create long processing queues, a.s.o.
I am the one who usually does not make any time calculations, I simply took the problem and insisted (for myself, without any one asking me) to understand why things work as they do ... aka ... badly ... and did not stop until I found the real reason.
Yes, it required some research time, but if we ultimately add up all the time that was spent checking things that proved not to be relevant for the issue, sending e-mails back and forth
WITHOUT solving the problem, I think that all the time that I invested was fully justified ( for my own knowledge anyway ! ).
What I wanted to illustrate with this example was that in real life, in most of the cases, the TIME factor matters much less than the CORRECTNESS .
If a problem really exists, then you should solve it, no matter how much time that demands.
That specific SELECT statement was in fact correct, in the sense that it returned the expected result, but was it "100% correct life-wise" ? Surely not !
And its 90-95% correctness was NOT enough for the real life !
( will be continued )
I agree with Kim's statement above, that probably you have no chance to answer correctly in a Playoff a quiz for which you have no idea about that specific feature and have to look it up
ReplyDeletesomewhere ... typing is also not a chance when you have a very short time ... and the code is not simple and short ...
But, even if you do master a feature, you need some thinking time for comprehending correctly a usually not every day scenario, without a full understanding you can really only guess.
I think that the Playoff loses a lot from its value if you have
to decide to begin "to guess" just for running to the next quiz (one that you may master better), for avoiding to remain entirely unscored for a quiz/topic in which you can be really 100% correct.
Managing time is important, but for being able to do this in a reasonable way, you SHOULD KNOW from the start ALL that you have to accomplish, that is, ALL the quizzes, so that to divide your time in the most efficient way, just like in the real life, when you know ALL that you have to do.
Of course, without any knowledge, you cannot do anything correctly, neither in the Playoff nor in real life ( back again to the case of that SELECT statement, about which no one did anything because they did not posses the necessary knowledge and past experience ... which also means time invested ) ...
But I think that scoring 0 in a Playoff for a quiz which you even did not arrive to read because of not having enough time is something that should be avoided by any means.
And in this sense the 10 quiz Playoff formula was definitely better.
Regardless of who wins the Playoff by applying one algorithm or another, I still think that for making the Playoff results valuable indeed, they should somehow reflect a player's knowledge also, not just his speed and ability to act under time pressure.
Being both correct and fast is of course ideal, but, if it comes to choosing one of them only, real life proves us every day that correctness is THE one who matters most (not to say ONLY !), being just fast (and, deciding "to guess" only instead of "to think") usually DOES NOT solve your effective problem and then the fact that you did not invest time in it becomes meaningless.
I'm sure that the debate will not stop here, just like real life ...
Thanks & Best Regards,
Iudith
P.S.
Maybe the U.S. elections algorithm was not that bad after all ? :) :)
Firstly, I may be biased because I'm one of those who goes fast at the expense of correctness. I've also come first in the Quarterly Challenge once (and lower down the rankings other times).
ReplyDeleteTime managing in the playoff is tricky. You start assuming the first question is 'typical' of the others, but you might find that it is significantly easier or harder than the last. Not necessarily for everyone, it just may be a topic you know well or not at all.
Five questions allows some scope for breadth of knowledge, but there is that element of chance about whether you get topics you are familiar with. Given a different set of questions the results would differ. There's no 'right' answer about who's best. There's probably a whole bunch of equally good PL/SQLers who, because of any manner of reason, don't compete often enough in the daily challenge to make the playoffs.
To me the Quarterly Challenge serves as an extra safeguard that, like the 'too fast answer adjustment', stops people who might game the Daily Challenge. As long as it succeeds in that, I don't believe any player is significantly better than any other.
In my (somewhat biased) opinion, scores should be still be weighted. If the time were used only as a tie-breaker among players with the same percentage of correctness, this would favour extensive googling and trying things out on your database, whereas the PL/SQL challenge should reward players for their *knowledge*, not for their typing / googling skills.
ReplyDeleteI'd suggest two modifications, though:
- use only advanced quizzes in the Playoffs - this would prevent people (at least me :-) ) from getting confused by the playoff rankings, where players with (apparently) same % correctness were not ranked by time (due to the fact that some players got the one intermediate quiz right and some one of the advanced quizzes)
- cut the time limit. 35 minutes is too long, 20-25 should be enough
Kind regards,
Frank Schmitt
Kim Berg Hansen makes a good point - the play-off has a time limit so the time weighting on the score is less relevant.
ReplyDeleteIn a playoff, maybe time taken should only used when scores are tied.
I can't help but feel that an IT quiz should value accuracy over speed - would you prefer code that is 80% correct or 100% correct but took longer to create?
I do believe that the time should be included, but perhaps with less weight so that it would be difficult for someone with 80% to be ranked above someone with 95-100%. Try using 1/2 the current weight of the time taken and see the results (don't publish, leaving past results as is).
ReplyDeleteI would think targeting a bump of about 5% with the time factor would be OK (95% could beat 100%, but 90% could not?).
For me, there's a clear distinction between a quiz and a real-life situation.
ReplyDeleteIn a quiz
- I "work" alone
- I test my knowledge
- I am curious to learn about new things / features
- I am not too worried about getting something wrong (since it's only a playground); on the contrary, I *expand* my knowledge by getting things wrong and checking the correct answers afterwards
In real life (as a professional SW developer)
- I always double-check my decisions / code (if possible, by consulting with a colleague)
- I mostly rely on well-known features
So for me, a quiz and the "real" world are two completely different things, and therefore no need to value 100% correctness over speed in a quiz, but YMMV.
Kind regards,
Frank
-
From Viacheslav Stepanov
ReplyDeleteI think the time should only be used as tie-breaker. Let's consider weightlifting or highjumping for example.
Is it possible that the winner is the one who lifts less / jumps lower but does it quicker?
When two or more results are the equal, then, and only then, some tie breaking should be used. In our case time is perfect as tie breaker.
When I see 2 players (80% in 15 minutes and 100% in 35 minutes) I think the second has more knowledge in his brain and deserves a higher ranking than the player with 80% in 15 minutes which looks more like he was just lucky.
If a player clicked the "no correct" option on each question as fast as they could, he/she probably could finish the quiz in about 10 seconds and would have a 61% accuracy for a final score of 2129.
ReplyDeleteThat's not a top score, but a race of blind guessing would outscore a fourth of the participants, all of whom are probably pretty good with pl/sql. That does seem to indicate time is overly weighted.
I don't know what the best weighting would be, but it does seem odd to me that someone can use a strategy of "I may not be right, but at least I'm wrong really fast"
I like the idea of shortening the test time a little bit. 25 minutes for 5 Advanced questions seems like adequate time pressure. In addition maybe skew the time factor to 1 point per 5 seconds instead of 1/3.
Hello All,
ReplyDeleteAs long as time DOES matter in the final ranking, I really don't understand what's exactly the point of shortening the overall playing time.
The only effect of this would be that players who are slower (like me) would miss more and more the chance to at least visit each quiz before the overall time expires.
I don't think it is fair to receive a ZERO score for a quiz that you haven't even read !!!
If somebody already compared this to the sports area, it is like running alone after others were cancelled from the start.
If the time matters, then the faster one will always win if he is also correct enough, so our problem here is to try to agree upon what do we mean by "correct enough" combined with
"fast enough".
If someone is fast and self-confident, nobody stops him from pressing submit after 10 minutes only or even less, regardless of the total time being 20, 30 or 35 minutes.
He will of course have a better chance to rank high, but for others, even if they will not rank in the top part of the list, they still deserve a chance to acquire the maximum correctness
that they are able to, because ultimately correctness is what matters for life.
Along my many school years, at all levels, I was always taught that in any exam, if you still have time left, then use that time to read once again your answers and make sure you are as correct as you can be, instead of finishing it too early and being wrong.
I totally agree with Sean in that pressing "No choices are correct" and finishing in 10 seconds is indeed a problem ... and it is the same problem as the one we have in the daily quiz
with marking more than one choice as correct, "to be on the safer side", when the quiz text itself only allows for one single correct answer (even if not defined as a One choice quiz).
The idea which several players raised, namely to allow for a "maximum difference" in correctness between two players (like for example a 5% difference) before allowing the time factor to become decisive seems to me a good and reasonable one, though I don't have yet the effective SELECT statement for it ... but start to think about it ...
Thanks a lot & Best Regards,
Iudith
I think Iudith and Gary Myers both have very valid points, and, they are direct opposites when it comes to "playing" for the championship. One takes advantage of time and the other spends as little time as possible. Both, I am convinced, are brilliant PL/SQL developers yet they achieve significantly different scores.
ReplyDeleteAdding a notation of "brilliant" to the scoring algorithm may prove difficult, even for Steven ;-) so what to do?
Thinking real world scenarios, I would much rather see 99% correctness in 10 minutes than 95% in 5 minutes, and I believe the algorithm should better reflect that!
Going for top speed is a bit like gambling. Top score for correct answers vs. nothing for wrong answers. If you have general sound knowledge of PL/SQL, you can be quick and still get a fair number of correct choices which (due to top score) will outweigh the wrong ones to an extent that players who take their time and get more correct choices will lose.
I do not have a ready-to-go solution, but I do believe that some kind of time based adjustment is warranted.
Mike
I have been watching the discussion from the beginning and a lot of good points have been made. Let me share my point of view.
ReplyDeleteUsually I play with correctness in mind. That is clearly visible in lifetime ranking, where no one can beat me in the time spent on solving quizzes (at least in top 10). ;)
So someone can think that I will vote for the correctness as primary factor and time only for tie breaking. However, I know that the time factor is also important (especially in real life) so it should also be one of the main ranking criteria.
Consider following example: 2 players end up with scores 95% (in 30 minutes) and 94% (in 10 minutes). Using the time only to break the ties the first one would be the winner. Do you think it is correct? For someone yes, for me no.
I share the same opinion as Iudith on shortening of the time limit. It will not please me much when I will not have chance to go through all the quiz questions. I know that I will be "punished" for taking extra time to think and validate my answers, but that's that same in real life and I accept it.
It is all about the good balancing of current system. If we want to favor correctness, then we can decrease the amount of point taken for each second of play.
Regards,
Michal Cvan
Hello All,
ReplyDeleteMaybe an idea for a ranking select, using a results table similar to the one in the first post:
SELECT * FROM
(
SELECT p1.player_name, p1.correctness_pct, p1.unweighted_score,
p1.weighted_score, p1.time,
( SELECT COUNT(*)
FROM playoff_results p2
WHERE ( p2.correctness_pct >= p1.correctness_pct - 5
AND p2.weighted_score > p1.weighted_score )
OR ( p2.correctness_pct >= p1.correctness_pct + 5 )
) AS count_preceding
FROM playoff_results p1
)
ORDER BY count_preceding ASC,
weighted_score DESC
/
I did not test it, just thought of it theoretically.
Here i used a difference of maximum 5% allowed in correctness, before allowing the weighted score to prevail as a ranking criteria.
It is just an example, maybe the 5% difference is not the appropriate one, it would be interesting to play around with the existing data and see the effect.
Also, using such an ordering does not exclude the eventual modification of the calculation algorithm of the weighted score itself, by lowering the impact of the time factor, as suggested by several players.
Thanks a lot & Best Regards,
Iudith
A few thoughts.
ReplyDeleteThe existing formula seems to provide a pretty fair way to rank the results. The formula is known going in, so it is up to the individual to determine how they want to play. Answer quickly with the potential for more mistakes or take more time and be more thorough. If this weren't a competition, I think most of us would strive for 100%, as we couldn't turn in non-functioning code and expect our jobs to be completed. I have tried both methods during the playoffs, with inconsistent results, so I don't know if there is a right way or a wrong way to attack the playoff competition. Somewhat ironically, in my best placing, I did not feel that I had performed any better than previous playoffs, but several things just fell into place. (everyone else seemed to struggle more than other playoffs, resulting in a relatively low winning percent correct; several of the questions were topics I was more familiar with, so I could answer them quickly; the one question I did not know, I was able to rule out one option, then successfully guessed that 2 of the 3 remaining would be correct, so picked all three, resulting in a quick time to answer with only one miss). Would I just guess during a production database recovery or guess that I could introduce a hint to production code without testing? Of course not. But in the framework of the playoffs, that was the smart choice at that time and in that sample, I benefited.
Looking at the playoff results, although there are a few individuals that regularly place in the top 3, for the most part, a large number of different individuals have placed near the top at one time or another and are consistently in the hunt with top 10 placings. This seems to indicate a very balanced and competitive field each quarter. If the scoring were modified, players would adjust their approach to account for it. It might impact who is at the top for a given quarter, but would likely not impact the wide variety of individuals that see their name at the top over time.
I did not participate in the playoffs when it was the 10 questions in 10 minutes. Although I would likely not perform very well in this format, as I tend to take relatively longer to answer, I do see the appeal of this format. Either you know the answer or you don't. It was mentioned that in this format, many players did not complete all quizzes and it was difficult to come up with 10 new challenging questions. Perhaps this format could be modified. Do all of the questions need to be advanced or intermediate? Why not throw in a few beginner questions? Do all of the questions need to be new questions? Perhaps increase the number of questions to 20, raising the time limit to 20 - 30 minutes, but also include some of the questions from the existing question library. Reusing existing questions does not seem out of the realm of possibilities, as answering those questions faster and/or with fewer missed choices than when taken during the daily quiz would seem to indicate that some learning has taken place, and ultimately that is what we are after.
And finally, just for fun. Perhaps we could weight individual question choices differently. Perhaps 50 pts possible for question 1 choice 1, 250 pts possible for question 1 choice 2, 0 pts possible for question 1 choice 3 (or any one of the other choices where I was incorrect), etc. Of course, this choice weighting would have to be determined post-competition completion based on how I answered, such that I would end up number one. I'm not quite sure how to deal with the situation where someone else gets 100% correct and has a faster time than me. Maybe in that case time to answer could be discarded and a tie declared.
Chad
Wow! Thanks for all of your very thoughtful feedback. It certainly would have been nice if a consensus grew out of the conversation, but that was surely asking for too much.
ReplyDeleteAs I read through these, I would think, "Yes, that sounds reasonable: Cut down the time, remove time as a factor." and then "Yes, that sounds reasonable: keep things are they are." and then "Yes, that sounds reasonable: come up with a complex algorithm for when and how we break correctness ties." and then...
I will think more about this, and read these entries a few more times, and then report back as to any changes we will make for the Q4 2012 playoff.
But I won't think about any of this tonite...when my wife and I babysit for our granddaughter for the very first time!