I just applied the forgiveness policy to your rankings for the first time. Since the final policy is different from that posted on the Rules page (I will update it as soon as possible), I thought I would share with you what I did this weekend:
1.I did another round of corrections for "too fast" answers. Some people may find their rankings well below what they were on Friday. Remember: take your time, read the questions carefully and thoroughly - and you will not see such a change happen to you.
2. I adjusted the forgiveness policy and applied it as follows:
For up to ten days in a quarter, we will adjust a player's quiz record as follows:
- A missed day will not count as a zero score; instead you will be given a score equal to 2/3 of the average of all your previous answers in the current quarter. In other words, we will pretend you took the quiz and performed at 2/3 of your usual level. I do not give you 100% of your average because that led to too big a boost in the scorings, effectively penalizing those players who did not miss any quizzes.
- A zero score will be replaced with a score equal to 2/3 of the average of all your previous answers in the current quarter. See explanation in first point.
- If you missed or scored 0 points for less than 10 days (let's refer to the balance of those days as "FD" or "forgiveness days"), then we will identify the FD # of days with the biggest delta between your weighted score and the maximum possible weighted score (you had a bad day). We then add to your weighted score for that day 75% of the difference between the maximum possible and your score.
Finally, the adjustments that I made today for "bad days" or lowest rankings are only provisional. I will run these adjustments again at the end of the quarter (you may have even "worse" days ahead), but this will at least give you a sense of how the forgiveness policy will affect you. If I waited till the end of the quarter to apply any of these lowest ranking adjustments, those who missed lots of quiz days would suddenly be ranked well ahead of others who played each day and scored at a similar level.
So....again thanks for your patience as I apply increasingly sophisticated rules to the quiz data, to handle increasingly "subtle" requirements from you, the players.
Warm regards,
Steven
Hi Steven,
ReplyDeleteGreat initiative! (although vacations in Europe tend to be a bit longer than in the US :-) ). Unfortunately, the policy has a dramatic effect on the correctness percentage, since missed days now count as wrong answers, it seems. I doubt that this is intentional. But the general idea of the forgiveness policy is most welcome. Thanks!
Regards,
Rob.
Thanks for pointing that out, Rob. I will look into correcting that.
ReplyDeleteI have noticed the same thing as Rob, my lifetime % correct dropped from about 90 to 82.9. But I also noticed that I dropped from a ranking 169 in this quarter to place 247. So this forgiveness policy doesn't work for me.
ReplyDeleteNo, blogje, is sure didn't work for you. I will take a look and see if I can figure out why that happened. I plan to tweak the algorithm for "lowest ranking" adjustments for end of quarter (the "real" and final adjustment).
ReplyDeleteOK, off to Oracle OpenWorld!
Regards, SF
Hi Steven,
ReplyDeleteI did welcome the initiative, however it's results are strange. I thought that you wanted to lesser the impact of a missed day (up to ten of them). It seems to me that you made it greater.
To be less general: I missed 8 days at the end of July/begin of August and I thought that I'll jump some of the people close to me who didn't miss a thing (but had lower correctness) after the policy is applied. And that the difference between me and 25th ranking player would be lower.
The opposite is true. I lost about 600-700 points relative to players I can remember so I can compare (Pavel Zeman 100%quizes, near 100% correctness, way above me before; Pavel Noga 99-100%quizes, lower correctnes than me, lower in rank than me before, higher in rank now; 2200 point -> 2800 points to 25th player). Note that I am no speedclicker, I may submit below 30s once a month, so I don't think that I could be penalized for speed.
Maybe you can make stats of the average bonus for player for missed day and for nonmissed day (maybe separated for ranks 1-100, 101-200, ...).
Seems to me, that the initiative backfired :-).
MP
I think I figured out why the current policy didn't worked for me. I started this quarter by scoring two days 100% and and then 2 weeks vacation. So my "2/3 of the average of all your previous answers" dropped very rapidly to a very low level.
ReplyDeleteI have no idea what happened to me either. Before the forgiveness policy was applied I had no or only a few people in front of me with more quizzes missed than I did due to my holiday, so I expected to be a rising "star" after the forgiveness. The rising star became however a falling star ;-). I only dropped in score considerably, which feels strange, but well, it may still be correct.
ReplyDeleteBut it is not easy to determine (I would love an Oracle Flashback query to show me the rankings just before the forgiveness policy was applied :-) and one from just before I went on holiday, when I had a nearby perfect ranking). I better not miss any day anymore...
Have fun at OpenWorld!
Toine
Hi Steven,
ReplyDeleteNice to see that you have started the forgiveness policy. But I need to point a thing or two out. before the weekend, I took out the rankings and my scores to see how this would come out for me. Bear in mind thought, that the data I took out is only to the 13th of September and apparently last week was bad on my behalf and I received 3 fixes for that week alone...
But looking at the other forgiveness fixes, I see something that is very strange. First it is the 14th of July quiz, that you deleted from the list of quizzes, as the site was down for more than 4 hours. I receive this as a forgotten adjustment (I actually managed to answer the quiz that day).
Then there are two days, the 10th of august and 7th of July. Actually I gain a lot from the forgiveness policy on those two days, but I don't think it is completely deserved. 7th of July was a beginners quiz, with top score of 250. I actually got everything right that day and was in 232 place with a reasonable score. Now I have received 605 points for that day (even though the top score should be 250), and moved to 98th place.
Also it is kind of funny, seeing that almost every day I drop down the list a bit, but on the quarter rank, I seem to gain (up 20 places). But I guess that is normal (the days one is not being "forgiven", a few people should under normal circumstances jump ahead).
Have fun at OOW.
Regards,
Ingimundur K. Gudmundsson
Hello Steven,
ReplyDeleteIs it correct that "average score" is calculated on total player answer statistics, i.e average score is 88 % and then 2/3 rule is applied to this 88%, say we get about 57% and last value is applied to the missed question score?
If it works in this way it gives some opportunities for cheating.
It would be better if average score were calculated depending on difficulty level.
So player may have 3 types of statistics
Beginner (for example in average is 95 %)
Intermediate (for example 90%)
Advanced (for example 75% )
If you calculate forgiveness score based on this detalisation it will be more accurate. Otherwise "beginners" will be treated as "advanced".
Actually I have no idea for the case when one of the statistics types is missing by coincidence. It should not happen if player has played enough number of games in a quarter.
Another wish is to get Email reports about own changes (which days and by what score).
Thanks a lot for all your efforts.
Kind regrads,
Sergey Porokh
Hi,
ReplyDeleteintention of the policy is to relieve part of the burden of a missed quiz. It seems to me that the opposite is true (my previous longer post has lost somethere, if it "only" has to be approved to be shown, then delete this one pls).
MP
Well....I think that I am going to have to reverse the forgiveness changes made this weekend. And quite honestly, I am not sure how I can apply this policy so that everyone is happy and feel that they are treated fairly.
ReplyDeleteHere's the problem as I see it: how do I "forgive" players who missed quizzes without penalizing those who have diligently played each day?
That was the idea behind the "lowest ranking" adjustment - but if I tweak the algorithm so that players who didn't miss a quiz are not penalized, then benefit is the forgiveness policy for those with missed quizzes?
It could also be simply that there is a bug or multiple bugs in my code (see report above of an adjusted score higher than max possible score). Yes, that does seem to be the case; you definitely should not see any scores above the maximum possible.
And then we run into another difficulty: constraints on my time. As you probably know, I do a few other things besides the PL/SQL Challenge.
Sigh....well, I suppose I get "points" for trying, right? Uh, no, no I don't.
So I will give it a bit more thought, but it looks like I will reverse the forgiveness adjustments made this weekend. I will see if I an identify and fix any bugs.
But in the meantime, if any of you have any thoughts on how I can "forgive" missed days without penalizing those who DID play, I would love to hear about it. I find myself coming to the conclusion, though, that this was a nice idea but does not have an "execution path" that will be fair to all or have any real impact.
Thanks for your patience,
Steven
And of course the more time I spend on these kinds of issues, the less time I have for writing quality quizzes that don't have mistakes. :-)
ReplyDeleteI noticed my % Correct dropped from (an embarrassing) 79% or so to 70.6%. I'm guessing this is because of my 2+ week vacation in August. I'll admit I haven't read in detail (nor do I want to) what happens to me when I forget to take the quiz (I missed 9/17 for some reason - I have no idea why right now!).
ReplyDeleteAt any rate, the only reason you end up with controversy is because people of your motivation to take the quiz. Frankly, I don't give a hoot. In the beginning, I started to strive really hard to do well (yes, I'll admit, I frequently researched documentation before I answered most of the time). But I found the it plain old took too much of my time.
I originally started taking the quiz - and still do - just as a gentle diversion, and perhaps (actually, most of the time) learn a little. I usually do my learning reviewing the prior day's quiz, NOT researching in order to take the quiz.
If the decision were mine, I'd strongly consider getting rid of the prizes for the rankings entirely. Instead, I'd just randomly select from those who took X% of the quizzes during the period, and scored better than, say, 50%, and hadn't won anything in the last six months. That way, you'd remove the motivation to cheat, along with the heavy pressure on the scoring algorithm and the related "discussion" (as well as the accuracy, etc., of the quizzes).
My only problem with the quiz is the same problem I have with any "test" is that I'm always over-thinking and wondering if every subtle quirk is really a trick question!
Thanks for putting your time and very thoughtful work into the PL/SQL Challenge.
-Dan Schoch-
Oh yeah, one more thing: Put the link to the blog on the top toolbar. I had a dickens of a time finding it!
ReplyDeleteAnd now the latest update:
ReplyDeleteDue to lots of negative player feedback, at least one bug in the application of the "lowest ranking" algorithm, and growing doubts about the feasibility of the whole policy, I have reversed all adjustments made this weekend for missed quizzes, zero scores and low rankings. My apologies for the confusion and changes in course. I do not at this point know if I will apply a forgiveness algorithm to Q3 quiz answers at all.
I have to admit: I am strongly tempted by Dan's suggestion (which others have made as well): reduce the incentive to cheat and the emphasis on rankings-based prizes. I am definitely going to give this some thought, and will open up a blog entry on the topic so you can all comment.
In the meantime, it is time to prepare for my first Oracle Develop presentation. Oh, and get some breakfast!
SF
Hi,
ReplyDeletemaybe all the negative feedback is due to "over the max" bug. Still think it was a great idea. Still trying my best although the chance to win anything in this quarter is tiny due very bad start.
Thanks for all that effort anyway.
Hi,
ReplyDeleteThe best forgiveness policy (as for me) is quite simple - consider only the top 80% results of the given player for the period scored (with non-participating considered as zero result).
Regards,
Oleksandr
The problem with doing something as simple as that is that it effectively punishes those players who have played every day. Or do i misunderstand how this would work?
ReplyDeleteSteven,
ReplyDeleteWhen applying these formulas, would it be possible to show the raw score/rank along with the adjusted score/rank for players, or perhaps just a (+/- ###) next to it to show the adjustments on the totals?
To be clear, I'm talking about the overall rankings page. This way it would be visible to all players.
This might help us visualize how the policies impact the rankings, and provide some transparency to the players. I think it would also help us identify issues, so it's kept fair and balanced. :)
Thanks,
Richard
I think everyone will have their idea of what is fair to them. People with a smartphone that can still play the game while on holiday will want their diligence rewarded. Some-one on shift work might typically do four week-days.
ReplyDeleteI wonder, would anyone be upset if the individual scores were published (eg Fred scored 450 on August 3) rather than the weekly summaries. It wouldn't worry me, and with the raw data people can come up with own rankings based on just the days they played
I actually like Oleksandrs suggestion of a very simple algorithm.
ReplyDeleteThe very complex forgiveness algorithm tried out so far makes it difficult to figure out why you go up or down in rank.
Consider this:
Say a player have a "TotalScore" and "RankingScore" for a given quarter. TotalScore is the sum of all your scores for the quarter. RankingScore is the sum of the scores of your top 55 days. (There'll be about 65 playing days in a quarter.)
TotalScore is mostly for fun, for watching your own progress and for the warm feeling of beating your comrades :-) RankingScore is the score used to calculate your rank in the competition.
The first 55 playingdays of the quarter TotalScore and RankingScore will be identical. After playing day 56, RankingScore will only use the 55 best days, so your worst day (possibly an absent day) will be "discounted." After playing day 57, your two worst days are discounted. An so on.
Those who miss days will have their missing days be the "worst" days (score zero) and so up to about 10 days missing will be discounted.
Those who play every day get the advantage of having more opportunities for getting a higher score. If I had a bad day on playing day 24, I can make up for it by getting a high score on playing day 56. The more days I play, the more of my "bad days" are discounted.
So as I see it - such a simple algorithm helps those who cannot play on vacation, but still rewards those playing every day as for each day more than 55 you play, you get a free chance of improving your score. You cannot loose by playing more than 55 days, but you can better your rank.
(Thanks to Oleksandr - I just grabbed the ball and ran :-)
Kim,
ReplyDeleteI still have the same concern: doesn't this still effectively punish those players who play regularly and score high on a regular basis? They lose a lot points relative to the other people.
SF
Step back for a moment and ask what you are trying to encourage / achieve.
ReplyDeleteIs it better to have two people playing three days a week or one person playing five days a week ?
Of course, this assumes rankings affect participation rates. You could look at other rewards. Stackoverflow awards contributors 'badges' for certain types of participation, with some badges easier to get than others (and some you get once, some you can get multiple times). You could have badges/medals for 'participated every day for a month / quarter', 'no incorrect answers for a week', 'first to report error'. Maybe a national flag for best score from a particular country.
I like Gary's badges idea.
ReplyDeleteAlthough, it would serve a different purpose than on SO - stackoverflow uses badges to encourage everyone to exercise all the various features of the site - you get a badge just for filling in your profile, for example.
For the PL/SQL Challenge, I think badges would be fun to be awarded for achievements, especialy for those not related to being highly ranked, such as Gary suggests, plus:
"participated at least twice a week for a year"
"reported an error"
"reported a bug"
"enhancement request" (when accepted)
"participated in quarterly championship"
etc.
Nice thing about badges is they cost nothing (except for the initial development effort, of course).
Steven,
ReplyDeleteI would not say that those playing every day really "lose" points relative to those missing days.
If I play day 56, I gain the difference between my lowest score so far and the score in day 56. The extra days played can only add score, never subtract score. And a person of lower rank with missed days cannot jump my rank. Missed days will never add to a score.
Say 2 players both on average scoring high 80% of the time. One plays 55 days in the quarter - one plays 65 days. The first one is stuck with a score reflecting that he is an "eighty percenter". The second one will be rewarded for playing more days - his RankingScore is computed almost entirely of his "good days."
As another example lets compare a player thats always correct with an "eighty percenter".
Whether the "hundred percenter" plays 55 or 65 days, he will get the same top possible score (except maybe speed differences.) The "eighty percenter" cannot get the same if he only plays 55 days, but he can get almost the same score by playing 65 days.
So the eighty percenter can by diligence get a score almost as a hundred percenter. That can be discussed whether this is fair or not. You could argue that it was fair if the hundred percenter had only played 55 days, but unfair if the hundred percenter had played diligently 65 days.
Steven, you stated the problem very neatly yourself some time ago: "How to forgive missing and bad days without penalizing those playing every day?" It's an impossible goal to achieve 100% - if some guy with missing or bad days jumps rank by forgiveness policy, some other guy must have been pushed down.
I think most of the problem in your algorithm is that "suddenly and magically" some people were jumped up the ladder and some people were pushed down.
Oleksandrs and my suggestion will not "suddenly" move people around in rank. Scores can never go down only up. Rank is still calculated every day.
A person missing days can only move ahead of me in rank if he actually play and scores. And if I also play that day, I have a chance of keeping him behind me in rank if I score better than my lowest score so far and therefore gain a net improvement in my RankingScore.
I see two potential problems though with the simple 55-day rule.
1. In the top of the ranks there will be a tendency to "even out" the scores. A player with a general average of 99% correct and a player about 97% correct will be competing very "evenly" for rank. The 99 percenter will not have much of an advantage (except in that by the discounting of lowest scores he can still by dilligence gain a fraktion of an advantage.)
2. The feel of how your score and rank changes will be different in the first 55 days compared to the last days of the quarter. In the first 55 days it will be "straight-forward competition" - in the last days it will for diligent players be "can I improve my score" and for those having missed days it will be "can I make up for what I missed."
You could go with a "no forgiveness" policy - simply state that it will be to complex to make it fair. I would accept that.
But the simple 55-day algorithm is a compromise that in my humble opinion is acceptable. I believe that the penalizing effect in such a scheme is minor compared to the benefits. But it is just an opinion - it will always be a compromise that some will feel is fair and some will feel is unfair :-)
PS.
ReplyDeleteSteven, you might try it out on your data for Q2 2010. Select "sum of top 55 days" score for players for that quarter and see how much rankings would have changed?
Just a suggestion if you have the time (which you don't really I know - you must have access to some more hours in the day than the rest of us mere mortals :-)