08 August 2010

Proposal for Forgiveness Policy

Several players have requested a way to avoid being penalized heavily for missing quizzes or for making a silly mistake or for Internet failures that result in long delays is submitting responses. This issue has come up quite a bit lately, given the summer months and the tendency of normal human beings in the northern hemisphere to take holidays during this time.

In addition, I now manually make adjustments to scores when players report a problem. This is very time consuming and simply not sustainable or scalable (as we add more players).

I therefore plan to implement a forgiveness policy starting with Q3 2010. We will "forgive" a certain number of missed days or low scores per month/quarter. Once this is in place, we will also no longer make adjustments to scores, such as when your quiz submission is interrupted and you have to go back and "take the quiz" a second time.

I share with you below my ideas on how to implement/apply this policy. I would love to hear what you think. It may take a little while to sort it all out, but of this you can be certain:

The forgiveness policy will be applied at the end of Q3 (and every quarter thereafter). The determination of participation in the championship playoff will be made after I apply the forgiveness algorithm. So if you miss a few days of quizzes, do not panic!

Forgiveness Algorithm
  • Must be applied equally to everyone. That is, if I do not miss any quizzes, I should not be penalized, relatively speaking. Even I should see a benefit from the forgiveness policy.
  • Apply forgiveness to 10 days of quizzes in a quarter(out of about 60 days).
  • For each missed quiz (up to 10), give that person a score equal to average of all scores for that day. This way, you do not take the "full hit" of a 0 score, but you also will not "jump ahead" of those who rank lots higher than you (and remember: this only happens for a small number of days out of the total).
  • If you missed less than 10 quizzes in a quarter (let's call the balance N), then I will apply a different algorithm to the N lowest ranked days in that quarter. Suppose, for example, you usually rank in the top 100, but on 15 days during the quarter, your ranking was around 200. You also missed 3 quizzes. Then I would "forgive" 7 of the lowest ~200 ranked days.
  • Forgiveness for low ranking: change your score for those days to match the score of the ranking on that day that equals your average ranking throughout the quarter. In other words, if you average ranking is 125 then on a day in which you were ranked 250, I find the score of the person ranked 125, give you that score, and then re-rank.
What do you think? I need to find a way to give players some leeway/flexibility, but also not "punish," in effect, those who are diligent and most devoted to the quiz.

Thanks in advance for your thoughts.

SF

6 comments:

  1. Hello Steven,

    I think the algorithm is fine. The only thing I worry about is when the changes in score will be applied. Will this be applied every day or only at the end of the quarter? I prefer every day because else the ranking could really change at the end of the quarter when it comes close to qualifying for the playoff. What I mean is that if you play every day and score your normal (average) score every day you could really drop in ranking if people with greater fluctuations in scoring get rescored at the end.

    ReplyDelete
  2. Very legitimate concern. But I don't see how I can do this on a daily basis. I need to be able to compare "apples to apples". I could easily apply the rule for missed days immediately, but different people miss different numbers of days, so the % of quizzes taken could get funky.

    I certainly don't see how I could apply the rule for "worst rankings" because that could change the next day or week.

    I am hoping, though, the algorithm would have the effect of "compressing" the range from lowest to highest without necessarily changing the rankings very much.

    Of course, that's easy to say, but tough if you are the one who shifts down just enough to miss automatic qualification in the playoff.

    So - how do you think I could do this on a daily basis?

    SF

    ReplyDelete
  3. Actually, I have been thinking more about this. It seems to me that rather than use only "lowest ranking" (which means wait till end of quarter), I can do the following:

    After a day ends, if the player either missed the quiz OR if their score is 0, immediately apply the forgiveness for that day. You can't do worse than 0, so immediately forgive that.

    This way, the only corrections happening at the end of the quarter would be for those players who did not miss 10 quizzes and/or score 0 on up to 10 days.

    SF

    ReplyDelete
  4. Hello Steven,

    very thoughtful algorithm I think. But Gerben's right though. I've got a suggestion which may respond to his concern and yet be "easily" implementable.
    1. apply "lost day forgiveness" daily
    2. at the end of a quarter rank players and qualify top N to the playoff.
    3. apply "worst days forgiveness"
    4. re-rank and qualify those in top N who haven't qualify yet (you may flag the former group qualify reason as "rank" and the latter as "'forgiven' rank" or whatever suits you).

    MP

    ps. if you want to be really playful you can even compute and display 'forgiven' score and ranking daily. The only thing you need is not to update original scores. Then you'll be able to compute fictional scores everyday. But this seems to me as much more complicated both in terms of development time and runtime.

    ReplyDelete
  5. One more thought - instead of averages you may use medians. It usually better reflects the reality of ranked lists.
    MP

    ReplyDelete
  6. Be careful to not overdo this. In the end everyone gets the same score....... ;-)

    ReplyDelete