07 January 2012

Fact Mining and the Weekly Logic Quiz (10983)

In addition to the PL/SQL, SQL and APEX quizzes, the PL/SQL Challenge offers a weekly logic puzzle, modeled on the Mastermind game.

In the puzzle for the first week of 2012, the second choice (9084) stated:

"If 1 is in the solution, then 7 cannot be in solution."

This was scored as correct and offered a step-by-step logical "proof" of why this was so.

Unfortunately, as several players pointed out, my logic was flawed from the very start - because 1 could not be in the solution at all. As Jennifer explained so clearly:

"We know from the clues that 1 cannot be in the solution. Based on the 3rd clue that only one of the set '2143' is in the solution, we know that 5,6, and 7 must be in the solution. Therefore 1 cannot be in the solution because the 2nd clue states that only 2 of '1456' is in the solution. We know that 5 and 6 must be so 1 and 4 cannot be."

I will change that choice to incorrect; everyone's scores will be updated within the next 24 hours.

While I am unhappy with my error and the need to issue a correction, I am delighted that several players analyzed the puzzle closely enough to uncover the problem - and also because this process reinforces what is to me one of the most important lessons for programmers when playing Mastermind (and similar games):

Make sure that you fully "mine" all "clues" (results from tests of your code) for all possible information.

All too often we (I!) barely look at the results of a test, or the report of a bug, before we rush to the source code and scramble (flail around?)  to apply a fix. In doing so, we (I) often overlook critical information, and this oversight can lead to lots of time lost and even the introduction of new bugs.

In this puzzle, I did not perform enough analysis to conclude that 1 could not be in the solution. By missing this fact, I then introduced a mistake into the quiz.

Lesson learned: before you start messing around with your code, make sure you have extracted all possible conclusions from tests and specifications. Then apply that knowledge in a systematic fashion.

Thanks to Jennifer, DKennedy, Bobby and Pavel for identifying this problem!

Steven
Posted by at

7 comments:

  1. iudith07 January, 2012 17:51

    Hello Steven,

    I have observed the same problem while playing the quiz, and, if you remember,
    I already signalled a similar case in one of the first quizzes played in this competition
    and we seemed to agree that the best would be to avoid including such choices in the quizzes.

    The fact that "1 cannot be in the solution" is indeed TRUE,
    BUT, very important, this still DOES NOT render false the IMPLICATION that says:
    "If 1 is in the solution, then 7 cannot be in the solution".

    As per the boolean logic's table of truth for the "implication" operation,
    a FALSE (predicate) can imply anything, including a TRUE (predicate),
    as it happens in our case.

    When we say "a implies b" we DO NOT state anything about "a" being absolutely TRUE or absolutely
    FALSE.

    Here the rationale that proved the statement "if 1 is in the solution then 7 is not in the solution"
    was completely correct, and the choice asked about the TRUE or FALSE value of the IMPLICATION
    predicate, and NOT about the truth value of "1 is in the solution" OR "7 is not in the solution"
    taken as separate statements, which would have been a completely different question.

    So, I think that it will be wrong to change the scoring of this choice.

    If any objection can be raised, then it pertains to the "CORRECTNESS of the choice as a choice",
    which in fact DOES NOT render the choice as FALSE, but merely questions its appearance in the quiz.

    So, if you want to make everybody satisfied, the most you can do is to remove this choice
    from the quiz and rescore based on the remaining 3 choices only.

    Thanks a lot & Best Regards,
    Iudith

    ReplyDelete
  2. Pavel07 January, 2012 18:52

    I'm one of those, who identified the problem. However, my reasoning and conclusion were different than presented in this post.

    I agree, that 1 could not be in the solution at all, but that does not automatically imply that the choice is incorrect. In fact, the choice is still correct. The reason is quite simple. We have an implication "If 1 is in the solution, then 7 cannot be in the solution." with false assumption ("1 is in the solution"). Such an implication is always true regardless of the second part (using a false assumption, you can prove anything - the following web has a good example: http://news.ycombinator.com/item?id=1187530).

    As a result, the choice itself is scored correctly. It is the explanation about its correctness, that should be changed.

    Pavel

    ReplyDelete
  3. Wim de Lange07 January, 2012 19:11

    Funny. If you handle this quiz as an exercise in logic, then you should not change the answer. Why?
    "1 in the solution" implies "7 is not in the solution". Or A implies B. But if A is not true, the statement A implies B is always true. Look up the truth tables on "implies". And the reader is asked to say which statements in the quiz are true. So the second choice (9084) is a true statement.
    So, who is wrong here? (me? not really).

    ReplyDelete
  4. Steven Feuerstein07 January, 2012 21:20

    Thanks for the reminder, Iudith. Yes, I did it again. I need to stay away from these "IF x THEN y" in the logic quizzes.

    I can change it back to correct, but really the bigger issue for me is that I want these quizzes to be helpful and interesting not only on the day taken but on into the future.

    So I would much rather change this choice to something like:

    "1 is in the solution."

    and then give everyone credit for marking it as correct. That messes things up a bit for our "first takers" but makes the quiz much better for future takers.

    Will you be mad at me if I do this? :-)

    Pavel, I can't get to that URL you provided. Could you confirm it?

    ReplyDelete
  5. iudith08 January, 2012 10:30

    Hello Steven,

    No, I will not be mad on you if you do this and give everyone credit :)

    I would only be mad if you would leave it "as is" and consider it incorrect, which is WRONG !

    You do not need to GENERALLY stay away from "IF x THEN y" choices, subject to the only condition
    that "x" can be both TRUE and FALSE for the different possible solutions of a specific puzzle.
    ( that is, "x" is neither always TRUE nor always FALSE ).
    In such a case, the IF choice could be a "living" one that cannot be objected.

    I am "mad enough on myself" for how I started this quarter ...
    no more "resources" to be mad on you as well :) :) :)

    At least in this Logic competition, I am still alive ...

    Thanks a lot & Best Regards,
    Iudith

    ReplyDelete
  6. Sean09 January, 2012 14:06

    I like the example in Pavel's link, but I don't think that proves the point.

    If a false assumption means you can prove anything. Then you can both prove it is correct and prove it is not correct.

    I chose incorrect based on the judgement of the previous fallacy where it was marked incorrect. I didn't post a comment because there was already precedent that fallacies are scored incorrect.

    If this position is changing then I suggest everyone be given credit for that answer regardless of choice.

    ReplyDelete
  7. Steven Feuerstein09 January, 2012 17:51

    Thanks, Sean, and all others for your comments. I am going to change that choice, get rid of the "IF...THEN..." entirely, and give all players credit for that choice.

    ReplyDelete

Subscribe to: Post Comments (Atom)