A Fly in the Ointment, part II

In recent posts, I’ve been mentioning my desire to redefine the fairness (C) measure so that it’s right-side-up again, and can be interpreted as a percentage.

Another silly error. I ought to know better. I’m trying to use ratios on data that supports an interval scale, but not a ratio scale.

That often led to very anomalous results when either the expected aggregate payout or the ideal aggregate payout was negative. I suggested a rather clunky way of dealing with this problem in A Fly in the Ointment. But that’s not the only anomalous result. By assuming a ratio scale, I’ve made it so that a tourney with better players looks more fair than one with less good players. Thus, for example, in my standard 8SE, if the skill levels of the players are [2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9], I get a calculated f(C) of 92%, but when I run the trials with less skill [0.2 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], the calculated f(C) is only 76%. And with skills [-0.2, -0.3, -0.4, -0.5. -0.6, -0.7, -0.8, -0.9] the darned thing goes berserk. But with any of these skill distributions, the old-style fairness (C) gives about 21.7. That may be a difficult number to interpret, but at least it doesn’t get distorted by the absolute value of the skill levels of the players.

I’m tempted to keep the new fairness (C) measure to trot out on special occasions (and when I know it will be well behaved). But it’s a mistake, and rather a fundamental one. So, the last couple of posts that used it, Magic Mirror on the Wall … and The Dolly Bracket will remain curiosities until (if it every happens) I get around to redoing those experiments with the good old form of fairness (C).

This episode reinforces my earlier comments about publication bias in Revenge of the Pies. I vaguely remember, now, realizing that a ratio-scaled fairness (C) would be a mistake, but I didn’t document that at the time, and so the this same mistake was waiting for me to make it again when I went back to remind myself how fairness (C) was calculated.

It’s rather embarrassing to keep admitting this sort of error. But at least now I’m unlikely to make this particular mistake yet again.