A Fly in the Ointment

I started to apply the new fairness (C) measure to a number of different designs to see if it was working properly. I had a nagging suspicion that I must have had a good reason to reject the more obvious formulation of the statistic when I settled on the present measure. It didn’t take long for a problem to appear.

After a number of successes on larger brackets, I thought I’d check the measure out on some very small ones, like the 8SE I’ve been using as an example in my TGT series. It seemed to be working, until I looked more closely.

The new statistic for the 8SE at luck 1 was a plausible number in the high seventies. I notice, though, that the calculated confidence interval was very large – even with a million trials, the range for 95% confidence was about 0.1, or a full 10%, wide. And, sure enough, repeating the yielded numbers that differed by about three percent. Obviously, a measure that couldn’t pin down a reliable number in a million trials wasn’t going to be of much use.

So what’s wrong? The problem was that for a small proportion of those million trials, the best of the eight players was going to have a Z score very close to zero. That meant that the fairness (C) statistic was going to be obtained by dividing the actual aggregate payout by a number very close to zero, which would yield wild swings, thousands of percent (plus or minus, depending on the signs of the actual and ideal distributions). This didn’t happen very often, but it happened often enough to make the average value highly unstable.

I haven’t given up on the new measure. There is probably a way of tweaking it to avoid the problem. As an initial expedient, I’m constraining the value of fairness (C) for an individual trial to the range -1.0 to +1.0, and this is yielding reasonably stable averages again. But I need to understand more about what’s happening with the small number of runs that are causing the problem, and how I’m distorting the statistic by constraining the value of fairness (C).

Suggestions from readers will be even more than usually welcome!