For some time, I’ve been a little unhappy with my fairness measures. It’s time, I think, to try something new.
Something along these lines:
Fairness (D): Fairness is the correlation between the skill level of the player, and the player’s mean reward.
The reward could be expressed in dollars and cents if it is evaluated in terms of prize money (hence fairness ($) in the title to this post). But I’d like to to be a little more flexible than that – to cover situations where the reward is couched in terms of ranking points, the opportunity to play additional matches, or prize money, or whatever else, including various combinations.
The immediate impetus for this was my work drawing brackets for the Viking Classic. The main events there are in a single-elimination-with-progressive-consolation format, for which fairness (C) is not defined. Yet I needed a bracket I’ve never drawn before – essentially a 64 with a DE shift. And while I’m pretty sure I did the drops right, I couldn’t test them with my simulator.
But this is not the main reason I’m ready to try something else. Fairness (C), my first effort at a metric for fairness, simply takes the inverse of the sum of the differences between the skill of the best player and the skill of the eventual winner (+0.01). It’s at a maximum when the tournament is always won by be best player. The only value it recognizes is the value of winning the tournament as a whole, and so it doesn’t capture, very well, anything that affects players unlikely to win the whole thing. And, when you think of it, unless you set the chance factor to zero it would be highly suspicious if the best player did always win – that would take fairness (C) to 100, but it would also have me searching through my simulation to see what went wrong.
Fairness (B) was an effort to measure inequities that affected players from the beginning of play. It also depends on overall winning of the tournament, though it cares about small percentages at the outset, and thus is good for assessing the effect of byes. But unfortunately, it’s not good for much else – it measures nothing except random errors in a balanced bracket.
Fairness (D) could, at least potentially, address the shortcomings of both other measures, and unify them in a single one. It would recognize that the fairest result is neither for the best player always to win, nor for every player to have an equal shot.
There are undoubtedly some problems to be solved along the way, and I’m not ready to retire either fairness (C) or fairness (B). But I’m ready to start, and will report on my progress in future posts.