The Effect of Bad Referees

2 November, 1996

For as long as I can remember, I have always simultaneously admired and despised the neutral referee that keeps control of a game. For them not to get caught up in the action, cheering for the underdog, correcting players' errors as a coach would, I admired them. For their inconsistency in calling or not making calls when I thought they should have, I despised them -- though I never actually earned a technical foul, either as a player or as a coach. (It was probably because I never mentioned their mothers.)

Since I started analyzing basketball in some detail, I have been frustrated in my attempts to understand the ref's role in the game. How do you analyze a neutral contributor to the game? I could look at whether they truly were neutral, whether they called an equal number of fouls on different classes of players or teams; but that was flawed because different teams and different players clearly did commit different numbers of fouls. I couldn't come up with a way to identify classes of teams or individuals that "should" receive the same number of fouls.

Finally I've figured out how to do it. Now I know how to strike back at the refs, to let them know how they can change the odds of winning or losing if they are doing a bad job. I finally figured out what it means for a ref to do a bad job.

A bad ref is an inconsistent ref. A terrible ref is a biased ref. I'm interested in just the bad refs and really hope that the terrible ones are kept from being refs by the professionals concerned with their standards.

An inconsistent ref can change the odds of a game by introducing extra noise into the game. For those familiar with some of my other work (Basketball's Bell Curve, What Strategies Are Risky?, etc.), this one insight should tell you exactly how an inconsistent ref changes the odds. You can check your answer by pressing the button below. This way, if you don't want to know yet, all you have to do is control your mouseclicking finger, not your eyes.



Rule Effects Ref Effects

One of the things that always kept me in good favor with refs was that I told them that they didn't make as many mistakes as the players did. Players miss shots, commit turnovers, pass to the wrong player, hold on to the ball too long, and foul other players. All a ref does is occasionally miss a call or make a call that changes what should have been the outcome of the play, sometimes permitting Mark Price to get away with a takedown and ever so rarely whistling Charles Barkley for a foul that "everybody knows" he didn't commit. Everyone in the game is inconsistent, but the players are more inconsistent than the refs.

We know how players' inconsistencies and errors affect the game, but how specifically does a referee error affect the score? A ref's error alters whether a score actually goes up on the scoreboard. This is an assumption for simplicity, but not a bad one. For instance, a ref can miss a travelling call and a team gets a score they shouldn't get or a ref can miss a defensive foul and a team doesn't get a score they should get. With just this simple rule and without introducing any bias in those errors, I can show that, although "it all balances out in the long run", inconsistency changes the odds of one team winning over another.

My method is simply to simulate two teams that score with a frequency typical of basketball teams, then add in the effect of the officials. For example, I simulated 5000 games between Team A, which scored on 54% of their possessions (its floor percentage was 54%), and Team B, which scored 50% of the time. In each game, I ran through 95 possessions for each team. On each possession for Team A, they had a 54% chance of scoring and on each possession for Team B, they had a 50% chance of scoring. Over the course of a 95 possession game, it should be obvious that Team A would be the likely victor. In reality, given no mistakes by referees, I was simulating that Team A had a 70% chance of winning. In order to simulate imperfect referees, I said that reasonable refs made a mistake that affected a score on 5% of possessions, a fairly low rate and probably one that Bobby Knight would disagree with -- I just hope he doesn't throw a chair at me. So, for the same sequence of possessions and games as before, I input this error rate in, and found that Team A's winning percentage dropped a measly 1% to 69% -- now I know Bobby Knight is going to throw a chair at me. (I ran larger simulations that took much longer, but they showed consistently a 1-2% difference.)

So before Coach Knight actually hits me with one of those chairs, I will raise the referee error rate to about 15%. He may still think that is too low, but hopefully he'll only slam a phone. For this case, the winning percentage of Team A should still be 70%, but their actual winning percentage drops to 65% due to referee errors. Over the course of an 82 game NBA season, this means that a team like the 1996 Los Angeles Lakers (which had a winning percentage of about 65%) could blame three or four of their losses on bad reffing -- if NBA reffing is really that bad, which I doubt. On the other hand, if the refs really are this bad, the 1996 Los Angeles Clippers (which had a winning percentage of about 35%) could thank them for three or four of their wins. And they probably thought that they never got a break. (If you know how to read Visual Basic code, the Visual Basic 4.0 code for this program is here, the SimGm and RunSims routines being the heart of the calculations. If you have a VB4.0 compiler, the project file is here. I am planning a simplified Java form of the program for inclusion here, but the program will run slowly.)

This is the general trend I found. Favorites were the ones affected adversely by inconsistent officiating. Underdogs were aided. I knew this before I ran the simulations once I realized that bad officiating was inconsistent officiating. In previous works (Basketball's Bell Curve, What Strategies Are Risky?), I have shown that inconsistency translates mathematically into a higher standard deviation of points scored and allowed (or of offensive and defensive ratings). According to the Correlated Gaussian Method Formula:

            __                                 __
            |           (Rtg-Opp.Rtg)           |
Win% = NORM |-----------------------------------|
            |       SD(Rating Difference)       |
            --                                 --
SD(Rating Difference) = SD(Rtg - Opp.Rtg)
		      = SQRT[Var(Rtg)+Var(Opp.Rtg)-2*Cov(Rtg,Opp.Rtg)]

NORM means to take the percentile of a mean-zero variance-one
normal distribution corresponding to a value given by
that in the brackets I faked.

Rtg:		Points scored per 100 possessions (offensive rating)
Opp.Rtg:	Points allowed per 100 possessions (defensive rating)
SD():		Statistical standard deviation of quantity in parentheses ()
Var():		Statistical variance of quantity in parentheses ()
Cov():		Statistical covariance of quantities in parentheses ()
It's time to learn a little stats. A team's win percentage goes up with an increase in the difference between their average offensive and defensive efficiencies divided by the standard deviation of this difference. So, if a team has a positive difference between their offense and defense (for example, they score 100 ppg and allow 95 ppg), how successful they are then depends on how consistently they maintain this difference (in other words, whether they score 100 and allow 95 in every game, meaning they'd win every game, or whether that fluctuates from game to game a lot). A team's ability to consistently maintain this difference depends on their own talent, on their opponents, and on neutral things like the weather, the shoes they are wearing, and the refs. It's an inherent problem of human nature to be inconsistent -- in this case, it not only affects the person being inconsistent, but the entire game. There's nothing we can do about it, but train the refs to be as consistent as they can be, except...

If refs show a little favoritism for home teams and for great players, they are essentially aiding favorites, which they hurt by being inconsistent. That was the most interesting thing (to me) that came out of this study. Let me repeat that: a ref that shows some bias for home teams and for better players counters the inherent favoritism to weaker teams caused simply by being human.

Is this a small part of the game? Yes it is and I believe that introducing more reality into the simulations would dilute the ref's influence a little further. For the most part, it is the timing of inconsistency that gets a ref noticed, not the frequency. Unfortunately, this can afflict even good refs and there isn't much anyone can do about that.

(This overall effect is roughly as important in college and high school, where the games are shorter, allowing less time for ref errors to balance out. If refs at these lower levels are more inconsistent than their NBA counterparts, which is disputable, then this is more important.)

Now, for those of you interested in the point spread, this issue can be important. A ref that makes a mistake only once in twenty possessions introduces enough noise to change the average spread in the above games from about 8 to about 7 or 7.5. For you gambling readers, keep in mind that the NBA would rather not hear a word about point spreads (especially if you're Michael Jordan), so don't even consider calling the league to tell them I said this. I like staying on the NBA's good side...