Do MLB umpires want to be fair, or just perceived as fair?

TLDR: "A strike is 5.9 percentage points more likely to be incorrectly called a ball if the previous two pitches were also strikes."

"When faced with the prospect of making the same call three times in a row - thus creating a 'pattern' of calls - umpires make erroneous calls nearly six percentage points more often than in situations without obvious patterns."

In the average baseball game where the umpire is forced to make about 150 subjective calls, there are bound to be a few mistakes. But what if the umpire is systematically making erroneous calls?

As a Major League Baseball (MLB) umpire, one must be perceived as neutral. Since each pitch is a game between the pitcher and batter, making too many calls advantaging one or the other may be construed as favouritism. While umpires want to appear neutral, natural human biases may lead to a fear that patterns in their subjective calls are seen as anything but.

I want to uncover if there is a systematic bias to avoid these so-called 'patterns' in the called balls and strikes by MLB umpires. In order to tackle this question, I first collected all the calls by a home-plate umpire in the MLB from 2016 by Baseball Savant's Statcast Search. I am only concerned with the pitches that required the umpire to decide whether the pitch was either a strike or a ball. This excludes any time that the batter swung the bat or any time the pitcher threw a ball intentionally or threw a pitch into the dirt. What I am left with is 360,278 pitches for which an umpire had to make a subjective call, answering the question 'was the pitch a strike or a ball?'

The data also tracks the location of where the ball passed through the strike zone: this identifies whether the pitch was a 'true' strike, as per the MLB definition. Comparing the umpire's call to the true call, I find that the correct call was made 88.8% of the time. In other words, the error rate of umpires is 11.2%. The false-positive rate for strikes (i.e. called strikes that were actually balls) was 17.2%, which, although high, is similar to other's findings.

I then identify a few situations in the ball and strike calls to test my theory that umpires avoid patterns. The first pattern I identify I term X-X-Y, where X is either a ball or a called strike, and Y is the opposite (e.g. three consecutive pitches that were called strike, then another strike, then a ball). There are 17,481 times this pattern occurs in the data. Shockingly, the unconditional error rate of the Y in the sequence of X-X-Y is 14.4% - a 3.2 percentage-point increase from the overall error rate. This is the first piece of evidence that umpires avoid patterns by erring more often when the correct call would be the third consecutive strike or third consecutive ball than they err in another situation.

Next, I consider a different counterfactual to the X-X-Y pattern, which I call X-Y-Z. Again, the X is either a ball or a called strike and Y is the reciprocal. I let Z be either a ball or a strike (i.e. I allow for either a X-Y-X or a X-Y-Y pattern). If the umpire avoids a pattern of three of the same call, an X-Y pattern should have no impact on the call of the third pitch. This X-Y-Z pattern occurs 28,271 times in the 2016 data and the unconditional error rate of the Z pitch is 9.4%. This means that umpires make an erroneous call on the third pitch of a X-X-Y pattern 5 percentage points more often than that of a X-Y-Z pattern!

Up until this point, I have only considered unconditional probabilities (i.e. not considering other factors that may affect the error rate of the umpire). This includes the speed and type of pitch, the location of where the pitch crosses the plate, or even the inning of the game. In order to control for all these necessary characteristics, I run a logit model to predict an erroneous call. After playing with the specification, I settled on the following model:

prob(wrong call) = f( X-X-Y, pitch type, release speed, spin rate, location, inning, top/bottom of inning)

Pitch type is a set of dummies indicating the type of pitch thrown (four-seam fastball, curveball, etc.). Release speed is the velocity in which the ball is thrown measured in miles per hour, spin rate is the number of times the ball rotates while traveling from pitcher to catcher in revolutions per minute. Location is a set of dummies corresponding to the location of where the pitch crossed the plate (see this diagram for more information), and top/bottom of inning is a dummy indicating whether the game is in the top or the bottom of an inning. Lastly, X-X-Y is a dummy that indicates the pitch is the Y in a X-X-Y sequence.

After running the model, I calculate the marginal effect of the X-X-Y dummy. After controlling for pitch characteristics, being in a X-X-Y situation increases the probability of a wrong call on the third pitch by 5.9 percentage points over a X-Y-Z situation. This can be stated alternatively as a strike is 5.9 percentage points more likely to be incorrectly called a ball if the previous two pitches were also strikes.

Below is a graphical depiction of all the aforementioned probabilities.

If umpires did not care about the perceived fairness, the error rate would not be different for a given situation over another. This effect may be a conscious or subconscious decision to keep the balance of calls which favour the pitcher and batter. Another explanation is that umpires suffer from the Gambler's Fallacy and falsely assign a low probability to the event of three consecutive strikes or three consecutive balls. In the Gambler's Fallacy scenario, the umpire would think that the likelihood of three strikes in a row is so small, the third pitch 'must' be a ball.

In summary, when faced with the prospect of making the same call three times in a row - thus creating a 'pattern' of calls - umpires make erroneous calls nearly six percentage points more often than in situations without obvious patterns.

Please feel free to offer comments, questions, or even suggestions for future work!