The 2024 season has been phenomenal for the Kansas City Royals and Matt Quatraro. Less than a full season removed from a 106-loss season in 2023, the team is now deep in the hunt for a playoff spot with just a month left in the regular season. For Quatraro in particular, success hasn’t come without its share of criticism. Whether playing time for bench players such as Adam Frazier or Garrett Hampson, or bullpen usage in losses such as Wednesday, the gripes seem to continue for Quatraro. Perhaps the favorite for American League Manager of the Year, it seems like a true disconnect between results on the field and fan perception of the job Quatraro has done.

With all that in mind, it seemed more than worth it to analyze Quatraro’s decision-making with some data behind it. Unfortunately, there’s no manager fWAR stat. So much of what a manager does is intangible. It’s difficult to put a number on how lineup construction, bullpen moves, or playing time decisions impact a team’s win-loss record on any given day. Today, looking at just bullpen usage, I’ll attempt to do just that.

Using Leverage Index and WPA data to better judge bullpen usage over a long season

To better understand the process here, it’s important first to understand what each statistic means. The first data point for each reliever is Win Probability Added or WPA. This one is rather simple, and FanGraphs provides a solid explanation for how WPA is calculated:

“For example, say the Rays have a 45% chance of winning before Ben Zobrist comes to the plate. During his at-bat, Zobrist hits a home run, pushing the Rays’ win expectancy jumps to 75%. That difference in win expectancy (in decimal form, +.30) from the beginning of the play to the end is Ben Zobrist’s WPA for that play. The pitcher receivers a -0.30. If Zobrist strikes out during his next at bat and lowers his team’s win expectancy by 5%, his overall WPA for the game so far would be +.30 – .05 = +.25, as WPA is a cumulative statistic and is additive.”

It’s rather fundamental for relievers. If a relief pitcher gives your team a better chance to win the game, they’re benefitting your roster. For managers, it becomes more complicated than simply using your best relievers. The art of bullpen usage lies in managing workloads and using the best pitchers at the most opportune time. In doing so, a great manager keeps their best relievers available when they’re needed most. For example, using your best reliever in back-to-back days with a six-run lead could make him unavailable for a save situation in game three. That would clearly be poor management. To gauge when a pitcher is used, FanGraphs’ Leverage Index statistic can be very useful. The Leverage Index — or LI for short — is a method for quantifying what situations a player is used for. More specifically, gmLI measures the leverage index with a pitcher enters the game.

WPALI Decision Value

By comparing these two statistics, we can come up with an output number that gauges how good (or bad) a manager’s usage is for each pitcher. Put the entire bullpen’s data together against an average of the leverage index gives a number for the manager himself. Let’s start with each individual pitcher. By taking each pitcher’s WPA this season, and normalizing it per nine innings, you can then combine that with gmLI to compare how those two numbers match up. In this example, you’re left with more or less this result:

• POSITIVE WPA, HIGH gmLI = > 0.00
• NEGATIVE WPA, LOW gmLI = > 0.00
• NEGATIVE WPA, HIGH gmLI = < 0.00
• POSITIVE WPA, LOW gmLI = < 0.00

Essentially, where zero is considered normal expected usage, you’re given a value for each pitcher. Anything under zero means poor usage from the manager compared to expected. Anything more than zero would be better usage than expected from the team’s manager.

Using this data, you’re left with the results above. There are outliers on either end of the spectrum. Carlos Hernández has a negative WPA in 2024, paired with extremely low-leverage usage. That’s better than expected usage. It’s a similar case with Nick Anderson and Matt Sauer. Lucas Erceg has posted a positive WPA this season and has been used in high-leverage innings. On the flip side, James McArthur has been worth a negative WPA in 2024 but has been used in higher-leverage innings on average this season. The result is a number below zero, showing some opportunity in the decision-making for McArthur’s usage.

The data certainly isn’t precise (I’m no statistician) and I’m confident there are flaws in my process here. However, more or less the data matches what you’d expect. Kris Bubic stands out, as he’s been excellent this season but ranks here in the negative. The result lies in FanGraphs’ WPA data. Despite his success this season, FanGraphs gives him a -0.06 WPA in 2024. Others make a ton of sense. Chris Stratton and Will Smith have drawn the ire of fans all season long. Both have been worth heavy negative WPA and have been used more or less in medium-leverage situations. Based on their WPA value, they should have been used solely in low-leverage (if at all).

Judging Matt Quatraro based on the data and formulations

There are limitations to this exercise, such as roster control and small sample sizes. A reliever could be so ineffective that no matter what leverage they’re used in, it’s a negative value. In that scenario, the issue lies not in the manager’s usage, but in the team’s decision to roster the pitcher at all. Beyond the limitations, however, we can take away a total number to help judge Matt Quatraro rather than each individual reliever. In 2024, Quatraro has been given a bullpen that’s collectively been worth -0.84 WPA. He’s needed to navigate a relatively bad bullpen for 375.5 innings, with an average leverage situation of 1.03. Despite having a severely negative value from his bullpen as a whole, Quatraro has posted an average mark of 0.09 mark using my formula above.

There are many more aspects to consider when judging a manager. Bullpen usage is just one aspect, but Quatraro has more or less made decisions exactly as you’d expect, given the arms and situations he’s faced. Sure, there’s been a handful here and there that haven’t worked out. Those will happen no matter who the manager is. However, Matt Quatraro hasn’t managed his bullpen anywhere near as poorly as some might have you believe. Perhaps I’ll continue this exercise and attempt to place a number on lineup construction and start/sit decisions. In doing so, it would be possible to combine them all into one cumulative gauge of how effective a manager is.