Measuring ABS Decision Quality

Challenge Decision Quality Metrics

Every batter and catcher in baseball (and very occasionally pitchers, but we're going to ignore them here) are now making dozens of decisions per game that didn't exist before. The lion's share of the decision making process is based on where the ball crossed the plate. If a catcher closes his fist on a pitch middle-middle and doesn't hear the ump call a strike, he's more likely to tap. But some of it is based on the state of the game - what inning is it, is this a good opportunity to score, and am I possibly going to overturn a strikeout or walk with this challenge? We can look back on this Ronald Acuña Jr. challenge and see that none of these apply the way that they do on this much scrutinized challenge by Matt Wallner.

A week into the season we can see that some players are good at this and some are terrible. Until now the measurements used on this website haven't had a way to measure aggregate challenge value that go beyond win rate, run creation, and WPA. Win rate tells you the outcome, which could be muddied by many relatively easy challenge calls. A pitch 6 inches off the plate called a strike is much easier to identify as a challengeable pitch than one that just barely missed the edge by a smidge. Run creation tells you about value creation, but accruing it is reliant on the player having had the opportunity to challenge in high potential run expectancy situations. WPA tells you about impact to a team's probability of winning a game, but again its accrual is reliant on the player or team being put in high leverage situations. These metrics don't tell you about the decisions the players didn't make, which pitches they correctly let go, or how difficult each decision actually was, and that's what CDQ is for.

What are we measuring?

bCDQ (Batter Challenge Decision Quality) evaluates every called strike a batter sees when their team has at least 1 challenge remaining.

cCDQ (Catcher Challenge Decision Quality) does the same thing for catchers on called balls.

For each of these pitches, we already have a model predicted probability that the pitch gets challenged. A borderline pitch right on the edge of the zone might have a 15% xChall%. A pitch 6 inches off the plate has an expected challenge rate that approaches zero. The models are trained on how often batters and catchers challenge pitches in similar situations based on proximity, count, leverage, challenges remaining, and some other factors.

bCDQ and cCDQ use the expected challenge probability as a "difficulty" weight. Before I explain the formula, let me show you the model accuracy because seeing it is the foundation for trusting and understanding CDQ.

The two models

There are two separate logistic regression models. The offense model (oxChall%) predicts how likely a batter is to challenge a given called strike. The defense model (dxChall%) predicts how likely a pitcher or catcher is to challenge a given called ball. For cCDQ, we multiply dxChall% by 0.96 to get the catcher's share specifically, since catchers initiate about 96% of defensive challenges (I think at the time of me posting this it's around 95% but I'm gonna take a healthy bet that pitcher rates shrink down below the 4.4% rate from Spring Training and leave this hardcoded for now).

Are the models accurate? Here's the calibration chart for both models against 2026 regular season data 1 week into the season. The x axis is what the model predicted, the y axis is what actually happened.

Both models track the perfect calibration line closely. When the model says a pitch has a 5% chance of being challenged, about 5% actually are. When it says 10%, about 10% are. The one data point that falls off a bit at the end for defensive challenges is likely due to the lean sample sizes we're working with at the start of the season and I fully expect it to tighten up as more data comes in.

The CDQ formula

Now that you understand what xChall% represents, here's how CDQ uses it. Every eligible pitch falls into one of four buckets:

Ump was wrong, player challenged: reward = (1 - xChall%)

You correctly identified a blown call and acted on it. The reward is bigger when the challenge was less likely to happen. If 25% of players would challenge that pitch, you get +0.75. If 50% would, you get +0.50. This metric values spotting what others miss more than spotting what everyone sees, but is balanced by heavily detracting from the aggregate if you miss a risky challenge:

Ump was right, player challenged: penalty = -(1 - xChall%)

You wasted a challenge on a correct call. The penalty is bigger when the challenge was less likely to happen. If only 2% of players would challenge that pitch, you get -0.98 for a terrible read.

Ump was wrong, player didn't challenge: penalty = -xChall%

You missed a real opportunity. The penalty is bigger when it was more obvious. If 60% of players would have challenged, you eat -0.60 for letting that slide. If only 3% would have, the penalty is just -0.03 because it was some combination of genuinely hard to spot or not worth the challenge given the game state.

Ump was right, player didn't challenge: reward = +xChall%

You showed restraint on a pitch where challenging was tempting. If the model predicts that 10% of players would challenge, you get +0.10 for holding back. If you're the catcher and the pitch is spiked in the dirt 2 feet from the plate, the model may reward you with nothing for not challenging.

CDQ Output

If you challenge at exactly the rate the model predicts, your expected CDQ is zero. Positive CDQ means better decisions than average. Negative means worse.

The average bad challenge costs about -0.89 CDQ points because most challenged pitches have a low xChall%. A correct challenge rewards about +0.78 on average. The ~96% of pitches that aren't challenged contribute small amounts (+0.01 to +0.15 each, most on the much, much lower end).

Who's at the top?

CS = called strikes seen, CB = called balls seen, CC = correct challenges, BC = bad challenges, MO = missed opportunities.

Edgar Quero

In my opinion the most interesting output of this analysis is that Edgar Quero is dead last for catchers at -3.47 cCDQ while being 5th in Catcher Net Runs at +0.99. He's won enough high run value overturns to look good in run value terms, but cCDQ sees the full picture: 7 bad challenges and 9 missed opportunities alongside those 5 wins. Here's a full look at every called ball Quero's seen at the time of me writing this:

If you're curious what some of these correct non-challenges with no value being given to Quero look like, here are some examples: 1 2 3 4 5

bCDQ and cCDQ are live on the Players and Teams pages.

Thanks for reading,

Nate

Originally posted at 8:55AM ET on 4/4/2026