Teams with higher CQBR win 75% of the time

By
Updated: April 1, 2013

Complete Quarterback Rating (CQBR) differential is extremely effective at predicting winners in individual games, but no statistical formula can predict a Mark Sanchez butt fumble.

You just learned that Complete Quarterback Rating (CQBR) accounts for 33 percent of variation in a team’s win/loss record. Do you still doubt the magnitude of the quarterback importance? Do you question the utility and validity of  the ingenious WCSN CQBR? Did you eat paint chips as a child? Are you Trent Dilfer?

If you answered “yes” to either of the first two questions, we present you with some more evidence to jam in your math-hole. If you answered “yes” to either of the last two questions, the math will probably slip by you: don’t fear, we’ve included some pretty pictures for your entertainment.

Winning individual games

The analyses below focus on the outcomes of individual games, rather than entire seasons as has been more common in our previous analyses. Here, we examine all 2012 NFC North and NFC South divisional games and all games between the two divisions.

In addition to CQBR, these tests looked at CQBR differential (CQBRdiff) and scoring differential (SCOREdiff) in each game. By CQBR differential, we mean the difference in CQBR put up by the two teams in any one game. Where more than one quarterback played for a team in a given game, their CQBRs were averaged to produce a team CQBR.

Because we know who won and lost, we can assess the level to which differences in the quality of overall quarterback play in a single game influence that game’s outcome. There are many available statistical approaches that can test for this effect, so we ran multiple tests to demonstrate as clearly as possible the precise, numeric impact of quarterback play on wins and losses.

To the numbers!

T-party

Leading off this round of statistical battery is the humble t-test, which compares means between two groups (in this case winning teams vs. losing teams), and determines the probability that those means are different. In the case of CQBR, the mean for winning teams of 77.3 is in fact significantly higher than the mean of 52.7 for losing teams (P < 0.01), such that winning teams tend to have substantially higher CQBR scores. (See Figure 1.)

Figure 1. In any given game The average CQBR for the winning team is significantly higher than the average CQBR for the losing team. The errors indicate that there is a great deal of variability in CQBR differential.

Figure 1. In any given game The average CQBR for the winning team (77.3) is significantly higher than the average CQBR for the losing team (52.7). The wide error bars indicate that there is a great deal of variability in CQBR differential.

Regressive behavior

Simple linear regression analysis, which has been explained elsewhere on this site, determines whether two variables are related to each other in a systematic way, such that higher or lower values of one variable are consistently associated with higher or lower values of a second variable.

In this case, we wanted to see if CQBRdiff was significantly and positively related to SCOREdiff, with the expectation that better quarterback play is associated with a greater likelihood of outscoring your opponent, and thereby winning the game.

As hypothesized, SCOREdiff is significantly and positively related to CQBRdiff (P <0.01), with variation in CQBRdiff explaining roughly half of variation in SCOREdiff (R2 = 0.55). Yes, that’s correct: a full half of the points differential in a game can be attributed to the quality of quarterback play. It’s difficult to find a better way to emphasize the importance of the quarterback position; remember, a quarterback is only 4.5% of a team’s starting players and only 9.1% of the offensive personnel on the field at any one time.

The slope of the relationship is 0.19, whereas the intercept was not significantly different from zero (P = 0.99). Thus, the equation:

SCOREdiff = 0.19 ∙ CQBRdiff

For every 1 point of CQBRdiff, then, a team can expect to outscore its opponent by 0.19 points. That means that a difference in CQBR of just 5.3 equates to a one-point advantage, and thus a winning margin, for the team with better quarterback play. The average 24.6 point difference in CQBR between winning and losing teams seen above translates to a  scoring differential of ~5 points. See Figure 2.

Figure 2. Scoring differential is positively correlated to CQBR differential. For every 1 point of higher CQBR differential, a team can expect to outscore its opponent by 0.19 points.

Figure 2. Scoring differential is positively correlated to CQBR differential. For every 1 point of higher CQBR differential, a team can expect to outscore its opponent by 0.19 points.

If you don’t no mi bi now, you can call me Al

With sincere apologies to Harold Melvin and Paul Simon, we next turn to the binomial test, which assesses whether the proportion of cases in two groups differs from a predetermined hypothetical proportion. In this case, we use 0.50 (50%) as the hypothetical proportion, which is the case of assignment to two groups is equal to chance. This is analogous to the probability of getting either heads or tails in a coin flip.

In this case, the two groups are:

  • Group 0: Cases where a team had a positive CQBRdiff and won the game, or conversely, had a negative CQBRdiff and lost the game.
  • Group 1: Cases where a team had a positive CQBRdiff but lost the game, or conversely, had a negative CQBRdiff and won the game.

The null hypothesis is that individual cases will sort into the groups according to chance, and would thus be split evenly between the groups (50% in Group 0, 50% in Group 1). Our alternative hypothesis, which we can derive from the above regression analysis, is that cases should not sort according to chance but rather according to a positive relationship between CQBRdiff and SCOREdiff (the latter of which ultimately equates to wins). Thus, we expect cases to fall into Group 0 more often than in Group 1.

In fact, that is exactly what the binomial test finds: the cases split unevenly, with 75% falling into Group 0, and just 25% falling into Group 1. This pattern is significantly different from chance (P <0.01), supporting the hypothesis of a positive relationship between CQBRdiff and winning. See Figure 3.

Figure 3. The team with the higher CQBR wins and the team with the lower CQBR loses approximately 75% of the time, while the team with the lower CQBR wins and the team with the higher CQBR loses approximately 25% of the time.

Figure 3. The team with the higher CQBR wins and the team with the lower CQBR loses approximately 75% of the time, while the team with the lower CQBR wins and the team with the higher CQBR loses approximately 25% of the time.

Ratio Discrimination, or: The Eigenvalue Returneth

Admittedly, there are no ratios involved in this next analysis, but a solid pun can’t be left out simply for old-fashioned notions of accuracy in language (sorry, George Orwell). You all certainly have fond memories of the Eigenvalue from previous SportsNerd principle components analysis articles, and now it’s back! But this time our friend is hanging around with a different-but-related statistical tool, the discriminant function analysis.

Discriminant function analysis uses known group membership from a ‘‘training sample’’ to produce linear equations (functions) that describe how groups vary in terms of specified independent variables. Function values are then calculated and used to assign individual cases to groups. In this case, the training sample is our aforementioned collection of games, there are two groups (winners vs. losers), and the independent variables are raw CQBR and CQBRdiff, which we analyze separately.

CQBR and CQBRdiff each produced significant discriminant functions for sorting cases into winners and losers (CQBR: Eigenvalue = 0.16; Wilk’s λ= 0.863; P < 0.01; CQBRdiff: Eigenvalue = 0.45; Wilk’s λ= 0.691; P < 0.01). See Figure 4.

Figure 4. Discriminant function analysis shows that the winning team typically has a positive CQBR differential, while the losing team typically has a negative CQBR differential.

Figure 4. Discriminant function analysis shows that the winning team typically has a positive CQBR differential, while the losing team typically has a negative CQBR differential.

To test how well these functions worked to classify the training sample, we ran a “leave-one-out” test that classifies each case by re-calculating the functions while excluding that specific case, and compares the classification to the known group memberships. For CQBR, the function correctly classified 60.7% of cases; for CQBRdiff the accuracy rate was higher, at 71.4%. Thus, the CQBRdiff function will accurately name the winner and loser of a game almost three-quarters of the time. Not too shabby. See Figure 5.

Figure 5. CQBR differential correctly predicts the winner of a game 71.4% of the time.

Figure 5. CQBR differential correctly predicts the winner of a game 71.4% of the time.

These functions can be validated in the future by bringing in new quarterback data — say, from the AFC East — and seeing how well the functions classify winners and losers. They will, however, never be capable of identifying butt-fumblers, which is a serious limitation. Ultimately, this type of analysis could be used to generate a set of algorithms that account for nearly all aspects of the game (other than weather, touchterceptdowns, and clown riots), to provide the basis for predicting the outcomes of individual games.

Don’t hold your breath for that one, though. It could take awhile . . .

Fast facts

For those of you less nerdily inclined who skipped to the bottom as quickly as possible, here is a brief summary of the findings:

  • The winning team averages a CQBR of 77.3, while the losing team averages 52.7.
  • For every one point of CQBR differential, a team can expect to outscore its opponent by 0.19 points.
  • The team with the higher CQBR wins approximately 75 percent of the time.
  • The winning team usually has a positive CQBR differential, while the losing team usually has a negative CQBR differential.
  • CQBR differential predicts the winner of the game approximately 71.4 percent of the time.

About the author

SportsNerd columnist Co-winner of the prestigious Water Cooler Sports Award for Best Statistical Study, Andy Froehle is proud to call the Green Bay Packers his team (literally: he has a worthless piece of paper that says so!). Andy uses statistical analysis extensively in his work and is interested in applying the same kinds of techniques to analyzing football data, with the chief goal of making your math-hole bleed a little bit. Feel free to contact him with any questions, comments, rants, raves, suggestions for future analyses at afroehle@watercoolersports.net. He also accepts donations in cash or in kind.

669 comments
adambballn
adambballn like.author.displayName 1 Like

Nice try Redbox... could have done better.

andylet445
andylet445 like.author.displayName like.author.displayName 2 Like

Finally i can see Nate's POV about public transportation

robertj72
robertj72

Lamar Odom is getting torn a new one on Inside the Lines right now!

JVince 11
JVince 11 moderator

 FUCK that show and every one on it... that show is complete bullshit that spells rumors before they can even talk about them.... it is because of that fucking show that the whole Ryan Braun thing even happened.. it is policy that nothing is to be released until after the appeals process.. Braun appealed and the whole thing was over turned... it shouldn't have come out at all... PERIOD. That Ley guy can suck a fat cock.

Maized and Confused
Maized and Confused moderator like.author.displayName like.author.displayName 2 Like

  

Tell us how you REALLY feel.

JVince 11
JVince 11 moderator like.author.displayName 1 Like

   "Skip to page 12 if you agree.. or page 45 if you don't" "if you use a MAC skip to page 90"

Maized and Confused
Maized and Confused moderator

   

I feel as if my background at Google changes every time you mention it.

It's like one of those "Choose your own ending" books.

JVince 11
JVince 11 moderator

   oh... and fuck guys with mustaches that worked as an intern for google.... buncha hipsters.

JVince 11
JVince 11 moderator

The Milwaukee Journal-Sentinel reports that the Packers are close to finalizing an extension with Clay Matthews.


For Woude.

tmonson78
tmonson78 moderator

 

Still not a link...

jwoude23 Bear Down
jwoude23 Bear Down moderator

 cool.  I figure it's coming, just wondered if you had anything with numbers.

I saw this morning rumor was that Pack had offered Rodgers $21M per, he wanted $23M per.

JVince 11
JVince 11 moderator

  I heard a few weeks ago that it would be wroth 12M per year for Matthews... Rodgers thing isn't on 2M per year salary wise.. it is 2m Per year on where his signing bonus will be distributed.. Packers want it to be almost all up front because they have more cap space this year and next but Rodgers wants it spread out but that would raise his cap hit each year.

JVince 11
JVince 11 moderator like.author.displayName like.author.displayName 2 Like

NFL.com's Ian Rapoport confirms the Packers will explore an extension with LE B.J. Raji this offseason.


Maized and Confused
Maized and Confused moderator

 

Has PFF weighed in on this?

JVince 11
JVince 11 moderator

  There are no currently blocking NT metrics on PFF.

jwoude23 Bear Down
jwoude23 Bear Down moderator

Preparation_A
Preparation_A

  

Son of a bitch. We've already answered some of the questions posed in that shit.

jwoude23 Bear Down
jwoude23 Bear Down moderator

woah, long-ass link.  Interesting article about the value of RBs 

SDL
SDL

 

bitly.com can help

jwoude23 Bear Down
jwoude23 Bear Down moderator like.author.displayName 1 Like

  i know, just didn't realize it would be this long.  Link still works though

JVince 11
JVince 11 moderator like.author.displayName like.author.displayName 2 Like

GB has an extension in place for both Matthews and Rodgers...