Just How Accurate Are DVOA Mean Win Projections?

Yesterday, Chris Harris Jr. and other Broncos took much issue to an article on ESPN that projected the defending Super Bowl champions to go 7-9 in 2016.  The author of the article in question was Aaron Schatz, founder of Football Outsiders and the concept of DVOA.  FO has been a read of mine for a long time, as long as Schatz started up DVOA anaylsis in 2003.

Schatz’s bearish forecast on the 2016 Broncos hits on some common points he’s made in the past: extreme sides of the ball, good or bad, tend to regress to the mean, and performance on defense is more inconsistent than offense from season to season. Throw in a projection of Paxton Lynch more likely to “bust” than not, and Schatz sees a team that’s more like the Rams teams of recent years than a repeat champion.

As I said above, DVOA projections have been run for the past 13 seasons.  So given the disrespect that Schatz has inadvertently injected in the Broncos’ locker room, let’s take a look at that track record.

First, it’s important to note that what’s being dealt with here are mean wins.  FO runs many simulations of the season in question based upon their data, and averages the resulting records together.  So a team projected to have around 10 mean wins will have a few projections in which everything goes right and the team goes 13-3 or better, and a few in which everything goes wrong and the team has a losing record.

As a result, when FO projects mean wins based upon DVOA, it’s unlikely to see teams with either a very good or very bad record.  After rounding to whole numbers, I count only 13 of 416 team seasons in which a team was projected to have more than 11 mean wins or fewer than 5.  For this reason, as I evaluate DVOA mean win projections of the past, I will normalize on a scale that has wins, both projected and actual, in a range from 11 to 5.  Think of it as a table like this:

Wins Type of Season
11+ Great
10 Good
9 Above Average
8 Average
7 Below Average
6 Bad
5- Terrible

Using this normalized scale, how do the DVOA mean win projections look? Well, it means that FO overrates the entire league by about 0.024 wins.  That’s a pretty thin margin.  So is it time for Broncos fans to start gulping? Well, let’s take a look at the team-by-team breakdown for a rosier picture.  The section on the left indicates the difference in mean wins to actual wins. Positive numbers mean that DVOA mean wins underrated the team, while negative numbers means it overrated the team.  On the right, the standard deviation of the differences are listed, sorted from lowest to highest.  The lower the standard deviation, the more consistent the team was underrated or overrated.

Team Diff # Std. Dev. #
IND 1.154 1 2.115 13
CIN 1.038 2 2.005 5
ARI 1.000 3 2.483 24
DAL 0.769 4 2.862 30
HOU 0.615 5 2.022 7
MIN 0.500 6 2.041 9
DEN 0.462 7 1.854 3
CAR 0.423 8 2.235 14
NO 0.385 9 2.364 22
SF 0.385 10 2.256 16
NE 0.385 11 0.768 1
PIT 0.308 12 2.016 6
BAL 0.154 13 2.609 26
NYG 0.154 14 2.075 10
GB 0.115 15 2.103 12
ATL 0.077 16 2.871 31
KC 0.077 17 2.290 18
SD -0.077 18 2.499 25
CHI -0.154 19 2.340 19
BUF -0.154 20 1.864 4
TEN -0.231 21 3.086 32
MIA -0.231 22 2.351 20
STL -0.269 23 2.027 8
NYJ -0.308 24 2.780 29
SEA -0.308 25 2.626 27
DET -0.462 26 2.259 17
WAS -0.769 27 2.713 28
PHI -0.808 28 2.250 15
JAX -1.077 29 2.100 11
CLE -1.077 30 1.754 2
OAK -1.154 31 2.444 23
TB -1.692 32 2.359 21

You’ll see Denver listed prominently there as the 7th most underrated team, and the 3rd most consistent, which is a good thing if you’re underrated.  That looks better, doesn’t it?  The breakdown of each Broncos season is as follows:

 Year Mean Wins Actual Wins
 Diff
 2003  10 10 0
2004  10 10 0
2005 9 11+ +2
2006 10 9 -1
2007 9 7 -2
2008 8 8 0
2009 5- 8 +3
2010 8 5- -3
2011 6 8 +2
2012 8 11+ +3
2013 11+ 11+ 0
2014 11+ 11+ 0
2015 9 11+ +2

The one interesting point I’ll make here is that in 2009 DVOA projected the Broncos to be the worst team in the league, yet they finished 8-8.  One year later, it projected around 8 mean wins, and Denver was the second-worst team in the league.  Just one year too late on that one.

After having taken you on an adventure with numbers, this is where I break from that form and say to step away from the numbers.  Thousands of simulations can be conducted, but only one of them will become reality.  Shit happens in the NFL, there are tons of unknowns when rookies and former second stringers step on the field, and every prognosticator is going to get some correct and some incorrect due to not much more than sheer luck.

Meanwhile, there’s nothing wrong with some subjective guesses on how each team in the NFL will fare in the upcoming season, especially since those are better understood to be guesses.  The FO writers themselves do this every year.  Which is why I thought it was curious that the point of disagreement Schatz had the most with the mean wins DVOA spit out was not with the Broncos, but the Raiders (the ESPN article had them at 6-10).

Schatz isn’t the only one–I’m seeing a growing trend among analysts that are bullish on the Raiders.  If I may enter my own subjective opinion, I sense some underdog bias in play among the masses.  The Raiders have been terrible for the past 13 years, never generating a winning record, and now that they’ve finally shown some glimmers of competency, I feel like heartstrings are being tug to give the poor Raiders a chance.  More likely, the Raiders could vary anywhere from very good to very bad, because again, shit happens.  But I’ll conclude with a couple statements:

  1. If DVOA is right about the 2016 Broncos, I sure as hell hope they’re also right about the 2016 Raiders.  Denver missing the playoffs is easier to swallow when Oakland is even worse.
  2. The Raiders are the 2nd most overrated team in DVOA mean win projection history.
Show Rich Editor
Show Tweet To Image