Okay so let's talk about this Watchtower thing

Recently, Commenter Matt—longtime friend of this blog—challenged several of the principles this Watchtower thing is built on. Many of you weren't around for the Watchtower's genesis, and I've been doing these things for three years now. Now's a good time to air this stuff out, explain my methodology, and make sure I haven't made a wrong turn somewhere in my execution.

The original mission of the Watchtower was to find out what the new-look Lions were going to look like. To, like the forest ranger in the old Lassie episodes, use a combination of familiar landmarks and mathematics to pinpoint a direction and distance to the truth.

With so much of the new-look Lions roster being turned over from the year before, we didn't know what to expect when they took the field that first week. Since Linehan and Cunningham have long track records with distinct schematic styles--and, because I've always thought the schematic interplay between offense and defense is overlooked--I  decided to analyze historical matchups between the Lions' coordinators and their first opponents'.

I found their season averages of points scored and points allowed, saw how each team performed relative to their averages when facing the other coordinator, and concluded the following:

As all three metrics of output--per-play passing, per-play rushing, and points scored—are way above their averages for the season, I’m going to say that given superior talent and execution, and/or excellent pass protection, Scott Linehan’s balanced offense disproportionately gives Gregg Williams’ attacking 3-4 defense fits.

Given equal talent, Cunningham’s hyperagressive 4-3 is extremely effective against Payton’s pass-heavy offense, but only if that aggression leads to mistakes and turnovers—otherwise, the holes in the defense will be exploited. Effective quarterback play may neutralize the defensive advantage.

Therefore, the most probable outcome of this game is a shootout that the Lions lose.

I didn't project a score, and didn't project passing or rushing effectiveness. I just tried to retell the story of the upcoming game, as clearly as I could read it in the data. Over time, I've added more data, made more projections, and reviewed the articles and methodology for their accuracy and completeness. I emphasized method's qualities as predictive tool because it showed great promise as such--and, I came to discover, that's what people want: a prediction.

However, it is still a study of offense vs. defense, and analysis based purely on per-play effectiveness and scoring output has limitations. Sacks, turnovers, penalties, injuries, special teams play, weather conditions, crowd noise, and clock management all dramatically affect the relationship between "how good is the team doing" and "how many points the team has scored or allowed". Look no further than the Lions’ 2009 trip to Chicago for proof of that: the Lions outgained the Bears 440-292, but lost 24-48.

When reviewing historical games, I always check for these factors to see if they're a more likely cause for unexpected outcomes than "Ol' Gun's just got Coach So-and-so's number," and I not only note that but build it into the prediction. Frequently, Cunningham's defenses have held offenses to lower totals than you'd expect based on their relative talent levels, frequently because Gun's Ds are designed to get sacks and force mistakes.

Matt said:

I totally disagree that fumbles and fumble recoveries are "random." Unpredictable, sure, but not random.

Here's some reading material on that:

Football Outsiders Basics: "Fumble recovery is a major reason why the general public overestimates or underestimates certain teams. Fumbles are huge, turning-point plays that dramatically impact wins and losses in the past, while fumble recovery percentage says absolutely nothing about a team's chances of winning games in the future. With this in mind, Football Outsiders stats treat all fumbles as equal, penalizing them based on the likelihood of each type of fumble (run, pass, sack, etc.) being recovered by the defense. Other plays that qualify as 'non-predictive events' include blocked kicks and touchdowns during turnover returns. These plays are not 'lucky,' per se, but they have no value whatsoever for predicting future performance."

Advanced NFL Stats: “The ratio is not significant for wins, points scored, or points allowed while overall fumbles and fumble rates are significant. This result indicates recoveries are indeed random. It also suggests that fumbles, and not fumbles lost is a better stat for estimating a team's future likelihood of fumbing.”

CougarBoard.com: “Look at LSU. They were 70th and 102nd in 2005 and 2006, then suddenly finished 1st in back-to-back years in 2007 and 2008. Oklahoma went from 118th to 1st in the space of two years. BYU went from 112th to 2nd in two years and two years later were back at 95th. There is no pattern here. If you take a histogram of average fumble lost rankings by team, you get a tight, centralized cluster which is the hallmarks of a random distribution.”

As the New York Times explained, Jim Schwartz once told Bill Belichick the same thing: "“Fumbles are a random occurrence,” Schwartz said he told Belichick. “Being able to get interceptions or not throw interceptions has a high correlation with good teams. But over the course of a year, good teams don’t fumble any more or less than bad teams. Bill didn’t agree. He said, ‘No, good teams don’t fumble the ball.’ But actually, they fumble just as often as bad teams.”'

Long story short, Matt is right that fumbles themselves aren't completely random. Daunte Culpepper fumbled the ball 80 times in 73 games as a Minnesota Viking; that's definitely his fault and not pure misfortune. But in general, using a team's current, or recent, fumble rate and projecting it forward to the next game simply won't be accurate.

When it comes to recoveries, there's no doubt--once that ball hits the turf, it's statistically random. Just look at the ridiculous tip drills that occurred last Sunday. There were several near-heroic individual efforts on the part of the Buccaneers to force that fumble, keep it in play, and come up with it; that it ended up where Rob Sims could make a play on it was pure chance. The Bucs could have put their hands in their pockets and gotten the same result!

The same thing goes for interception and fumble returns; whether they can be returned for scores depends more more on the spot on the field they occur than on team defensive ability. In my opinion, Matt is right that the Will Heller tip-pick was caused by a combination of Stafford throwing it too high and too outside, Heller batting it up instead of down, and Talib having excellent awareness and ball skills.

However, Talib picking that pass off with nothing but green turf and white stripes between him and the end zone was purely a matter of location. Chris Houston's interception over Arrelious Benn's head was no less outstanding a play, but he was in no position to return it. If the same players had made each other's plays, the result would have been the same.

The upshot of this: even though fumbles, fumble recoveries, special teams play, return touchdowns, and the like are either completely random or completely unpredictable, Matt’s right that without accurately accounting for them I’ll never be able to accurately predict the scores of NFL games. In fact, Advanced NFL Stats concluded that 42% of all NFL wins-and-loss records are determined by random variance, not relative performance. There’s a very hard cap on how close my projections can get . . . that’s why I remove, or adjust for, as much of these non-predictive factors as possible and then see how close I got.

Maybe I’ve taken my eye off the ball a bit. Maybe I’ve been too focused on the score projections, and the accuracy thereof. I really only do it because, as Brian Cook from MGoBlog says, “the strictures and conventions of sportswriting compel me.” What do you folks think? Is The Watchtower a worthwhile read even if I don’t take a stab at the score? Should I be reviewing every Watchtower I write?