Hidden Game: How a cult book from the 1980s foretold the future of football and the fourth dow

The package arrived in early January, and out slipped a single hardcover book, its dust cover a retro shade of Kelly green, its title spelled out in orange, ‘80s-style block letters, its pages filled with charts and tables and arguments. “Revolutionary,” the subtitle promised.

There, on my kitchen table, was a used, dog-eared copy of “The Hidden Game of Football,” a 1988 book authored by three pioneers from sports’ first age of statistical enlightenment, an opening salvo in a battle that had not yet begun. Its 415 pages include ideas, formulas and, by today’s standards, some hokey wordplay. As one data scientist put it, it remains “the cornerstone of how we all kind of think about football now.”

Advertisement

I ordered a copy in mid-December, four days after Chargers coach Brandon Staley triggered another round of debate after an overtime loss to the Chiefs, and 19 days before he and quarterback Justin Herbert would do the same in a Week 18 loss to the Raiders. In the first instance, Staley had gone for it on fourth down three different times when in short field goal range — including two failures on fourth-and-goal in the first half — which set off the typical arguments about the use of analytics and data to determine in-game strategy. (“I don’t think that any decision that I made tonight was a gamble,” Staley told reporters.) In the second instance, the Chargers had failed on a fourth-and-1 from their own 18 in the third quarter while trailing by three, a decision supported by advanced models but which, for a moment, turned the noisy din into a roar.

The discourse was predictable. More interesting was how we arrived here, to an NFL that saw unprecedented levels of fourth-down aggressiveness for another year, to a league where the Lions set a record with 41 fourth-down attempts, where Staley’s Chargers followed the data to increasingly aggressive ends, where seven teams went for it on fourth down at least 30 times. (In 1988, for comparison, just two teams recorded even 20 fourth-down attempts.)

This was the football of the future, as foretold by “The Hidden Game of Football,” one influenced by better statistics and new analytics, where coaches would have data to shape their decisions, which is slightly ironic because almost nobody really read it at the time.

The authors — Pete Palmer, John Thorn and Bob Carroll — borrowed from earlier ideas, invented others (the stealth value of the arm punt!) and developed a framework for assigning value based on field position and down and distance. The process created a primitive version of “win probability,” a statistical concept that allowed one to give value, in the form of expected points, to a play, a series, a chunk of yards or a turnover, which made it essentially the flux capacitor of football analysis.

Nearly 12 years after its publication, a radio DJ and writer named Aaron Schatz found a copy of “Hidden Game of Football” in the remainder pile of a New Jersey bookstore and was inspired to create a statistic called DVOA — Defense-adjusted Value Over Average — and a website called Football Outsiders. Nearly 20 years after that, NFL coaches such as Staley recommend books like “Think Again,” by Penn psychology professor Adam Grant, and extol the virtues of data and process. “It’s not anything new,” Staley said last year. “It’s just a fancier term for information.”

Advertisement

Even with Staley at home, football’s era of Big Data could determine a playoff game or who hoists the Lombardi Trophy. So to understand the future, I decided to start with a 33-year-old book. The story of “The Hidden Game of Football,” however, is not just one of numbers. It includes a journeyman quarterback from BYU, a cameo from Bill James  and a radar installation in Alaska built to monitor the Russians.

Mostly, it starts with an 83-year-old retired engineer in New Hampshire.

Pete Palmer was an unlikely football revolutionary. He spent most of his life as an engineer at Raytheon, the U.S. defense contractor. On Sundays, he worked the stats crew for the New England Patriots at Foxboro Stadium, though no one from the team asked for his views on the sport. He was bookish and practical, a quant, in other words, the kind of person who buys a Toyota and runs it into the ground, who didn’t purchase his own personal computer until he was 50.

When John Thorn, a young baseball writer, met Palmer in 1981 at a convention for members of the Society for American Baseball Research, he was struck by how deeply involved Palmer was in the subjects that interested him. “Nerdy,” Thorn told me. He liked Palmer immediately.

By then, Palmer was 43 and one of leading minds (read: only) in the nascent field of sabermetrics, the mathematical analysis of baseball stats. He’d grown up in Wellesley, Mass., where he fell hard for the Boston Red Sox. He came of age during the era of Ted Williams. As a kid, he’d buy up stacks of baseball cards and use the stats on the back to sort players by various categories. “Bookkeeping,” he said. He headed off for Yale in the late ’50s, watching the football team closely but never diving into the numbers. When he graduated in 1960, he took a job as an electrical engineer at Raytheon, its headquarters not far from his home. When he had spare time, he returned to baseball stats.

Advertisement

In the early days of baseball, back in the 19th century, the sport’s organizers and boosters had started keeping statistics, in part to offer a sheen of seriousness to a child’s game. The tradition persisted into the 20th century, in the form of box scores and records, but when Palmer started investigating the numbers in the 1960s, almost no one was doing serious academic work. Brooklyn Dodgers executive Branch Rickey hired a man named Allan Roth to be the sport’s first full-time analyst in 1947, and an engineering professor named Earnshaw Cook challenged baseball conventions in the 1950s and ’60s, but when Palmer read a book from the 1950s that supposedly offered analysis, he figured he could do it better.

In the beginning Palmer set out to understand the value of a run, to see how team runs scored might correlate with winning, and how those “wins” could be credited to players. He found that approximately 10 runs equaled one win, which allowed him to create a new system for ranking players. When he wrote an article on his findings and sent it to the Sporting News in 1969, they declined to run it. Too strange. “I got a letter back accusing me of being a screwball,” he said.

By the 1970s, the analytical movement had begun to grow, meaning there were nine or 10 people doing it instead of three. It was a world, for the moment, almost solely contained to baseball, one of eccentrics and contrarians and outside innovators. One of those figures was James, who was 11 years younger than Palmer and trying to tackle baseball questions from his home in Lawrence, Kan. “Nobody was making any money doing this stuff,” James told me via email.

James recalls meeting Palmer at a SABR meeting in the mid-’70s, an early event during which baseball nerds from across the country would congregate in a like-minded hive. To James, Palmer was friendly and modest, a man interested in the same big, bizarre questions about baseball. “The very large questions that hang down from the sky,” as James puts it. What is the best way to measure offensive production? Why do teams win the pennant? How much does defense affect pitching? The men were after the same answers, but in the case of Palmer and James, their central methods were different.

James took what he calls “a trail of breadcrumbs,” approach. He’d ask himself a million little questions — some trivial, some ultra specific, some that might lead to some other idea — and only then could he begin to get to the big stuff, to the stuff that really mattered. Palmer could be more direct. “I don’t have Pete’s mathematical training or skill,” James said.

What reassured both men was that they often ended up at the same point. As the scene matured, James came up with the formula for Runs Created. Palmer invented Linear Weights and realized the usefulness of OPS — the now ubiquitous on-base percentage plus slugging. James started writing his “Baseball Abstract” in 1977, before it became a publishing phenomenon, opening the market for others. Palmer wondered if his analytical methods could be used for other sports. By day, he had his job at Raytheon, where he worked on Cobra Dane radar installation in the Aleutian Islands, which gathered intel on the Soviet Union’s missile program. By night, he had his office’s IBM computers to himself. He also had a football test he wanted to run.

If baseball was in the first days of a sabermetric revolution, football in the early 1980s was just starting its great awakening. To borrow from a later passage in the “Hidden Game of Football,” baseball was a linear game in love with its numbers, a simple contest of hitter versus pitcher, one dimensional, forever moving forward. Baseball, the book noted, was a thread. Football, meanwhile, was a fabric, and that made it harder to measure.

Advertisement

That’s not to say that folks weren’t trying. In 1983, a group at Indiana University — including pioneering statistician Jeff Sagarin — built a computer program to help the school’s football coach, Sam Wyche, tailor his game plan for certain situations. “Literally tells me what should be the best call next,” Wyche told the local newspaper. It was the kind of innovation that excited Palmer. In fact, it was similar to a concept he’d discovered years earlier, when he working for the MITRE Corporation in the early ’70s. The organization, Palmer said, had an impressive library full of research journals and academic papers, and he often found himself searching the library’s index for those related to sports. In those days, he would stumble upon quite a few focused on baseball, but one day, as he remembers it, he was surprised to find one about football in the journal “Operations Research”.

It was authored by Robert E. Machol, a professor at Northwestern, and Virgil Carter, a quarterback for the Bengals who was attending graduate school in the offseason. In early 1970, Carter had coded every play of the 1969 NFL season, using more than 8,000 punch cards, and determined the expected point value on first down from various points on the field. The study laid a foundation for future statisticians — and merited a wire story on the nerdy quarterback. (“We have developed a probability theory,” Carter told a reporter. “While at times the opposite situation might prevail in practice or under game conditions, the computer study in the long run will prove to be accurate.”) But for years, nobody came along to build much more.

By the early 1980s, that had started to change, and with James’ work convincing publishers there was an audience for analytics, Palmer and Thorn started to collaborate. First came “The Hidden Game of Baseball,” published in 1984, and when that sold well, they landed a multi-book deal with Warner Books. As was often the case with statistical projects, the football book was somewhat ancillary. Thorn estimates that the author’s advance for “The Hidden Game of Football” was around $50,000. (The big book was 1989’s “Total Baseball.”) They recruited Bob Carroll, the founder of the Professional Football Researchers Association, to help with the writing, while Palmer set out to confirm the research of Carter and Machol.

Using data from the 1986 NFL season and thousands of punch cards, Palmer duplicated the basic findings from Operations Research, and then, because he had more numbers, he went a step further. He found that a team’s point potential increased by one every 12.5 yards, though because most drives, on average, result in no points it actually took 15 or 16 yards to produce a point. He found that a turnover produced a four-point swing, no matter where it happened on the field. He found, according to the numbers, that teams should almost never kick a field goal unless they have at least six yards to go on fourth down. “Pete was the genius,” Thorn said.

The book came out before the 1988 season, the one that ended with Joe Montana spotting John Candy at the Super Bowl. It argued for coaches to be more aggressive; it questioned the formula for passer rating; it considered abolishing extra points. It attacked football orthodoxy and sought to answer big questions. And then something funny happened: Well, nothing. Nothing happened. It didn’t sell particularly well. None of the authors heard from NFL or college coaches. It received a mixed review from The New York Times. “The trouble is that while baseball statistics are fun to think about, these football statistics seem cumbersome,” the review said.

“It’s not like a lot of people in football bought this book so they could ignore it,” said Schatz, the creator of Football Outsiders. “They never even heard of it.”

Somewhere along the way, people started referring to “The Hidden Game of Football” as seminal, as in a seminal work of football analytics, which always entertained Palmer because for many years it was essentially lost in the ether, doomed to the same fate as Virgil Carter’s original research. It was what you might call a cult book, though not necessarily a cult classic. As Thorn said, “Innovation is rapid; adoption is slow.”

Advertisement

That changed some when they released a revised edition 10 years later. It changed more when Schatz started Football Outsiders and Brian Burke founded Advanced Football Analytics in 2007. The internet offered more tools and more data and a megaphone that football people at least had to ignore. The models became more complete. The statistician Dean Oliver created QBR, an improved metric for evaluating. The area of player evaluating has never been more detailed. Eric Eager, the director of research and development at Pro Football Focus, estimates the field of football analytics has grown more in the last decade than it had in the previous 40 years. It’s not just fourth downs anymore.

“It’s funny this stuff that is now taken for granted was once sitting around collecting dust,” said Michael Lopez, the NFL’s director of football data and analytics.

The sport of football today looks closer to the one envisioned in “Hidden Game of Football” than the one played in 1988. It makes you wonder what other ideas are floating out there in the ether, forgotten for years, ready to be picked up.

Back home in New Hampshire, Palmer is set to turn 84 later this month. He officially retired from his day job in 1998, and while he continued to keep stats and maintain his historical database, he never worked for a team or became synonymous with the field like his contemporary James. Back when nobody cared about any of this, he invented OPS and advanced the concept of expected points, a side hobby that allowed him to put his kids through college without any student loans. But money was never really the motivation, either. He never really considered how others might relate to his work, he says. He just enjoyed doing it. It was fun.

Palmer, though, still has one gripe when he watches football. It’s about the coaches.

Too many of them, he said, are scared to follow the numbers.

(Photo of Justin Herbert: Stephen R. Sylvanie / USA Today)

ncG1vNJzZmismJqutbTLnquim16YvK57kmltcWhlaXxzfJFrZmlpX2aBcLTInZuepl2crq6xjKGmsGWRYrC2uNNmmainm2Kzs7vMZquhnV1mhnl80madqKqVqbytsIytn55llqrBtr7EZqafZZakvLWuwKWjZpmemXq1tMRmnaitoqm1brDOsKVmnJWXrrWxjg%3D%3D