Bias and art of projecting the College Football Playoff rankings

Now we know what it is going to take.

The release of the first rankings by the College Football Playoff Selection Committee revealed to the world how the football “gods” are looking at the landscape of contenders. There were a few surprises that caught some people unaware. Certainly, Georgia ahead of Alabama is one such surprise. Talk about “poking the bear” … or the tide, as it were.

I’d hate to be LSU right about now.

Husky fans have reacted negatively to the fact that UW showed up as #12 in this initial ranking. Comment after comment and tweet after tweet about UW’s prospects for getting into the playoff indicate that UW fans collectively feel that their odds lie somewhere between the Crimson Tide going to the Super Bowl and Chris Petersen breaking out the Numa Numa dance at his next presser.Let’s set aside for a moment the fact that the “top four” in the initial week 9 ranking has never made it all the way to the CFP intact. Furthermore, let’s pretend for a moment that teams as lowly ranked as #16 in the initial rankings have previously made it into the playoffs. UW fans want to know how a small-time blogger isolated in the frozen tundra of Minnesota can possibly make the argument that UW’s chances are better than you think.

To make this case, we have to first acknowledge the elephant in the room. It is a truth that is as unquestioned as deliciousness of bacon (or, more specifically, Brad’s famous “Bacon Explosion”):

Everyone is biased.

Let me say that again.

Everyone is biased.

Oliver Luck is biased. Barry Alvarez is biased. Condi Rice is biased. Nick Saban is biased. He who shall not be named is definitely biased. Kirk Herbstreit is biased. Chris Petersen is biased.

Gliderdawg, OWW, Norman Einstein, Crazidawg, MikesBike? All of them are biased.

UWDadVanc, Oside, THD1, and dawgincostarica? Them too.

Brad, Gabey, John, Ryan, Jeffrey, and Max? Yup.

Gekko? Oh, hell yeah.

In fact, each of us is biased in more than one way. We are all probably biased in more than ten ways.

How so? Depending on what psychology journal you subscribe to, researchers have identified anywhere between 57 and 212 forms of cognitive bias. Most human beings display several of those biases in their day to day decision making processes.

Authority bias

Egocentricity bias

Lake Woebegone effect

Confirmation bias

Recency bias

Familiarity bias

Ingroup bias

Naive realism

Projection bias

…

The list goes on.

The fact that bias exists and that all thinking human beings suffer from it should not be considered debatable.

What is debatable is the best way to deal with it when making critical decisions such as whether or not to have a baby, picking a mortgage offering, proposing a project to a client and, yes, selecting a college football team into the College Football Playoff.

I suspect many of you believe that the selection committee for the CFP, a group made of up accomplished athletic directors, former coaches, and academic types from all over the country, is a lawless enterprise. That it meets in secret places, smokes cigars, sips on excellent Scotch and engages in unstructured debate on which teams “deserve” to get in until the strongest personalities in the room break the will of the weaker.

Nothing could be further from the truth....

... I think.

The truth is that I don’t know how the committee actually processes data and input to churn out its weekly rankings. Nobody does. It is an opaque process. It might even be secret. However, I am 99.871% certain that they have a method for doing so.

It is the fact that such a method exists that allows for the observation that there are at least some attempts by the committee at controlling the various forms of bias - the kinds of bias that many fans either fear or put into practice in making their own projections - that we know exist in this sort of enterprise.

We can assume that one of the methods that the committee uses is a scored criterion method. This is a basic process for decision-making that almost every decision-making body employs in one form or another. The evidence that such a method is utilized is found in all of the various public comments that have been made related to the use of criteria in the deliberations of the members.

In fact, we know what several of those criterion are: strength of schedule, conference championships, “quality” wins, and statistical excellence.

To show you how such a scoring method might work, I simulated my own CFP ranking process.

In this simulation, imagine that I’m one of the committee members and I have the opportunity to subjectively grade each team against several criteria using just facts in evidence as of the conclusion of week #9. Like the committee, I have access to hard data (things like records, offensive stats, defensive rankings, etc) and debate with my colleagues on “subjective modifiers” that can be used to add context to just the facts.

Below is the scorecard that I produced in my simulation of this process. You’ll see that I subjectively scored each contending team based on real data available (SoS, S&P Rankings for offense, defense and special teams, and record). While I’m certain that the committee uses a different scoring method, isn’t it interesting how closely sorted list resembles the actual CFP rankings as of Week 9? This was my first stab at it and yet I got pretty close to the rankings that came out yesterday

One thing you’ll note on my scorecard is the “modifier” column. This is a parking lot category that allows the committee member to modify the scores of each team by assessing things like “quality win”, “momentum”, “bad loss”, and “quality loss” if they so choose.

Now, let’s fast forward to the end of the season. For the purposes of this illustration, I made the following assumptions about how the season concludes:

• Washington wins out
• Oklahoma wins the Big 12 but loses to OKST along the way
• TCU loses to Oklahoma along that same path
• Clemson wins the ACC
• Notre Dame wins out
• Georgia loses the SEC championship to Alabama
• Some slight adjustments to offensive, defensive and special team scoring
• Some tweaks to the “modifiers” based on how the season likely turns out (for example, no “quality wins” for Wisconsin)

Here is how using the exact same scoring method turns out once you apply those assumptions.

UW is left out of the playoff in this particular scenario even though they hold the PAC 12 championship . The reason is that both the Big 12 and ACC were able to get their more highly rated champs in ahead of the Huskies (a scenario that I dreaded in my last column). But UW does finish ahead of Notre Dame on the strength of the most heavily weighted attribute in the formula - the conference championship.

This is just an illustration of something that I think more closely resembles the selection process than the food fight that many casual fans by default assume happens every week. But I’m not in the business of hard core statistical modeling.

FiveThirtyEight.com is in that business. In fact, they’ve created their own modeling method that is meant to mirror (to the extent they’ve been able to) the method the committee might be using. They’ve successfully predicted 11 of 12 historical playoff teams using this method. That kind of success could only be possible if there were in fact something more than random debate behind the madness of playoff seeding.

Here is what FiveThirtyEight says about its own modeling.

The goal of a statistical model, however, is to represent events in a formal, mathematical way, and ideally, you’d like to be able to do that with a few relatively simple mathematical functions. Simpler is usually better when it comes to model-building....

We discovered in 2014, for example — when the committee excluded TCU from the playoff despite the team holding the No. 3 spot in the committee’s penultimate rankings — that it isn’t always consistent from week to week. Instead, it can partly re-evaluate the evidence as it goes. For example, if the committee has an 8-0 team ranked behind a 7-1 team, there’s a reasonable chance that the 8-0 team will leapfrog the other in the next set of rankings even if both teams win their next game in equally impressive fashion. That’s because the committee defaults toward looking mostly at wins and losses among power conference teams while putting some emphasis on strength of schedule and less on margin of victory or “game control.” ...

We’ve added other wrinkles over the years. Before the 2015 season, for example, we added a bonus for teams that win their conference championships, since the committee explicitly says that it accounts for conference championships in its rankings (although exactly how much it weights them is difficult to say)....

I spent some time this week evaluating our model’s published forecasts from 2014 to 2016 and found that they were pretty well-calibrated. That is to say, teams that are given a 60 percent chance of making the playoff will actually make the playoff about six out of 10 times and fail to do so about four out of 10 times over the long run.

The point of this exercise wasn’t necessarily to paint a picture that supports the argument that UW is going to the playoffs. That is by no means a certainty. My own example shows one very plausible scenario where UW doesn’t make it in.

However, I did want to make the argument that the selection process probably isn’t nearly as random and chaotic as many might think. There is rhyme to the reason on how the committee tries to control for bias. Even better, there is enough evidence in circulation that allows for people smarter than me to predict the committee’s decisions.

With that, I’m still projecting UW into the playoffs. And, yes, I’m biased.