Darin makes little attempt to keep us awake:
Let’s talk about evaluating a team, and why it’s so hard even after the season is played. To simplify things, let’s imagine that the season comprises a bunch of games the Huskies will definitely win, a few they will definitely lose, and then a few make-or-break games to go either way. I’m going to focus on Stanford, Arizona State, UCLA, and Oregon State as those games, and ignore the rest for the time being. In August, you ask yourself, How likely is it the Huskies will win those competitive games? Well, I don’t know yet how good they are, so I’m highly uncertain. There’s a small chance they’ll be worse than all four of those teams, and a small chance they’ll be better; but basically I can’t tell. I estimate their probability of being the better team in these games this way.
This is called my “prior probability distribution.” Prior because it comes before I have any data. It’s my subjective best estimate of how good the Dawgs are. I put the peak of the curve at 50 percent because I think they’re basically as good as the second-tier teams, but the curve is wide because I’m highly uncertain about whether they’ll be good or bad. I’m agnostic, but by implication I’m predicting 9-3 (losses to Oregon and two of Stanford, ASU, UCLA, and OSU) as the most likely outcome.
Then I start getting data – the Dawgs play Stanford and lose. Now I have to update my probability distribution a bit, based on new information. The chance that the Dawgs are 100 percent (that they’ll win every game) is now zero. But things don’t change too much. There’s a technique called Bayesian inference that defines the math I use, but hopefully the result is intuitive. I now have an updated distribution that looks like this.
The curve hasn’t changed too much, but I now think it’s more likely the Dawgs are really bad and less likely they are really good than I did before. I still think “medium” is a good estimate, though. This red curve represents the rational, rigorous estimate you should have, given that the Dawgs lost a game you thought they were probably 50/50 in. (This procedure is very similar to what Jeff Sagarin uses in his ratings.) Then we play Arizona State, and lose, and I do the same thing again.
We see a further shift to the left – I am starting to have a sneaking feeling that the Dawgs are more likely bad than good. Is that about how you felt after the ASU game? Then comes UCLA, and we update once more.
This is pretty much how I felt on Sunday. The Dawgs are probably a bad-ish team, compared to the second tier of the Pac-12. Before we go on, let’s see what would have happened if you had a different viewpoint in August. What if you were an optimist In August? What if you thought the best guess is that the Dawgs are at 75 percent – i.e., they “should” win three of the four competitive games? This would be something like a 10-2 season. In that scenario, your prior distribution would have looked like this.
The peak is at 75 percent, but the curve is still wide, accounting for the chance they’ll be very good or very bad. Your changes over time would have looked like this.
Same basic story: your initial, uncertain estimate of the goodness of the Dawgs is distorted downward by disappointing outcomes. You may also have been a pessimist, expecting only one win out of the four make-or-break games. In that case, you’d have done this.
Given what’s happened, how much difference should it make whether you were optimistic or pessimistic at the outset of the season? Not much, as it turns out. Below are what should be your current best-estimates of how good the Dawgs are in their competitive games.
As you can see, they’re reasonably similar. Everybody agrees that the Dawgs are probably more bad than good. The difference is just that the pessimist is more sure they’re bad. He’s had his initial skepticism confirmed by three losses. The other two still hold out some hope that the Dawgs have just been unlucky. Now comes Oregon State. It won’t surprise you to learn that if the Dawgs lose again, we’ll see another downward tick in our assessment.
Just so. But what if the Dawgs win at Oregon State? In that case, we’d want to bump our assessment back up a bit, using the same procedure, but in reverse. Here’s what that will look like.
Wait a second! Now we see three things. First, the chance that the Dawgs are monumentally bad has disappeared. Second, the width of the bell-shaped curve is narrower – we’re starting to become more sure of our assessment because we’ve got more data to work with. Finally, the peak of the curve (our best guess of how good the team is) is just about back where started. I.e., one win means we’ve learned almost nothing since the outset of the season!
This is the fundamental problem with assessing football teams objectively. There’s not much data to work with. We do have other data to make use of: we didn’t just lose to UCLA and ASU, we got killed. That might mean we should adjust our curve downward further. On the other hand, the Dawgs also beat Boise State and Arizona, both of whom are good teams. Those wins should mean we adjust upward some amount. (How much? Beats me.) We also look at recruiting, advanced stats, and whatever.
Brad is Up:
And that, ladies and gentlemen, is the best 1,000 word example of why you don’t actually ask engineers to make decisions. Because unless the data slaps them in the face with it, they can’t.
Look, if your whole point is that there’s no definitive proof one way or another how good this team is (and by extension, how good Sarkisian is), then you could’ve said that in two or three sentences and saved me the time of importing all of your pretty little graphs into this GD post. And I wouldn’t have disagreed.
But let’s get real world (true storaaay) here for a minute. You’re an engineer, but you also own your own business. In order to make money, you make suggestions and recommendations based on incomplete data, assumptions, and best guesses on a daily basis. Multi-million dollar ones. People don’t pay you to sit back and say “I just can’t tell right now,” they pay you to take the information you have available to you, extrapolate it, and come up with a solution that makes the most sense.
You’ve said that you like to use things like money lines when making predictions because they tend to keep you honest. So, in your most honest, engineer/business owner voice, answer me two questions.
#1. Are the Huskies a “good” team this year? The reality is that they pass most of the tests out there, including the eye one. Except when it comes to their record. So, are they genuinely good?
#2. If Steve Sarkisian was your employee, would he still have a job at the end of this season? You like to use probabilities and statistics, but when it comes to your own money, would you accept the mostly middling performance you’ve seen from one of your own employees, when said employee is critical to lining your own pocket?
Before you go there, I get the flaws in the analogy. It’s far from perfect. But at some point, you have to make a decision based on the information you have. Here’s a homework assignment for you – with each passing season with a coach that has failed to be genuinely “good” in previous seasons, is it more or less likely that he’ll be good in subsequent seasons? Extra credit for each graph.
Here’s my whole point: The Huskies have improved each of the last five seasons. They are better today than they’ve been in a decade. But it doesn’t mean jack-freakin’-squat if the teams they play year in and year out are improving at a faster rate. And when it comes to the results on the field, they very arguably are. There’s a cost to changing coaches, for sure. But sitting back and waiting for definitive proof one way or another has a fairly big cost as well.
Darin Tries to Hold it Together:
Hey, now, if you don’t like my assessment you can just say so. No need for nasty name-calling, like “engineer,” and such. It’s just hurtful and nasty and your mother would not approve. The point of all that was mainly to document the thought process I’m using, and I think we all should be using, when it comes to assessing the Dawgs. As you point out, that includes assessing Sarkisian, but he’s not really the focus.
I’m hoping to counter the thinking you hear from some Huskies – one close relative, in fact – who were, understandably, very frustrated after the UCLA game and started to make big conclusions about The State of Affairs. Even that close relative would never say that one game is enough to make big conclusions by itself, so he pointed out a trend, much like you did in your post above. In fact, he made many of the same points you just made, although he made them more colorfully than you did. I’m hoping to make the point that the UCLA game is important in a sense: it’s data, and data is what we need. Unfortunately, it’s not much data, and a win against Oregon State might be enough to undo a lot of the damage done by the other losses. If you’re a rational decision-maker, which I strongly believe we all should strive to be.
I do agree with your basic conclusion: you’ve got to make decisions about whether you’re on the right track or not. You can’t avoid it. And one big decision is whether you like Steve Sarkisian and want to keep him. I’m hoping to make the point that the on-field performance is ambiguous. Anyone who says, “We lost to Stanford, ASU, and UCLA, therefore I conclude that we’re on the wrong track,” is probably over stating his case. So how do you decide?
One way is to rely on the opinion of experts. Here in Seattle, we’re fortunate to have Hugh Millen, who is a thoughtful observer and fan of Husky football. He is on record as stating that the current staff is “doing things right.” In other words, we’d be looking for new coaches who aim to do things the same way the current staff is. How do you argue with that? (Not a rhetorical question.) As an observer I am less knowledgeable, less informed, and no more passionate than someone like Millen. Furthermore, he’s shown the willingness to speak out when he thinks things aren’t being done right. Why not just piggyback on his judgment? If you accept that argument, then you should be frustrated by the slow progress, pissed about the losses, and patient.
Brad Spares You Anymore of This:
I understand the thought process you’re using. But the problem with it is that we could very well end up at this same exact point next year. And the year after that. And the year after that. At some point, you simply have to decide that you have enough data to make a Big Decision. I get that you want all the data that you can get prior to making that Big Decision. It increases the odds you’re making the right choice. But at what point would you personally say you have enough? Next year? The year after?
You know that I like Sarkisian. And I desperately want him to succeed. I don’t think he’ll never be able to win on the road, and I honestly don’t see why he can’t. At some point, the performance on the road simply isn’t ambiguous. You start to think that maybe there’s just something about him – personality, mannerisms, the psychology he uses in the week leading up to a road game, whatever – that instills a lack of confidence, or a propensity to make mistakes, or a tightness in his team that keeps him from leading anywhere close to the same team away from Husky Stadium that you see at home, on a consistent basis. I can’t put my finger on it, but sometimes, even otherwise successful people have a roadblock through which they just can’t break. ¬¬It’d almost be easier to handle if they just had a fatal flaw on the road – offense didn’t work, defense always played badly, etc. But that isn’t really the case. While there are common threads in a lot of the losses on the road, the Huskies seemingly find new ways to beat themselves fairly frequently.
Let’s work under the assumption that Sarkisian is back next season, because he most likely will be. And based on what you’ve said (and I mostly agree), Sarkisian has generally done “good” here. In 2014, the Huskies lose Price, but assuming Bishop Sankey and Austin Sefarian-Jenkins stick around, the offense is pretty loaded. The defense should be very good as well. It’s a team that should be disappointed with “only” 9 wins, and could do even bigger things. But after the 2014 season, 4 offensive linemen graduate. Sefarian-Jenkins, Williams, and Sankey are gone. Danny Shelton, John Timu, and by all accounts, Shaq Thompson are gone. Do you feel comfortable with what Sarkisian has done here to say that his 7th season is going to build off the success of year 6, or was year 6 just a happy coincidence (thinking mostly of the lines, obviously…)?
Like I said, I get your desire for more data, but part of me wonders if you don’t frame the question incorrectly in your own mind when you evaluate this team. It isn’t, “What should Scott Woodward do?”, it’s “What would I do if I was the athletic director?” I bet you think you almost only make cold, calculated decisions based on hard data and the subsequent analysis, but if you really look at what you do on a daily basis, you make at least as many, or maybe even more, decisions based on prior experience, and a trend developing from less than complete data. That’s just the way the real world works. You aren’t afforded the time to sit back and wait, because while changing coaches has a cost, failing to make one has costs as well, if you turn out to be wrong. And the odds overwhelmingly say that any coaching hire is the wrong one.
I’ve got to say, I’m really surprised to see a plea to authority from you of all people. I like Hugh Millen. I think he’s an absolutely tremendous asset for Seattle-area football (both Huskies and Seahawks). But I think it’s far more for his technical expertise than it is his judgment. Part of that is obviously because he makes his opinions very public, and people are more inclined to remember when he’s wrong than when he’s right. But the fact is that his judgment comes from a very refined perspective. And for the most part, I think that Sarkisian fits within it. That certainly doesn’t mean he’s wrong, but it also doesn’t fail to recognize the biases that Millen most definitely has, and has exhibited on more than one occasion.
I still come back to one point. Sarkisian has failed to fix the lines in his time here, and there isn’t much hope that things progress on a consistent trajectory. The passage of time suggests that things will be better next season almost no matter what¬¬, but the failings of one, two, and three seasons ago suggest that it can actually get worse. And that Sarkisian’s success there relies as much on happenstance as it does a system that generates at least a moderate, consistent, improvement.