My younger brother is a big college football fan, far more than I. Today, he spotted an argument made by some ESPN football analysts about my beloved BYU Cougars that I thought would be a good springboard for talking about another fundamental principle of analysis — Make Your Assumptions Explicit.
First, watch this. My apologies for that and what follows if you're not a football fan.
A bit of context: the analysts were discussing BYU's chances of getting into the new NCAA College Football Playoff (CFP) tournament which replaces the deeply flawed (and deeply corrupt) Bowl Championship System that, for 15 years, picked who got to play for the NCAA college football championship. Here's the kicker — there's no mathematical formula for picking who gets invited to play in the CFP. Instead, a panel of 13 experts will choose the teams.
Now, after watching that 2.5-minute clip, my brother's immediate reaction to the analysts' comments was:
"...BYU can't go undefeated because the teams it has left are too tough. But if BYU does go undefeated, its schedule was weak. ???"
To understand his complaint, you have to understand the concept of strength of schedule. The idea is that beating a strong team should count for more than beating a weak team. Okay, fair enough. The guy who beats Goliath should get more glory than the guy who stomped on a weakling. If BYU only plays weak teams and goes undefeated, should it get into the tournament over a team that played some strong teams and lost a few games? Arguably not.
Here's the problem — strength of schedule can't be settled until the season is finished. At season's end, the 13 experts will have the benefit of reviewing the full season records of all NCAA teams to evaluate . But those ESPN analysts? Despite only being three weeks into the season, despite BYU being so-far undefeated, with a 41–7 butt-whoopin' inflicted on the Texas Longhorns, a perennial #25 team, they've already concluded that BYU can't get into the tournament. "So sorry. Strength of schedule."
On to the analytical lesson.
Sooner or later, everyone gets asked which possible future outcome seems most likely in a given situation; but the world is so amazingly complex, with so many different events occurring every single second, that it's impossible to be aware of them all, much less figure out how to factor them in properly to any prediction. Any and every prediction of a future outcome is based on wildly incomplete information.
For our football example, none of those analysts predictions is worth anything because no one can know how hard a team's schedule really was until the season ends and they can see the final tally of all NCAA games played and see who lost to who by how much. The evidence doesn't exist yet. Sporting events are exercises in chaos theory and any one random event can destroy the delicate foundation on which any given prediction rests. One-hard-tackle-and-a-compound-fracture later and a team's Heisman candidate is in traction, with the analysts' pick-to-win-it-all collectively praying it can go 4–4 for the rest of the season.
So, lacking hard facts, we make assumptions. We assume, consciously or not, that X will stay constant and therefore Y will occur...or that X will not stay constant and therefore Z will occur. Those ESPN analysts are assuming that strong teams are going to win the games they're supposed to win and no key players will end up in the hospital.
Making assumptions isn't a bad thing to do. In fact, we couldn't function on a daily basis if we didn't do it. We can't prepare for the future if we can't depend on some degree of predictability. But some things are so random or complex (or both) that all we can do is flat-out guess that things won't change or will at least change in at a constant rate and act on that guess.
So how do we handle assumptions? By doing the following:
1. Identify your key assumptions and declare them up front. Key assumptions are the ones so critical to your argument that if they fail, your argument collapses. Not being aware of your assumptions is a recipe for surprise. Not being aware of your key assumptions is a recipe for disaster. Being aware of them but not making your audience aware of them is a recipe for watching your credibility get gutted when your analysis doesn't pan out. So tell your audience. As Colin Powell liked to say, "Tell me what you know. Tell me what you don't know. Then tell me what you think. Always distinguish which is which." Your audience will usually forgive an "I don't know" answer. An audience will be less forgiving of an "I know" answer that proves false. They'll feel misled.
2. Review your key assumptions regularly. You have to revisit your assumptions regularly to see if they're still valid. The world changes. Yesterday's solid assumption might fall apart today. Good assumptions become bad assumptions.
So if the ESPN talking heads really wanted to be "analytically correct," the best they could say is, "If there are no major upsets or injuries for the remainder of the season and our current assumptions about how all NCAA teams will finish in the standings hold up, BYU's strength of schedule would not compare favorably enough with other likely candidates to secure a tournament berth."
But I'm sure that wouldn't be nearly as entertaining as yelling "BYU HAS NO CHANCE! NO CHANCE AT ALL!"