Beware the Seductiveness of Models

Executive Summary:

I’m sorry, I don’t mean super models; I mean mathematical and decision models.  We all use models to make our business decisions or to predict performances or outcomes.  We must beware that we don’t forget the limitations of models and rely on them to do our thinking for us.  To do so invites disaster.

The Rest of the Story:

Models are the basis of decision-making and prediction.  Even when we rely on our intuition for our decisions, we are using models constructed by our subconscious minds.  (I’ll come back to this thought later.)  In engineering, Six Sigma process improvement, or business many of us have become skilled in developing empirical or statistical models to help us understand cause and effect, predict outcomes, and make important decisions.

The collection of data to develop models is a very powerful means of gaining insight and can empower us to make much wiser decisions.  Such is the power of the Six Sigma methodology.  I am a big advocate of doing so.  However, we must always be wary of the limitations of our models.  When we allow our models to make decisions for us, like policy, or allow models to do our thinking for us, we can be surprised and disappointed, even harmed.

There are several causes for the 2008 recession triggered by the collapse of the investment and loan industry.  One cause that has been repeatedly identified is the use of certain loan-risk models.  These models attempted to predict human investment behavior.  They also became the standard calculations for determining whether a loan should be awarded.

In most cases, the users of the models didn’t understand the assumptions or limitations of the models; they just plugged numbers into a spreadsheet or Web page and received a risk result output.  For all I know, some users may have received a simple yes/no result or the equivalent.  This is the seduction of models.

We build them to help us make better decisions.  Unfortunately, we often forget that we built them to help us understand performance, behavior, and cause and effect, not to substitute rational judgment.   Models provide us with tendencies and probabilities.  They don’t provide us with the truth.  A model is not a crystal ball.

A friend of mine found the following quote and uses it repeatedly in his training regarding statistical modeling of data.  I have since taken its meaning to heart.  Sir David Cox, co-developer of the Box-Cox power transform, said, “All models are wrong; some models are useful.”  If you get nothing else from this post, meditate on that quote.  It can be a life, or business, saver.

Let’s examine why Sir Cox would say such a thing.  First, models are based on data.  This makes them powerful because they are, generally, based on facts.  It also means that they are based on incomplete information.  It’s rare that we collect every possible data point.  It’s generally not practical or possible.

Next, as we begin trying to build our models we often manipulate things.  In order to get a model that can explain the data, we often make a conscious decision to remove outliers from our data set.  This makes our models much more reliable, but we have a human tendency to forget that the outliers were there, and therefore, they can very well show up again.  But, our models wont predict it.

We also try to simplify our models.  I’ve built models from designed experiments or historical data that involved dozens of factors or coefficients (possible causes) and potential interactions.   In the end, in order to have a useful, practical model, we eliminate the “insignificant” factors and simplify the model.  After all, if we can eliminate 80% of the math and inputs necessary and only lose 3% of the model’s accuracy, doesn’t that seem sensible?

Lastly, models are based on historical or experimental data.  That means that our models explain what has happened in the past.  As soon as we use those models to predict the future, we have knowingly stepped out of the inference space of the model.  Therefore, all models are wrong.

I’ve already stated that I am an advocate of using models.  Obviously having something that accurately predicts the outcome 80% or 90% of the time is very useful.  Yes, some models are useful. 

Models, especially continuously updated models, of stable production processes, chemical reactions, or material use can be very accurate and reliable.  Even so, they don’t account for the impact of bored veteran operators of the process or overly enthusiastic new operators.  They don’t account for the human influence over time.

Models that try to predict human behavior, such as investment risk models, are not so accurate.  If a model of human behavior can explain 60% of the variation in the data set, it’s a very strong model.  Most of us wouldn’t think about making predictions or controlling production processes from a model that explains less than 90% of the variation in the output.  We must consider what we are modeling as part of our judgment call.

Similarly, the further away in time we progress from when the model was created, unless we continuously update the model, the more likely that our model will no longer represent the behavior of the process.  Environments change over time.  Operators change or get bored, other inputs or factors can come into play, or contamination begins to generate noise.  This is true for production processes and especially true for human behavior.  Human behavior can change just because the World Series of Baseball is taking place.

So how do we protect ourselves from being seduced into letting our models misguide us?  I have four recommendations.  Perhaps others will comment and add to this list.

1. Recall that all models are wrong; along with decisions, include a risk mitigation plan
2. As mentioned, continuously update and reassess models with real-time data, or as near as possible
3. Make an effort to observe the process yourself; take note of changes in behavior or environment
4. Always give your decisions and predictions the common-sense test

The first two hints, I think, are self-explanatory.  Let’s discuss the second two in a little more detail.

As I mentioned, in time, things change.  Don’t forget that metrics, intentionally or unintentionally, drive behavior.  As soon as we start collecting data on an operation, people will start trying to find ways to make that data look better.  Something is bound to change, either in the inputs that affect the data, or in the way the data is collected and recorded.

Also, over time, people’s behavior can change, or other changes within the environment, such as relocation of equipment, seasonal temperatures and climate control, or changes in volume can affect the process and how well the model fits its performance.  By observing or taking note of these, we can be alarmed to the possibility that our model might not steer us as accurately as it once did.

The common sense or intuition test is often touted, but just as frequently underestimated.  We can’t explain why our common sense or intuition might disagree with the recommendation of an empirical model and, therefore, we discredit the one we can’t support.  Don’t do this.

If your intuition disagrees with the model’s recommendation, give some thought as to why this might be.  Have you observed something changing?  Are you anticipating an outcome or a change that the model would have no way of predicting?  Have you have seen it before?  The best way to reconcile your intuition and your empirical model is to examine the model’s assumptions and make sure they still make sense, and to refer to suggestion 3 above and look for something that might have changed, which your subconscious has noted, but your empirical model has not.

Do not discredit your intuition.  Professor Gerd Gigerenzer, director of the Center of Adaptive Behavior and Cognition at the Max Planck Institute for Human Development in Berlin, Germany (or he was at the time my resource was written¹) has studied the way our intuition works and helps to make decisions.  Our intuition builds models from our experiences in much the same way that we build empirical models.  Intuitive models can be very simple or very complicated.  They can also be very reliable.

If your gut is telling you something different from your empirical model, it very well might mean that you have observed something that could interfere with the empirical model, you just might not be able to put your finger on it right away.  It’s worth an investigation.

Use models to make decisions and predict outcomes.  They are very powerful.  Don’t be seduced into false confidence or into forgetting common sense or the value of human judgment.  Models are there to help us understand so we can make informed or wise decisions.  They are not a substitute for human judgment there to do our thinking for us.

Stay wise, friends.

1.  Gigerenzer, Gerd.  Gut Feelings: The Intelligence of the Unconscious.  New York, New York:  Penguin Group (USA) Inc., 2007