Mistaking statistical modelling for science

Marketing isn’t the only discipline to have been seduced by the idea that modelling can somehow bypass the hard work of developing empirical laws.  Few seem to realise how heroic the assumption is that teasing out a few weak correlations can quantify precisely how much [something of interest eg sales] will change in the future when [various other things, eg media choices] are altered.

Added to this is the ‘Big Data fallacy’ that adding together bunches of weak correlations will lead to more and more accurate predictions – “once we have enough data, a clever programmer, and a powerful enough computer, we’ll be able to predict everything we want”.  It’s as if chaos theory taught us nothing at all.

The basic work of science is making empirical observations, looking for patterns, and then…. once you have found one, looking to see where it holds and where it doesn’t.  This requires lots of replications/extensions over different conditions (eg countries, product categories, and so on).  This is how scientific laws are developed, that give us the ability to make predictions.  These replications/extensions also tell us what conditions don’t affect the law, and maybe some that do.  This leads to deep understanding of how the world works.  Experiments can be used to tease out the causal directions and magnitude, what really affects the pattern and how much.  Again these experiments need to be done carefully, across a range of conditions that might matter.

Yes, this doesn’t sound very glamorous, it takes much time and effort (1% inspiration, 99% perspiration).  Sometimes we get lucky, but generally many many studies are required.  By independent teams, using creatively different approaches – so we can be sure that the empirical phenomenon really does generalise, that it isn’t a fragile result (or a mistake) that only exists in one team’s laboratory.

Unsurprisingly the idea that a computer model could bypass much of this hard work is seductively attractive.

Terribly complicated, yet naive, modelling seems to be everywhere.  In population health statistical correlations deliver precise estimates that if people eat particular foods (or amounts of fat/sugar/alcohol, or sitting around) then their risk of dying early will be such and such.  There is nothing wrong with this, so long as we recognise the weakness of the method.  Unfortunately these correlations often get handed over to engineers who, with a spreadsheet and a few heroic assumptions about causality, produce model predictions that if the government taxed this, or regulated that, then x million lives would be saved, and x $billion saved in hospital bills.  These predictions need to be treated with a high degree of skepticism.  We need tests before legislation is changed and money spent.

In climate science, a rather new, and until recently very small discipline, modellers now seem to dominate.  In the 1970s a short period of cooling led to worry about global cooling, but then temperatures turned around to rising again, and climate scientists started to become seriously concerned about the role of rising CO2 levels.  They rushed to develop models and in the early 1990s they gave their predictions for CO2 emissions to lift global temperature, along with accompanying predictions of oceans rising, ice retreating, polar bears disappearing and so on.  25 years later they are confronted by the overwhelming predictive failures of these models, that is, the models substantially over-predicted the warming that was supposed to occur (given that the CO2 levels have risen – the IPCC, even though they are ‘marking their own homework’, admit this in their assessment).  The modellers are now starting the work to figure out why.  Meanwhile the forecasting scientists who criticised the climate scientists’ forecasting methods, and predicted this result, have been vindicated.

Models that show wonderful fit to historic data routinely fail in their predictions*.  That’s why we revere scientific laws (and the theories built on them) because they have made predictions that have come to pass, over and over.


* *See also Dawes, J. G. 2004. ‘Price changes and defection levels in a subscription-type market: can an estimation model really predict defection levels?‘ The Journal of Services Marketing, 18:1, 35-44.


8 thoughts on “Mistaking statistical modelling for science

  1. Hello Dr. Sharp

    Where do you land on inductive quantitative models, rather than deductive statistical models. I think that statistical models are good for doing measurements that serve as the basis for developing theoretical models that attempt to replicate facts or laws, as you promote.

    Specifically, DSGE models in economics, with all their shortcomings, pose a discipline to researchers that I find quite appealing. Econometrics models, on the other hand, are subjects to the assumptions made by the researcher aside from those of the model itself (data handling, definitions etc.).

    By the way, is there such an attempt to generalize marketing phenomena into some kind of tractable quantitative model?

    Best wishes,

  2. The modelling approach you seem to be referring to relates I think to linear-based thinking and models.

    Have you applied and explored your findings and approaches in dynamic “systems”based models?

    The approach here is to develop computer simulations based on behavioural rules and causal assumptions and relationships and see what patterns are produced. Also Agent based modelling could be relevant here.

    Put the rules and relationships in the model and see what data is produced…rather than simply analysing the data that will hide the effects of feedback in complex systems.

    Surprisingly little work is done using this approach in marketing….

    We seem in marketing stuck in Newtonian world view of short term linear cause and effect….and digital qualification is making things worse if we think brands are developed over time.

    I wonder what your theories would look like viewed through a systems lens.

    • The modelling you refer to taught us about the limits to prediction. So my comments above are just as valid for this class of models. Happy to be shown I’m wrong, all you need show is the many examples of successful predictions. There should be hundreds/thousands. Novel predictions to vastly different circumstances than the original data. I’m not aware of any.

  3. Are you saying prediction accuracy is the most important value of a model?

    How do we measure/judge the value of a prediction?

    How does time horizon affect the ability to predict…and the “scale” of analysis of the system.

    I’m thinking seeing common patterns of behaviour across different market contexts is also a valuable indicator that underneath there is a common structure driving the pattern.

    A systems perspective helps do this and is lacking in much marketing thinking.

    I wonder what we would find if it was applied to your work. Maybe an understanding of how similar structure relationship of market elements creates the patterns of behaviour in brands.

Please comment on this article if you wish

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s