by Stuart Dennon
Variety, in the guise of variation or variability, is a fundamental concept in statistics – and in all of science and data science. In my time doing data science in academia, finance, marketing and advertising, I have formed the view that variability is perhaps the most fundamental concept in science, business, and perhaps every other area of human endeavour.
Yet I rarely see any mention of variability when reading articles and op-ed pieces on topics such as ‘What it takes to be a data scientist’ or ‘The next big thing in AdTech or Martech’.
In the absence of variability, everything – marketing included – would be easy. Since things don’t change, predicting the future is easy. If widget sales were 1000 units last month, then, absent variability, sales of widgets next month will be 1000, and 1000 the month after that. Expected sales of widgets for the whole of next year will be 12,000 units.
In real life, we don’t see 1000 sales each and every month. Instead, we see 800 units one month, and 1200 in another month, then 975, 2275, 1813 and so on. Total sales next year might well amount to 12,000, averaging 1000 per month, but no two months will likely have the same number of sales.
This kind of variability leads to uncertainty. In business, all decisions are made under conditions of uncertainty, and decision-making under uncertainty is a field of study in its own right. Most of us know it by its more common name: statistics.
Japanese manufacturers have excelled at harnessing and exploiting variation for their own gain. Measuring variability in the dimensions of cogs destined for the gearbox of a Mazda 3 has given the Japanese a seemingly unassailable competitive advantage as manufacturers of high-quality vehicles.
Just as no two months will have the same widget sales, no two cogs will have exactly the same dimensions. As machinery on the production line wears and tears with use, the dimensions of cogs vary increasingly until they reach a point where each and every cog produced is unfit for purpose. Put these poor quality cogs into a gearbox and sell that car to a consumer, and it is a guaranteed source of future warranty claims, product recalls and reputation damage.
Japanese manufacturers measure the variability of outputs to determine when the production line starts turning out lower quality items with dimensions that differ significantly from the ideal. Any worker who measures statistically significant variations can stop the production line and initiate maintenance to return the machinery to a fit and proper state.
Variation in statistics is like oxygen in biology – essential. Without oxygen there can be no life, and without variation there can be no statistics. Variation serves as the backdrop against which we measure statistical significance. And statistical significance is how we make decisions under uncertainty. That makes variation essential in business.
With the dawn of the age of Artificial Intelligence (AI), the importance of measuring and understanding variation can only grow. In the AI era, statistical significance will become the single most important ingredient in the training of algorithms to outperform their human counterparts.