|
By Chris Murphy
Hard, objective, accurate, definitive, precise numbers are beguilingly appealing. Yet in reality numbers are often subjective, vague, provisional and need to be qualified - they can even be downright misleading. So a few key danger zones are highlighted here and some elementary precautions suggested. They may seem 'self-evident'. However, a vast number of examples could be quoted to show that simple errors often trip up even sophisticated research.
Data coverage and treatment
Many statistical series exclude certain items, are based on sample surveys rather than comprehensive direct data and incorporate adjustments to cope with raw data deficiencies (such as non-replies to questionnaires) or background conditions (like seasonal adjustments).
Geographical coverage can also be problematic. For instance, some National Statistics datasets cover the whole United Kingdom (Great Britain plus Northern Ireland), others are for Great Britain and some are just for England and Wales. The creation of a Scottish government has been accompanied by an increasing divergence between what statistics are collected north of the border and their comparability with those for England and Wales.
It is also easy to confuse statistics confined to the European Union with those covering the narrower euro zone or the wider European Economic Area or, broadest of all, the continent of Europe as a whole.
Action point: Users of statistics must undertake the tedious but essential chore of reading footnotes and explanatory material to understand their extent, definitions adopted, sources of the data reproduced and ways in which it has been adjusted.
Which units are being used?
The units of measurement used can be a major pitfall in making calculations and taking decisions based upon them. Are the units Imperial ones (miles, pounds, gallons and so forth) or SI (metric) measures like kilometres, tonnes and litres? The following examples highlight problems caused by confusion about units of measurement:
- Astonishing though it may sound, the loss of the US Mars Climate Orbiter space probe in 1999 was largely due to one design team working in Imperial units and another in metric. The failure to translate the measurements correctly ended up costing the American taxpayer some $125 million.
- To add to the scope for confusion, the US Imperial Gallon is smaller than the British version. This led to a visiting private pilot who did not realise this having to make an emergency landing when he ran out of fuel.
- Once we start dealing with less familiar systems of measurement, the risk increases. There are at least six ways of calculating the 'tonnage' of a vessel and several of these are not based, as one might expect, on its actual weight.
Action point: To handle such difficulties I recommend the "Economist Desk Companion: How to Measure, Convert, Calculate and Define Practically Anything", which costs about £4 on Amazon or is available in public reference libraries.
Even when we are confident of the underlying measurement system employed, we still have to be careful. It is common practice to simplify long runs of figures and make them easier to understand and communicate by rounding them up or down to a shorter set of digits. 177 may signify the absolute value of 'one hundred and seventy seven', but it can often mean 177,000 or even 177 million.
Action point: Always check to see if the underlying numbers have been simplified.
Comparing like for like
Another potential statistical elephant trap is combining data from sources that do not compile it on the same basis. Failing to do so can have a dramatic impact.
In one case, using non-comparable data actually affected national economic policy. A UK government white paper released in early 1976 stated that public expenditure took over 60% of national income and was alarmingly high against other industrialised economies. In fact, when the data was later recalculated on a mutually comparable basis the figure was reduced to a far less frightening 46%. Unfortunately, the damage was already done. Sterling slid on foreign exchange markets, the UK had to beg for a loan from the International Monetary Fund and accept humiliating and painful budgetary cuts as a condition for receiving a bailout that was subsequently shown to have been unnecessary.
Such episodes have encouraged the development of 'comparing lemons with lemons' - and not oranges or bananas - in the statistics produced by international statistical offices like the United Nations, OECD or European Union and used in the databases and publications of national statistical services.
Action point: Users of statistics always have to confirm that the data is comparable before employing it. They also must be aware that much quantitative data is estimated rather than absolutely accurate and thus prone to a certain 'plus or minus' margin of error.
Corporate financial statements, which are often treated as wholly objective documents, are actually a mixture of facts, subjective judgements and regulatory conventions. For instance, since 2005, every company quoted on a European Union stock exchange been obliged to produce accounts that conform to International Financial Reporting Standards (IFRS) and these have also been adopted in some non-European countries. It is thus possible to make legitimate comparisons between Australian, British, French, German or Indian listed companies. However, this:
- usually is not the case for pre-2005 data;
- does not apply to the far more numerous privately-held companies;
- is not valid for trying to make comparisons with companies based in major economies such as the United States, Canada and Japan. They use their own national accounting standards (although there are plans to harmonise the American one with IFRS) and very different figures for key items like profits or assets will be generated than under the IFRS rules.
Action point: If source data is not guaranteed as comparable, it either has to be adjusted by the researcher to make it so (which may not be feasible) or they should be forthright in admitting its limitations and warn against placing too much reliance upon it.
And finally - a feeling for numbers
1. Reduce data entry errors
- Do a rough mental calculation beforehand - does the worked-out result surprise? It might then be additionally interesting - or just plain wrong.
- Perform the calculation in reverse - you should be left with zero. If not, another check is needed.
- Use Excel's 'Audit' tool to see what cells your result includes - it is easy to miss some when arriving at a final total.
2. Follow calculation best practice - do not simplify numbers and then do workings out with them - perform calculations with full precision first and only simplify the results at the end.
3. Apply source criticism
- What is the status of the source?
- Beware of bias - which particular interests or lines of argument may they be trying to advance?
- Is the data internally consistent? If not, the material is suspect.
- Do other sources confirm or refute it?
4. Never just depend on summary numbers - glance over the raw data. That 'average' can mask huge deviations from the typical; or two similar rates of growth may result from either smooth, steady gains or jerky, erratic steps forward.
5. Forget the prestige of apparently exact numbers - are they believable? Always remember it is the researcher's contextual background knowledge that turns a set of otherwise meaningless numbers into useable data.
By Chris Murphy
Chris Murphy is a director of business consultancy Ravensbourne Research Ltd
<ravensbourneresearch@tiscali.co.uk> and a Fellow of the Royal Statistical
Society. After working for the Department of Trade & Industry, he took a
post as a researcher at the London Business School. Chris was subsequently
head of research with a firm of conference producers before taking up his
present post in 1989. Since then he has been involved in a wide range of
research, advisory and investment activities and is one of the editorial
team on Jordan Publishing's flagship encyclopaedia "International Corporate
Procedures". His own book, "Competitive Intelligence: Gathering, Analysing
and Using It", was published by Gower in 2005.
FUMSI articles by Chris Murphy »
![[Get Copyright Permissions]](http://license.icopyright.net/images/icopy-w.gif "[Get Copyright Permissions]")
Click here for copyright permissions!
Copyright 2010 Free Pint Limited
Related articles:
You may also be interested in:
|
