The Royal Statistical Society ("RSS") has criticised the ASA's existing rules on substantiation and named leading brands as some of the biggest culprits for making statistical claims about their products based on relatively small sample sizes. 

The RSS says advertisers should be wary of focusing their campaign around statistical claims without the goods to back them up and has drawn attention to the following key areas: 

1. Sample size - the greater the sample size the higher the level of confidence consumers can have in the cited statistic. According to the RSS, any sample sizes of less than 100 may be merely due to chance and should be avoided.  In our view, it depends on how many data points are collected from the sample. It is possible for a smaller sample to produce statistically significant advice.

2. Sample suitability - avoid only questioning people that are likely to look more favourably on the brand and consider using a representative sample to ensure the statistic is not misleading. A example of good practice was the sample used by Costa Coffee to justify its comparative advertising against Starbucks with the statistical claim that "7 out of 10 coffee lovers prefer Costa". This was endorsed by the ASA, with Costa having limited their sampling to self-identified "coffee lovers", and conducting their sampling at several sites nationwide, equidistant from the nearest Costa and Starbucks.

3. Stretched evidence - do not overstate the results of any surveys if you are citing a statistic make it as specific as possible. Costa qualified the above claim in the accompanying text to specify that this comparison related to cappuccinos, which was found to be acceptable by the ASA.  

4. Substantiation - hold documentary evidence to prove any objective claims being made. The RSS has criticised the ASA's lack of explanation as to what will be considered "adequate substantiation".  But surely this has to be decided by the ASA on a case by case basis? It would be an extremely difficult task and introduce an element of inflexibility to the Codes to set out minimum thresholds for substantiation. And when it comes to certain sectors, such as health and beauty, the ASA does give guidance about what it will consider to be adequate. 

To be honest, we're struggling to see the RSS's problem here. We can find a number of examples of complaints that have been upheld by the ASA on the grounds of dodgy statistics and few where having conducted their investigation, the ASA has cleared the advertiser of any wrong doing. 

Lord Lipsey (chair of the All Party Parliamentary Group on Statistics and former council member of the ASA) has challenged a number of advertisements for substantiation over the years, the majority of which have been upheld.  

In January the ASA upheld a complaint by Lord Lipsey regarding an eHarmony advert which claimed to have a "scientifically proven matching system". One of the only examples of an advert which has been challenged by Lord Lipsey that has not been upheld is a Bulmers advert that stated "2/3 of drinkers prefer the taste of Bulmers Original to Magners Original". In small text underneath this claim the following was cited "Source: Cardinal. 65.8% in a head to head blind taste test, surveyed in Nottingham and London June 2017. Excludes those who expressed no preference. Sample size: 146 regular apple cider drinkers". After taking expert advice on the methodology, the ASA considered the sample size and methodology was adequate to substantiate the claim. 

Advertisers should continue to be responsible when using statistics, but the RSS also needs to better understand the practicalities of gathering evidence to support a claim.