Posted: 20 Jan. 2020 15 min. read

How can statistical sampling help my business?

Statistical sample testing is a way of testing that a business process is producing the right outcomes without needing to check every case.

Examples of business processes that benefit from statistical sample testing include quality assurance testing on customer phone calls, checking the quality of advice provided to clients, internal audit testing of controls put in place to mitigate the risk of errors occurring in the business, and cross-testing outputs of different systems.

Essentially statistical sampling is the process of randomly selecting a sufficient number of data points from a large homogeneous group, reviewing these samples, and making inference about the entire population from the sample. 

It enables a business to make inferences such as, “I am 99% sure that my sales team call quality requirements are not breached in more than 95% of customer phone calls.

In a world of data, connectivity and increasing consumer expectations, the need to generate timely insights from a large volume of data and information is growing. Sampling enables these insights in a cost-effective way. 

However, statistical sampling can be difficult for businesses to use because judgement and a robust process that includes statistical rigor are necessary. 

Unfortunately, there are many examples of misapplying statistical sample testing, meaning that the results of the tests cannot be relied on. The most common example of this is not sampling from a group that is homogeneous. Another common example is where the tester either ignores or does not properly control for errors observed in the sample. 

Sampling correctly
 

An organisation that seeks to demonstrate whether an issue exists (e.g. whether their customers were receiving services which they paid for) should ask a series of questions before deciding to use a statistical sampling approach including:

  • What is the specific question you are trying to answer?
  • Do you need to segment the population into multiple homogenous groups?
  • Are there any groups that are not suited to sampling (e.g. because there is an expectation as to the result and action can be taken on that basis)?
  • What are the confidence levels required for the relevant stakeholders?

In this example actuaries combine expertise in statistics with commercial business acumen to use statistical sampling in order to provide a specific level of confidence of client advice quality, within a reasonable time and cost.

This can be done in two parts. By dividing the population into homogeneous groups, and a robust sampling process for each of these groups. This would enable an insurer to advise the regulator that it is ”99% sure that my clients were provided with quality advice in more than 95% of cases” for each group that is sample tested. 

Homogenous Groups
 

The key requirement before statistical sampling can be done, is that the population being sampled is homogeneous. Determining homogeneity requires judgement, and there are four key criteria for assessing this:

  1. The group should be mutually exclusive (i.e. it should not be possible to be in a group other than the one it is in) and collectively exhaustive (i.e. the sum of all groups adds up to the total population size). 
  2. The individual elements within the group should have common characteristics.
  3. The group should be capable of a definition so that it is not easily misinterpreted as to what the group is, or represents.
  4. There should be data that can be measured to support that the group is homogeneous.

A simple example of a homogeneous group that may satisfy these criteria is that the same procedures, processes and controls were expected to have been applied in the group being tested.

Robust Sampling Process
 

Once a homogenous group has been established, sampling from that group must be random and sufficient in order to allow statistical descriptions to be made post the sampling work. 

The statistical sampling process is iterative. Before sampling, the sample size is determined based on the population size, and desired level of confidence and error rate. During sampling, if any errors are observed in the sample, the  sample size may need to increase, subject to any systemic errors observed. 

Deloitte has built a statistical sampling tool to simplify this process. The tool is in an Excel format with strict user controls in place and is designed to be user friendly with comments and clear definitions throughout to help understand each step of the process. 

The tool also includes a heat map that allows the user to quickly assess the level of accuracy of the sample given the number of errors observed during the sampling process. Finally, the tool provides the probability of observing no further errors from an additional increase in sample size. 

Contacts to find out more
 

You can contact us with your sampling query at our dedicated email inbox sampling@deloitte.com.au. Our Sample Size Tool is also available for purchase under specific terms and conditions – email or call to find out more.

More about the author

Mathew Ayoub

Mathew Ayoub

Director, Consulting in Asia Pacific

Mathew is a qualified actuary (FIAA) and Certified Enterprise Risk Actuary (CERA). He is passionate about increasing the level of understanding of financial products in everyday Australians to improve long term value for all stakeholders.  Mathew has worked in roles across most aspects the insurance value chain from capital and reinsurance to product, pricing, claims, strategy and risk management, spending a significant proportion of his career in general insurance and accident compensation. Mathew also worked at APRA advising and liaising on the interpretation and expectation of prudential standards, as well as driving policy recommendations.