Limited functionality available
Statistical sample testing is a way of testing that a business process is producing the right outcomes without needing to check every case.
Examples of business processes that benefit from statistical sample testing include quality assurance testing on customer phone calls, checking the quality of advice provided to clients, internal audit testing of controls put in place to mitigate the risk of errors occurring in the business, and cross-testing outputs of different systems.
Essentially statistical sampling is the process of randomly selecting a sufficient number of data points from a large homogeneous group, reviewing these samples, and making inference about the entire population from the sample.
It enables a business to make inferences such as, “I am 99% sure that my sales team call quality requirements are not breached in more than 95% of customer phone calls.”
In a world of data, connectivity and increasing consumer expectations, the need to generate timely insights from a large volume of data and information is growing. Sampling enables these insights in a cost-effective way.
However, statistical sampling can be difficult for businesses to use because judgement and a robust process that includes statistical rigor are necessary.
Unfortunately, there are many examples of misapplying statistical sample testing, meaning that the results of the tests cannot be relied on. The most common example of this is not sampling from a group that is homogeneous. Another common example is where the tester either ignores or does not properly control for errors observed in the sample.
An organisation that seeks to demonstrate whether an issue exists (e.g. whether their customers were receiving services which they paid for) should ask a series of questions before deciding to use a statistical sampling approach including:
In this example actuaries combine expertise in statistics with commercial business acumen to use statistical sampling in order to provide a specific level of confidence of client advice quality, within a reasonable time and cost.
This can be done in two parts. By dividing the population into homogeneous groups, and a robust sampling process for each of these groups. This would enable an insurer to advise the regulator that it is ”99% sure that my clients were provided with quality advice in more than 95% of cases” for each group that is sample tested.
The key requirement before statistical sampling can be done, is that the population being sampled is homogeneous. Determining homogeneity requires judgement, and there are four key criteria for assessing this:
A simple example of a homogeneous group that may satisfy these criteria is that the same procedures, processes and controls were expected to have been applied in the group being tested.
Once a homogenous group has been established, sampling from that group must be random and sufficient in order to allow statistical descriptions to be made post the sampling work.
The statistical sampling process is iterative. Before sampling, the sample size is determined based on the population size, and desired level of confidence and error rate. During sampling, if any errors are observed in the sample, the sample size may need to increase, subject to any systemic errors observed.
Deloitte has built a statistical sampling tool to simplify this process. The tool is in an Excel format with strict user controls in place and is designed to be user friendly with comments and clear definitions throughout to help understand each step of the process.
The tool also includes a heat map that allows the user to quickly assess the level of accuracy of the sample given the number of errors observed during the sampling process. Finally, the tool provides the probability of observing no further errors from an additional increase in sample size.
You can contact us with your sampling query at our dedicated email inbox email@example.com. Our Sample Size Tool is also available for purchase under specific terms and conditions – email or call to find out more.
Mathew is a qualified actuary (FIAA) and Certified Enterprise Risk Actuary (CERA). He is passionate about increasing the level of understanding of financial products in everyday Australians to improve long term value for all stakeholders. Mathew has worked in roles across most aspects the insurance value chain from capital and reinsurance to product, pricing, claims, strategy and risk management, spending a significant proportion of his career in general insurance and accident compensation. Mathew also worked at APRA advising and liaising on the interpretation and expectation of prudential standards, as well as driving policy recommendations.