Will Computer Software Assurance (CSA) improve product quality and patient safety?

It has been 20 years since the FDA issued their General Principles of Software Validation¹, and while technology and computerised systems have advanced greatly in that time, the life sciences industry has been slow to amend Computer System Validation principles in line with advancements.

For decades, life sciences companies have focussed on documenting and validating everything (“if it is not documented, it didn’t happen”) with a view to avoiding audit observations and warning letters. This is both time consuming and costly and the focus is more of a regulatory “checkbox” exercise than patient safety.

Patient safety and product quality should be the priority of any validation effort. In 2011, the FDA’s ‘Case for Quality’ initiative set out to reset this focus and identified that manufacturers who invested in quality and managed the associated product quality risks had fewer complaints and greater long term payoffs than their competitors. Innovation and the adoption of new technologies to allow a more flexible and less burdensome approach are key to ensuring patient safety and product quality always remain at the forefront.

So will the CSA guidance for industry² shift the focus from documentation to critical thinking and more efficient testing to deliver higher quality?

Let's recap on Computer System Validation (CSV)

Computer System Validation (CSV) is the process of testing, validating, qualifying a regulated computerised system to ensure that it does exactly what it is designed to do in a consistent and reproducible manner. CSV guidelines to industry were published by FDA in 2003 and since then there have been no changes in CSV regulations. Recently, there has been a shift in the technology landscape with the advent of new technologies such as – Internet of Things (IoT), Artificial Intelligence (AI), Machine Learning, cloud computing and automation. CSV practices or framework do not meet the pace with which the technology has evolved and as a result, there is a perception in industry that CSV is a road block and there has been a reluctance to adopt these latest technologies.

Perceived CSV paint points

There is a belief within industry that CSV is an activity which is burdensome, heavy on documentation and has lesser scope for technology adoptions. The following are some pain points which are identified by industry with the current CSV approach:

• While the life sciences industry is keen to adopt the latest technologies with a view to realising the potential of these technologies and the impact they have on productivity and efficiency, CSV is considered a barrier to adopting these technologies.
• Software development activities should integrate with a risk-based approach for designing an efficient CSV framework. Although industry has adopted the risk-based approach, there are still understanding and implementation gaps and thus, the risk-based approach is not being implemented efficiently.
• It is estimated that nearly 80% of the defects in testing are test script errors and 20% are related to software errors. Time and effort spent resolving test script efforts does not add value to the quality of the software.
• Gathering excessive objective evidence during testing just to satisfy auditors results in a significant increase in the validation time and effort.

These pain points have paved the way for the CSA guidelines and served as drivers for CSA implementation.

Key aspects of CSA

Critical thinking and adopting a risk-based approach over unnecessary documentation are key aspects of CSA.

Critical thinking:

This is the core element of CSA and places it on top of all the activities. Critical thinking techniques should be utilised to analyse and perform the assurance (validation) activities. Initial software risk assessments should be conducted to identify if the software has a direct or indirect impact to product quality, patient safety and data integrity, and this assessment should drive the assurance activities.

The rigor of assurance activities (testing and documentation) should be high for direct impact software and lowered for indirect impact software.

Risk based approach:

Risk assessment:
CSA recommends for a robust and streamlined risk assessment approach to calculate the risk of a functionality within the software and this risk should drive all the assurance activities. Risk assessment should focus mainly on the ‘intended use’ of the requirement and the risk of this requirement failure on patient safety, product quality and data integrity. Identifying the existing controls and proposing the mitigation controls should be done meticulously within the ambit of the requirement. Critical thinking should be combined with risk-based approach to ascertain the appropriate risk.

Risk based testing:
The main goal of CSA is to shift focus from more documentation (scripting) to more testing of software. The intention of CSA is to bring in the mindset of “document less, test more”. Critical thinking and risk assessment of a functionality should be considered to determine the amount of testing that is required. Industry should follow “focused testing” where-in only those functionalities that pose high risk should be tested rigorously.

Critical Thinking + Risk Assessment = Testing Method Selection

The following testing approaches³ may be adopted to drive efficiencies in testing computer software with a focus on its quality. Each type of test method should be followed based on the risk associated with the requirement:

Scripted testing - is similar to the traditional testing where a detailed step-by-step instruction will be given in a test script along with the navigational details.

Limited scripted testing – similar to scripted testing but the difference is, if the risk of the functionality is high and requires only positive testing, then this method can be used.

Unscripted testing - this does not require a detailed step-by-step test script. A simple test script should exist to capture the test objective and results of the testing performed by the tester.

Ad-hoc testing - similar to unscripted testing where there aren’t any procedures or test scripts in place. The tester would be given a free hand to test the entire system with the main aim to break the system.

Exploratory testing - a type of unscripted testing with an intention to find defects by performing an unusual activity, unanticipated user activity or an activity that might be done only accidentally. Thus, this testing is often described as simultaneous learning, test design, and execution.

Automation testing - automation testing is a type of software testing where a manual testing activity is mimicked using an automation script/tool. Automation testing should be used excessively as it increases testing efficiency and reduces time/effort.

Will CSA address the CSV pain points?

Like all guidance documents, effective implementation of the CSA guidelines will be the key to delivering the benefits for industry. Almost certainly, switching the focus from documentation to critical thinking, correct implementation of risk assessment, risk-based testing and adoption of new technologies should allow for an improved quality product, earlier identification of problems and faster delivery of compliant and higher quality software. While time will tell, it is looking hopeful that CSA will be a step change in how systems validation is approached - a step change which should ultimately positively impact patient outcomes.

References

1. General Principles of Software Validation; Final Guidance for Industry and FDA Staff, FDA, 2002
2. Computer Software Assurance for Production and Quality System Software - Draft Guidance for Industry and Food and Drug Administration Staff, FDA, 2022
3. GAMP 5 A Risk Based Approach To Compliant GxP Computerised Systems, Second Edition, ISPE, 2022