Model risk management

The ever expanding use of models across financial services to aid business decisions and risk management has prompted supervisory concerns around the extent to which firms appreciate and manage the possibilities that their models do not work as expected. These concerns span familiar areas, such as bank credit risk and insurance solvency modelling, but supervisors are also eyeing emerging and less well understood modelling challenges around climate risk and the use of AI/ML.

The PRA’s proposed principles on MRM for banks represent a significant elevation of the bar, and when finalised later this year will demand a significant programme of work to catalogue, categorise and risk-assess the models they use, and to improve governance and oversight processes. The PRA has said it expects an “Initial self-assessment” along with “prepared remediation plans” for them to comply with within a 12-month period.

MRM currently only applies to banks. However, the PRA intends to make MRM applicable to insurers once the Solvency II reforms are finalised. In the meantime, the PRA has indicated in its supervisory priorities for 2023 that insurers should consider how the MRM principles could be applied.

The PRA’s basic concern remains that many banks are not monitoring or managing effectively the aggregate risks they face due to the broad range of models they use. The overarching aim is to ensure that senior management and Boards have clear sight of the aggregate risk that models represent and receive reporting to enable them to be confident that the risk is being managed.

The main challenge for firms will stem from the PRA’s proposal for a very broad definition of what constitutes a “model”, covering any instance where a qualitative or quantitative input is subjected to a transformation that produces a qualitative or quantitative output. This captures traditional credit and market risk models, but also extends to a wide range of algorithms, estimators, heuristics, decision trees and spreadsheet-based calculators that banks may not currently define as models. Banks with regulatory approvals to use models for credit and market risk capital calculations may have hundreds of such models; beyond these categories, large banks may count their “models” under the PRA’s definition in the thousands.

The first stage of implementing the principles – subject to any changes the PRA makes to the definition of a model - will require banks to create an inventory of models and undertake a risk-classification exercise for all in-scope models. This will be challenging, not least because the technology capabilities required to support the expanded model inventories may go beyond those of existing systems. Furthermore, processes that may appear innocuous (for instance, a spreadsheet that collates inputs from three source systems and adds the figures together to form part of a finance/risk reconciliation adjustment for the monthly management accounts) may now count as a “model”. The creation of the inventory will need to be iterative to deliver the appropriate scope.

Banks already have in place model oversight and governance processes for models subject to regulatory approval, but the order of magnitude increase in the number of models within the scope of the proposed principles implies a significant resource stretch for teams that are in some cases already struggling with existing supervisory modelling requirements, including on internal ratings based (IRB) repair, hybrid mortgage models, and preparation for Basel 3.1 implementation.

Firms will need to take advantage of the risk sensitivity and proportionality built into the PRA’s proposals, to ensure that lower risk and less material models are subject to lighter touch oversight. Furthermore, not every component of a MRM process requires qualified, experienced statisticians, and firms should seek to identify aspects of their workflows that are amenable to automated solutions. The overall process may be time consuming, but it should bring transparency to decision-making and present opportunities for firms to rationalise what may be unnecessarily diverse and fragmented models across all areas of their operations.

Firms should look to develop remediation plans on the back of their implementation programmes, identifying opportunities to improve the consistency of model inputs and outputs by consolidating data sources and amalgamating models.

There is no EU equivalent to the PRA’s proposed principles, although EU banks face challenges of their own, continuing to deal with the findings of the Targeted Review of Internal Models (TRIM) exercise,¹ the resolution of which is expected to increase risk weighted assets in affected firms by 12%, or €275 billion. The 2021 TRIM report indicated the need for extensive remediation work also. While EU banks have made improvements, this remains work in progress, with firms also having to face a similar set of economic, credit, IRB repair and Capital Requirements Directive 6/Capital Requirements Regulation 3 pressures on model resources as those listed above.

Meanwhile, the European Banking Authority (EBA) and European Central Bank (ECB) are turning their attention to the robustness of the models underlying International Financial Reporting Standards (IFRS) 9 impairments and Interest Rate Risk in the Banking Book (IRRBB), given changes in economic conditions and in particular the reversal of a decade of low interest rates. One of the EBA’s areas of concern is whether IFRS 9 and IRRBB models built during a low interest rate environment remain robust and reliable in the current higher interest rate environment.

Supervisory attention on the incorporation of climate factors into risk modelling is also increasing. From a model risk perspective, two issues stand out. First, the challenge of building and validating models that can meet expected standards of accuracy and reliability given incomplete climate data that may rely on interpolation, extrapolation or proxies (see more on sustainability data). Second, the potential for over-reliance on third-party solutions for elements of climate modelling, particularly in climate stress testing.

Supervisors want to ensure that firms are not dependent on black boxes and that any bias in models is understood and managed, and we expect supervisors to press firms to improve the incorporation of climate risk into their risk and stress test modelling frameworks in 2023.

External auditors also increasingly expect firms to demonstrate they have undertaken additional modelling to show that the impairment allowances held under IFRS 9 are appropriate in the context of financial risks from climate change. This additional layer of modelling, particularly if significant management judgement is incorporated, is an area where robust internal oversight and challenge should be applied. See the climate risk and the climate-nature nexus chapter for more on climate risk.

Model risks also arise from the incorporation of AI/ML into models across all sectors. Supervisors understand the potentially significant benefits of using AI/ML in risk assessment and management, but want to be convinced that boards and senior management understand the strengths and weaknesses that AI/ML bring to models. No matter the level of analytical sophistication AI/ML may offer, supervisors in the EU and UK expect people, and not models, to be ultimately responsible for making decisions.³ Firms making significant use of AI/ML modelling techniques are likely to need dedicated technical teams to undertake work around AI/ML model risk and validation.

Model Oversight

• Ensure that once the expanded model universe has been identified, the Board and senior executives understand and are able to explain to supervisors the level of the firm’s reliance on models, and their models’ strengths and weaknesses.
• Ensure that all options for taking a risk-based approach to classification and oversight of models are fully understood, and that where possible, model oversight processes take advantage of technology options to reduce reliance on scarce validation resources.

Data and AI/ML

• Actively seek opportunities to expand sources of data, and improve the quality of data so risk models are as robust as possible, and ensure any weaknesses are understood by the Board and senior executives. Industry initiatives, particularly on climate data, may provide avenues to accelerate these efforts.
• No matter the level of analytical sophistication AI/ML may offer, supervisors in the EU and UK expect people, and not models, to be ultimately responsible for making decisions.

Supervisory review of models

• Ensure that where supervisory review of models is required, this is flagged as early as possible and that models are fully ready for supervisory review in accordance with agreed schedules.

¹ The ECB’s TRIM exercise looked at consistency of risk modelling across 65 EU banks, involved 200 on-site investigations, and resulted in some 5,800 findings

² Speech by Anil Kashyap, Member of the Financial Policy Committee

³ EBA, Discussion paper on machine learning for IRB models, 11 November 2021 & Bank of England, CP6/22- Model risk management principles for banks, 21 June 2022

⁴ BoE/FCA, Machine Learning in Financial Services, October 2022