Adoption of advanced machine learning techniques for IRB Models

Key takeaways

• ML techniques can be used to enhance the rank order of IRB models via exploratory analysis, as an aid to traditional techniques, and offer a powerful tool to explore the range of potential modelling choices.
• Industry feedback showed that most of the banks intend to use ML techniques in risk differentiation and within that mostly for probability of default (PD) modelling focused on credit decisioning. Apart from PD model development, some banks are using ML for model validation and collateral valuation.
• The EBA have set out a principle-based approach which IRB models developed using ML techniques should adhere to with the recommendations being consistent with already existing IRB and MRM frameworks.
• Management teams should conduct a review of their framework and consider how and when ML techniques may need to be used to inform credit risk model development and/or model validation.

The key points from the follow-up report on the use of ML in the context of IRB are highlighted below.

Selective use of ML for IRB

The potential challenges and benefits of using ML techniques were evaluated for the different steps of IRB modelling, namely risk differentiation, risk quantification and validation. Industry feedback showed that most of the banks intend to use ML techniques in risk differentiation and within that mostly for probability of default (PD) modelling. The PD models developed would be used to enhance credit risk decisioning, with the potential to extend into IRB and capital management, subject to concerns regarding regulatory approval. Apart from PD model development, some banks are using ML for model validation and collateral valuation. Most respondents mentioned that the specific skills and technical knowledge needed to carry out model development and validations when using ML techniques, is not consistently in place.

Our point of view:

• With an exponential increase in data availability and unstructured data, data storing capacity and improvements in computing power over the last couple of years, there is an opportunity to use ML models in analysing and transforming large and unstructured data. For example Natural Language Processing (NLP) techniques along with ML could be used to extract information from unstructured text-related data, which could add newer features in a quantitative module or help automate answers for a qualitative module of the IRB model.
• Our experience has been that ML techniques can be used to enhance the rank order of IRB models via exploratory analysis, as an aid to traditional techniques (see Zen Risk (deloitte.com)). Further, from a validation perspective, ML models offer a powerful tool to explore the range of potential modelling choices (and associated performance) i.e. the art of the possible. More technical insights on this can be found in these articles on the application of machine learning and challenger models in IRB Credit Risk modelling.

Complexity of ML techniques

The DP highlighted the key challenges when developing and validating IRB models using ML techniques. These include (i) statistical issues, (ii) skill-related issues, and (iii) interpretation / explainability issues.

1. In relation to the statistical issues, the DP described the challenges of dealing with overfitting, the difficulty to assess the representativeness of data and how to fulfil operational data requirements with regards to the inputs of the models.
2. The DP highlighted that the complexity of ML techniques may increase the time, computational resources, and human resources to develop and validate IRB models. It may make the assessment of modelling assumptions more challenging.
3. Another challenge that banks face relates to the explainability and interpretability of the results of the ML models. Therefore, with an increase in complexity, banks will require specific skills and technical knowledge to develop, document and validate ML models.

Figure 1.1 below shows that the most commonly used interpretability tools being Shapley values (40% of respondents), followed by graphical tools (20%), enhanced reporting and documentation of the model methodology (28%) and sensitivity analysis (8%).

Figure 1: Measures to ensure explainability of ML techniques

Source: EBA feedback from CP on DP on ML for IRB models, 14 respondents.

Our point of view:

1. Overfitting may become a challenge for ML models, especially in case of Decision Tree models with highly skewed data distributions. Overfitting in such cases could potentially get addressed by using the Gini’s Diversity Index while selecting the split criteria.
2. The implementation of ML techniques along with challenges of explainability and interpretability may put a strain on the already stretched teams in banks, especially if they do not have the right skillset. It is important to upskill and reskill resources to obtain the required skillset for model development and validation using ML techniques.

Interaction with regulatory frameworks

When incorporating ML techniques in credit risk modelling, the decision should not only be based on prudential regulations, but also reflect ethical and legal aspects, including consumer and data protection requirements (i.e., consider the use of ML techniques relative to the General Data Protection Regulation (GDPR) and Artificial Intelligence (AI) Act requirements). EBA’s report clarifies the alignment of these frameworks with the regulatory framework and highlights concerns about legal uncertainties under the AI Act.

Our point of view:

• In terms of the Prudential Regulation Authority’s (PRA) position on ML techniques, the PRA has so far maintained a technique-neutral approach. However, it did flag various considerations as a part of its final policy on MRM (see our blog From principle to practice model risk management takes effect).
• For Nordic banks looking for new and innovative ways to further enhance automation of credit risk management, the Model Risk Management (MRM) recommendations specified in PS6/23 and SS1/23 can be taken into consideration, focusing on the feedback for MRM in AI/ML models. We expect a follow-up / response to this consultation towards the end of this year. While it will not focus solely on MRM, it may give further insights/guidance on the PRA’s position and thinking, including any potential additional policy actions.

Principle-based recommendations

The EBA have set out a principle-based approach which IRB models developed using ML techniques should adhere to. The principles are intended to make clear how to adhere to the regulatory requirements set out in the Capital Requirements Regulation (CRR) for IRB models. The recommendations apply where ML models are used for risk differentiation and risk quantification purposes. In line with the regulatory expectations for model development, the key principles that the EBA have highlighted are as follows:

1. All the relevant stakeholders should have an appropriate level of knowledge of the model’s function.
2. Institutions are recommended to avoid unnecessary complexity in the modelling approach.
3. Banks need to ensure that the model is correctly interpreted and understood.
4. When human judgment is applied, a good level of understanding of the model is required.
5. For complex ML models with limited explainability or for frequently updated models, a reliable validation is important and might require depth and/or frequency.
6. For data preparation purposes, banks should ensure that there are clear rules and documentation.

Our point of view:

These principles are consistent with the IRB and MRM frameworks and are of a form that should help management teams in banks to inform changes to model development and model validation frameworks, which are consistent with these principles.

Way forward

The principle-based recommendations provide banks with a view on the use of ML techniques and how it complies with regulatory IRB expectations. However, the fast-paced developments in ML techniques require that authorities continuously monitor the implementation of these techniques. The EBA plans to monitor the developments in the ML field and may amend the principle-based recommendations, if required.

Our point of view:

Management teams should conduct a review of their framework and consider how and when ML techniques may need to be used to inform credit risk model development and/or model validation. It may not be straight-forward to adopt advanced ML techniques in IRB models given the complexities involved relating to interpretability and explainability of these techniques, along with skills around model development and conducting robust independent validation of IRB models.