Unleashing the power of machine learning models in banking through explainable artificial intelligence (XAI)

The explainability vs. model performance trade-off

Explainability has taken on more urgency within banks, a result of the growing complexity in AI algorithms, which is happening at a breakneck pace thanks to greater computing power, the explosion of big data, and advances in modeling techniques such as neural networks and deep learning.

Several banks are establishing special task forces to spearhead explainability initiatives in coordination with their AI teams and business units. The use of automated machine learning (AutoML) solutions from vendors has also increased considerably.

Concurrently, there is a growing focus on the trade-off between model accuracy and interpretability (figure 2).¹⁶ XAI can help model developers weigh these trade-offs more tangibly and advise on how they should begin bridging the gap between complexity and intuitiveness.

Today, there is a whole spectrum of models ranging from decision trees to deep neural networks (DNNs). On the one hand, simpler models may be more interpretable, but they often have less predictive power and accuracy, especially compared to more complex models. Many of these complex algorithms have become critical to the deployment of advanced AI applications in banking, such as facial or voice recognition, securities trading, and cybersecurity.

Other trends are prompting banks to prioritize XAI as well. For example, many banks are looking to adopt more off-the-shelf AutoML solutions offered by vendors, cloud providers, and software companies. These prepackaged solutions have varying degrees of explainability, and it may take months to analyze and document how they work. The push for agile development is also raising new concerns over explainability, since faster processes can make it more difficult to install appropriate guardrails and controls in early stages of model design.

Regulators’ perspectives on XAI

Most financial regulators do not mandate where and how banks can use black boxes, but some, such as Germany’s Federal Financial Supervisory Authority, have advised institutions to weigh the benefits of choosing a more complex model, and document why they decided against more interpretable options.¹⁷ In addition, financial watchdogs have recommended that banks run traditional models alongside sophisticated machine learning models, and assign analysts, or a “human in the loop,” to address major discrepancies that arise between the two.¹⁸

Engaging with regulators will be important for banks to continue developing advanced AI applications, since oversight groups are increasing their scrutiny of machine learning in every corner of the globe. Bank regulators in North America,¹⁹ for example, have solicited feedback on banks’ explainability practices, and the degree to which their limitations can impact model risk management. Recently, five US agencies formally requested information on how banks manage AI risks, including when a “lack of explainability” raises uncertainty about the soundness and reliability of their machine learning models.²⁰ In addition, the watchdog for consumer protection in financial services expanded its policing of discriminatory practices to include the ways in which banks use models and algorithms to advertise and sell products and services.²¹ Meanwhile, some policymakers in the European Union²² and Asia²³ have passed regulations that allow customers to request an explanation for any decision generated by AI and learn how their personal data was used to determine it.

Many regulators are taking a pragmatic approach, relaying that there is no “one size fits all” formula to assess these trade-offs. Instead, they suggest that banks should weigh the purpose of the model, the environment in which it will be deployed, and the goal of explainability. Some have indicated that it may be acceptable for banks to use opaque models to test theories, anticipate liquidity needs, or identify trading opportunities, so long as they use more interpretable models when acting on predictions.²⁴

Other instances where explainability may not be a priority is in the application of optical character recognition (OCR) systems that extract information from scanned documents, or natural language processing technologies that wade through contracts and legal agreements.²⁵ Similarly, banks may not need to seek a high degree of explainability for algorithms that yield accurate outcomes when identifying fraudulent transactions.²⁶

A playbook for implementing XAI

Implementing XAI more broadly across the enterprise is a multifaceted and multistep process, requiring potential changes to data sources, model development, interface with various stakeholders, governance processes, and engagement with third-party vendors. However, this may be easier said than done, since there are no commonly accepted practices to delineate how much explainability is needed for different machine learning applications, and which techniques should be pursued in light of those considerations. Nevertheless, there are several goals that should be central to banks’ implementation of XAI:

• XAI should facilitate an understanding of which variables or feature interactions impacted model predictions, and the steps a model has taken to reach a decision.
• Explanations should provide information on a model’s strengths and weaknesses, as well as how it might behave in the future.²⁷
• Users should be able to understand explanations—they should be intuitive and presented according to the simplicity, technical knowledge, and vocabulary of the target audience.²⁸
• In addition to insights on model behavior, XAI processes should shed light on the ways in which outcomes will be used by an organization.²⁹

Establishing XAI as a formal discipline can put banks on the fast track to achieving these objectives. This will likely mean introducing new policies and methods, from the premodeling stages to postdeployment monitoring and evaluation. It will also require every stakeholder who contributes to AI model development to act purposely and intentionally with each decision they make. For example, developers should apply explainability principles to their choice of training and input data for model prototypes. Instead of focusing solely on datasets that will maximize performance, they should also consider whether the input or training data may perpetuate hidden bias (e.g., historical lending data may favor certain demographics that had easier access to credit), whether the data contains customers’ personal information, and if it spans a long enough timeframe to capture rare or unusual events.

Model development teams should also conduct a preliminary assessment of model performance and interpretability, to get a sense of how accurate the model will be compared to simpler and more traditional analysis methods. This deliberation should begin in the premodeling stage, so designers can tailor machine learning architecture to the target explanation. In some cases, banks may want the model to be transparent to all users, and will prioritize an interpretable design (“glass box,” or “ante-hoc explainability”³⁰). In others, they may build a complex model, and either apply XAI techniques to the trained model (post-hoc explainability) or create a surrogate model that emulates its behavior with easier-to-follow reasoning.

Either way, banks should assess which techniques and tools are most helpful in advancing explainability. There are several factors that should drive these decisions, leading with the target stakeholder: regulator, risk manager or auditor, business practitioner, or customer (figure 3). For example, underwriting officers can be served well by counterfactual explanations, which show the degree to which different aspects of a customer’s application should be tweaked to change the outcome (e.g., increase income by a certain amount to gain loan approval).³¹ Other bank employees may need “an explanation of an explanation,”³² or visualizations that map out patterns and flag anomalies in the data, such as groups of individuals that may be inappropriately segmented for marketing campaigns. There are also varying levels of explainability that should be taken into account (see sidebar, “Varying levels of explainability”).

Designing XAI governance

Banks should not view XAI as independent of AI, but instead work to embed explainability into existing AI governance structures and processes. Figure 4 offers an example of how banks can infuse XAI considerations into three lines of defense and control activities for AI.

Then, to fully embed explainability into the AI production life cycle, banks should establish a cross-functional task force comprising specialists from the executive committee, businesses, IT, risk and compliance, legal, ethics, and AI research. The US military’s Defense Advanced Research Projects Agency (DARPA) Explainable AI program discovered that some of the most effective teams have cross-disciplinary skill sets and expertise, such as computer scientists who work with experimental psychologists and user experience and design specialists.³³

The XAI committee should have routine informed discussions about which models and AI solutions should be prioritized for the application of explainability techniques, using criteria that are most important to the bank. They should also decide which scenarios have serious implications for the algorithm to be used as is.³⁴ For example, a model designed to target media platforms that shows an ad to unintended target audience members 10% of the time may be more acceptable than a recommendation engine that advises clients on misdirected products 10% of the time.

This task force can begin with smaller projects within business units, gauging how effectively the lines of communication among the three lines of defense identify and address explainability issues. It can then scale processes that work across other businesses. It will be important for banks to infuse explainability into cultural norms and practices as well, from annual employee training to evaluating the ROI of new prototypes, and projecting multiyear returns on those projects. Over time, explainability should become a key part of the “ethics fluency” that personnel from the boardroom to interns must retain, until everyone in the business understands XAI principles, can ask the right questions, and elevate issues to appropriate supervisors.³⁵

For an added layer of oversight, many banks are also collaborating with academics who can serve as an independent third party that reviews explainability concerns and approves of model use within specific functions. In addition, more companies in financial services and other industries are appointing a chief AI ethics officer to serve as the lead on AI explainability, transparency, and trustworthiness.³⁶ In the absence of this role, banks should name a senior executive—ideally the chief financial officer or chief risk officer—who will be held accountable for decisions made on behalf of AI projects.

Auditors and risk managers may not need to track each model’s individual path to making a prediction as much as they need to see aggregate levels of model risk. These supervisors can benefit from scoring systems that provide a visual indication of XAI metrics related to functionality, performance, and usefulness.³⁷ Deloitte’s Lucid[ML], for example, can display three dimensions of explainability alongside results from a simpler AI technique and a surrogate “white box” model.³⁸ These dashboards can provide a high-level perspective of the explainability risks, as well as a deeper analysis of interpretability constraints (see sidebar, “Assessing risks with explainability dashboards”).

The XAI ecosystem is quickly expanding, and vendors that tailor services to the banking industry range from large software companies to more nascent tech startups. These providers run the gamut from “plug and play” solutions to tools and frameworks that can be fully integrated into commonly used cloud computing platforms. Banks can apply these XAI processes to functions including fraud identification, loss and churn prediction, and debt collection.

XAI techniques: Making choices

There are several approaches and techniques banks can take to activate explainability (figure 5). Some are already commonplace in the XAI practice, while others are still being refined. Banks can still quantify risk levels associated with model transparency—and assess how much explainability is needed, or whether alternative modeling options may be available to them.

The limits of explainability

Despite recent advancements in XAI research, banks still face many technical challenges implementing explainability into the AI pipeline. For example, many XAI techniques that are best suited for credit risk models tend to accommodate binary decisions made on behalf of consumers, such as “lend” or “don’t lend.” These explanations may not consider the full range of modifications that can be made to interest rates, repayment schedules, and credit limits, and overlook consumers’ preferences for different terms of a loan.⁴⁰

In addition, banks may want to provide counterfactual explanations that show customers why their loan applicants were denied, and coach them on how to improve their financial standing. However, because such advice may or may not be appropriate for the specific borrower’s context, caution should be exercised in accounting for how explanations may be interpreted by the public. Banks should also consider that explanations may not retain validity over time due to changes in external conditions or instances when the model must be retrained to improve performance. As a result, organizations must be aware of the actions they recommend to customers and communicate the time periods in which the results will be most applicable, in tandem with other caveats.

XAI teams can also be limited by resource constraints since it frequently takes longer to compute explainability techniques as models grow in complexity and precision. In addition, some institutions have expressed concern that explainability may allow competitors to reverse engineer their machine learning models, thereby revealing the “secret sauce” behind their proprietary algorithms. They have also highlighted the risk that XAI could make it easier for outsiders to “trick” their models or launch adversarial attacks that cause them to malfunction.

One of the biggest challenges also lies with talent acquisition. XAI specialists will likely remain sparse, given the dearth of candidates with a background in XAI, a field that’s even more niche than machine learning and data science. Banks can overcome this challenge by hiring talent outside their organization or recruiting engineering and computer science graduates straight out of school and training them on explainable AI internally.

What more should banks do to keep pace with emerging developments in XAI?

Many banks are exploring algorithms that are complex and inherently interpretable, such as explainable gradient boosting and neural networks. They should continue researching new techniques for developing deep learning applications that are transparent by design, and do not require post-hoc explainability.

They should also consider partnerships with think tanks, universities and research institutions, and groups such as Singapore’s Veritas Consortium, which brings together large financial institutions from around the world to develop common guidelines for the industry to follow.⁴¹ In addition, it’s important for banks to be active participants in conferences and workshops that cover emerging XAI topics, and collaborate on research that can drive the field and its practical applications forward. They can also push vendors to continue making prepackaged models more explainable, so they can more easily adopt third-party solutions.

Recently, momentum has also been observed for the idea of advisability, which incorporates feedback from users to improve the functioning of AI models that consume and produce explanations in collaboration with humans.⁴² In addition, explainability is expected to be fundamental to producing “third-wave AI systems” that allow machines to contextualize their outside environment, and bring that knowledge into XAI models.⁴³ Eventually, it may be possible for AI developers to create complex systems such as deep learning networks that are inherently interpretable by design.

It will be paramount for banks to share their experiences with regulators, and work with government agencies and other oversight bodies to produce guidelines that enable AI research and development but also protect customers’ interests. By taking extra steps to be ethical and transparent—while continuing to spearhead research and leading practices in XAI—banks can solicit regulatory trust, and find the ideal balance between tech innovation and safety.