Article
Trustworthy AI
Artificial intelligence (AI) has become one of the key technologies of this century and plays an increasingly essential role in answering the challenges we face. AI will impact our daily lives across all sectors of the economy. However, to achieve the promise of AI, we must be ready to trust in its results. We need AI models that satisfy a number of criteria and thus earn our trust.
Artificial Intelligence (AI) has long fascinated both computer scientists and the public since the term was coined in the 1950s. Since then, the sensationalist scaremongering about runaway AIs gradually gave way to a grounded, realistic view: AI is a sophisticated technology – or set of technologies – with the potential to deliver significant economic, scientific and societal advantages. It is an immensely powerful tool with wide-ranging potential. Over the next 10 years, experts expect an incremental economic impact of AI worldwide between $12 and $16 billion.
Implemented properly, AI enables us to become leaner & faster, smarter, more personalized. With AI, we can examine and learn from data at a speed and scale that took our predecessors generations. Proper implementation is not automatic – it requires skills, experience and discipline. Open source toolkits have effectively “democratized” software development and led to a rapid proliferation in AI-based tools – from experts and debutants alike. This dynamic introduces both opportunities and risks. For example, AI models can be easily re-trained on new data sets, keeping them relevant and up-to-date.
Proposed "Artificial Intelligence Act"
- Adopts a broad definition of AI
- Focuses on use-cases (vs the technology itself), defending fundamental rights
- Categorizes specific applications as forbidden or high-risk (Annex III)
- Establishes quality standards & disclosure of high-risk AI applications
- Defines requirements for assessment and ongoing assurance of conformity
- Envisions regulatory control mechanisms
- Quantifies penalties for breaches by type
AI is anything but objective
A common difference between AI and "classic" deterministic approaches is that AI learns from data rather than from a set of rules. However, it is a common misperception that having roots in data bestow objectivity on an AI model. In reality, AI is only as objective as its developers design it to be.
Computer vision tasks conveniently illustrate the point. An algorithm is trained on a set of image data that are labeled for concepts, such as "stop sign" – by humans. The Deep Neural Network (DNN) classifies each image, breaks it down into characteristics (e.g. edges, colors, and shapes) and associates the result with the label. The DNN can do this very effectively. Yet even the best architectures fail if misled by training data, such as “stop sign” images given the “no entry” label. The resulting DNN will not be able to distinguish stop signs, either consistently or erratically assigning them to the “no entry” category. The AI is only as good as the human trained it, entirely dependent on the selection of data, its completeness, and consistently correct annotation.
We live in the age of the so-called "narrow" AI, for example: AI which can identify objects or predict the next word in a sentence. These AIs are trained & pruned to excel at one task alone. The data they are fed are curated by their inventors. If trained to recognize cats, they will not recognize penguins as such (rather, that they are “not cats”). “General AI” remains a long way off.
The quality of an AI depends on numerous design decisions.
Beyond the perils of inaccurate or inconsistent annotation of training data, narrow AI also depends on countless design decisions, many of which can have profound impact on the functioning of an AI. An AI model may be trained on and applied to data, however:
- which data?
- with which objective function?
- utilizing which approach?
- which architecture?
- which tunings and tweaking?
Implemented improperly, an AI model can systematically discriminate against what it does not know (what was absent from the training data), inadvertently perpetuating historical bias. It may succeed in classifying images for the wrong reasons (the background of the image vs the subject), a defect which cannot be caught without sufficient transparency. It may be unstable, making a prediction one way, then another, despite similar inputs. These risks are not new: they have accompanied model designers long before the “age of AI”. However, the mathematical nature of AI renders AI models in many ways more susceptible to these risks, or at least in different ways than in the past.
To achieve the promise of AI, stakeholders must be ready to trust in their outputs.
Many of our customers already make use of AI, yet concerns or doubt about reliability remain. To resolve this challenge, Deloitte interviewed data scientists, computer scientists, mathematicians, as well as risk, ethics and economic experts worldwide. The result: Our "Trustworthy AI Framework", encapsulating the six key criteria that AI must fulfill to gain our trust:
(Please click on the graph)
A framework to define what makes AI trustworthy
Trustworthy AI
Artificial intelligence (AI) has become a key digital technology and an essential part of the answers to many of the challenges we are facing. AI will impact our everyday lives as well as all sectors of the economy. But to achieve the promise of AI, we must be ready to trust in its outputs. What we need are trustworthy AI models that satisfy a set of general criteria as described in this article.
Transparent & Explainable
Explaining the inner workings of an AI Model both requires and facilitates true understanding. Transparency inspires trust. Yet highly accurate neural nets are notoriously opaque. How can we have the best of both worlds?
Fair & Impartial
Without proper controls, AI risks perpetuating bias inherent in the data on which it was trained. From flawed decisions to illegal discrimination and reputational damage, consequences are far reaching. How can we inherently design fairness into AI?
Robust & Reliable
Failure modes of models must be consciously designed, their impacts contained. What happens when AI models are confronted with unexpected data (contents, formats)? Do they fail? If so, do they fail in a predictable manner? How sensitive are their predictions to the failure modes?
Preserving Privacy
AI must be designed for high predictive power without relying on private or sensitive data. How can we ensure that only permissible data is used, that privacy is preserved?
Safe & Secure
To realize their potential, AI models must be deployed, while minimizing vulnerability to cyber-attacks or abuse. How do we ensure all attack vectors are sufficiently assessed and resolved?
Responsible & Accountable
Application is as important as underlying technology: just because it can be built does not mean it should be. Are the “subjects” of the AI appropriately informed and in agreement with how they are “processed” by AI?
Trustworthy AI is relevant along the entire life-cycle
One approach to Trustworthy AI could be to leave it to the auditors. „Launch and learn“, „Fail fast“. Agile processes are highly effective in achieving concrete results quickly. However, we should be careful not to use the trendy terms in the wrong context. What works perfectly well for prototype development or A/B testing is likely not sufficient for solid and reliable implementation into a production environment.
As we know it from the manufacturing process: the quality costs increase exponentially the later an error has been identified. This is no different with AI. Failed AI models can not only incur economic costs, but also reputational damage. This can affect all developers – even the tech giants who owe some of their spectacular growth to AI. Yet true disaster cases have so far remained thankfully few. We attribute this to two dynamics:
- The tech giants are investing heavily in Trustworthy AI
- Other companies have little exposure as yet, only now emerging from the proof-of-concept (PoC) phase, when AI models have been shielded from the dangers of the outside world
Tight deadlines, limited budgets and other pressures increase the risk of errors. Clear priorities and a rigorous approach are needed to ensure trustworthy by design: from conceptualization to prototyping, integration to testing, and ultimately monitoring and general governance.
Trustworthy AI is transparent and explainable
Where AI models may excel in accuracy, they leave much to be desired in terms of transparency. Opacity increases with model complexity, non-linear models may well reflect the world they approximate, but are notoriously difficult to explain. Limited interpretability translates to limited acceptance. For AI to be trustworthy, all participants have a right to understand how their data is being used and how the AI system is making decisions. Organizations should be prepared to build algorithms, attributes and correlations open to inspection.
Strategize: Transparent & Explainable
Challenge
An insurance company wants to use AI to make faster, more precise decisions in calculating insurance premiums. Ideally, the AI-enabled calculator will not only deliver more exact outcomes but will also make the process easier to understand for a variety of stakeholders. When insurance premiums go up, most customers want to know the reason why, and – under Article 22 of the Germany’s General Data Protection Regulation (GDPR) – they have every right to. Developers need to fully understand how their AI works in order to improve transparency in these automated decisions. Ultimately, it is important for the AI solution to calculate insurance premiums as accurately as possible – and to do so in a way that is straightforward and easy to understand.
Solution
Our experts use Deloitte’s Stakeholder Assessment tool to identify who or what is involved in every step of the process from development to AI-enabled decision-making. We collect input from all stakeholders (i.e., developers, insurers, prospective insurance customers) and use their interests and preferences to determine what kind of explanations they need – e.g., a global explanation to help people understand the model itself or a local explanation to clarify one specific automated decision – how to present the explanations, and whether to display them as a visualization or with text. The objective is to ensure that the decision-making process is completely transparent, without jeopardizing the accuracy of the AI-enabled system or violating regulations. At the same time, we establish a systematic framework that makes it easier to document the data we collect and use, which is important if we need to retrace the decision-making process.
Outcome
Identifying stakeholders early on in the process enables the insurance company to draft a few different approaches to the AI solution and compare them, ultimately leading to a solution that provides straightforward explanations for all relevant stakeholders as well as premium calculations that are as accurate as possible.
Build: Transparent & Explainable
Challenge
A hospital plans to launch an AI-enabled virtual assistant to help doctors make faster, better decisions and detect disease with greater accuracy. AI-supported MRI imaging, for example, can automatically detect potentially malignant tumors. Regardless of the system’s benefits, doctors as well as patients expect the algorithm to reveal how it arrives at its diagnosis. This is necessary to prevent doctors from recommending ineffective treatments more likely to put patients at risk than to make them better. When it is a matter of life and death, it is essential for the AI system to fully explain its decision-making process.
Solution
Deloitte implements a state-of-the-art AI model that improves diagnostic accuracy and delivers the desired transparency. Thanks to the range of tools in Deloitte’s Lucid [ML], the system can explain the decision drivers at the global or the local level. Key features/drivers of the expected results are visualized in such a way as to convey which regions within the image determine the diagnosis. Lucid [ML] articulates the drivers in the model in a straightforward way, helping doctors understand and validate the process.
Outcome
The hospital now has a cancer detection system using a high-tech neural network that offers accuracy, transparency and traceability. With the ability to detect even more subtle signs of malignant tumors, doctors have a better chance at saving lives; with the user-friendly transparency dashboard, doctors and patients have more confidence in the quality of the diagnosis.
Integrate: Transparent & Explainable
Challenge
An insurance company notices an increase in complaints from certain customer groups. Particularly, customers reporting claims for minivans are consistently – yet inexplicably – automatically rejected. The claimants are exasperated and demand an explanation. Customer support invests time and effort into a more in-depth investigation of these automated decisions in order to provide a response.
Solution
The explainable AI toolset Lucid [ML] developed at Deloitte’s aiStudio enables companies to display the AI system’s decision-making process in an intuitive way suitable for both technical and non-technical staff. By integrating the explainable AI toolset into the code of the insurer’s AI application, customers have a dedicated field in the online user interface that proactively offers insight into the rationale for a claim denial. This feature is tested first internally with customer service staff to ensure that the underlying AI claims adjudicator operates correctly and then incorporated into the customer-facing web portal. Claimants can now see for themselves why their claims were rejected.
Outcome
Adding an explainability element to the insurance company’s portal increases policyholder satisfaction. Greater transparency in the claims process and instant availability of the reasons for claim denials improve the insurer’s credibility. Policyholders now have faster, higher quality information at the click of a button and the company saves time and effort in customer support.
Assure: Transparent & Explainable
Challenge
As part of its loan business, a large German bank seeks to improve its credit risk forecasts and offer customers more personalized services. The AI system developed by the bank for this purpose (based on an IRB approach) uses customer data to calculate and issue credit ratings as well as a total credit limits for customers. The system relies on an underlying AI model that has been trained with historic data and requires an annual risk-based evaluation of all aspects of the IRB systems. As this is a substantial new system (and underlying technology), the bank engages an external firm to conduct an audit.
Solution
Deloitte has long-standing experience in audits such as these. We expose the drivers behind a model using state-of-the-art techniques and tools, such as the aiStudio’s Lucid [ML] to explain the inner workings of the overall model as well as individual decisions in an intuitive manner. Additional measures form an important part of these audits, such as interviewing a sample of customers to determine whether the decisions made by the AI system were clearly articulated. In order to comply with regulatory standards, the audit must also identify which algorithms the system uses and establish whether the results of the model can be fully disclosed.
Outcome
The explanations constitute multiple components including feature importance analyses, surrogate models and counterfactual explanations. Our audit of the bank’s AI system finds some parameters required to articulate the decisions to be missing, meaning it cannot provide full disclosure. In addition, the parameters being used remain opaque; so even though users receive the correct numbers, they cannot properly interpret them. We present our audit findings to the bank, and they initiate steps to correct any potential deficiencies. After these adjustments, we issue a positive audit report and submit it to the financial authority BaFin. The bank can now rest assured that the results of its AI system will satisfy regulatory transparency requirements.
Trustworthy AI must be fair & impartial
A common concern about AI is how to circumvent bias introduced by humans during coding processes. As with any model, the predictive accuracy of models depends greatly on the quality of inputs. With algorithms trained on data, AI models are particularly sensitive. If training data is not representative, AI models will suffer from bias and likely systematically mislead. A further danger is the perpetuation of historical human bias designed into the model. To avoid the harm caused by training bias, companies need to determine what constitutes fairness and actively identify biases within their algorithms and data and implement controls to avoid unexpected outcomes.
Strategize: Fair & Impertial
Challenge
A media company plans to automate parts of the recruitment process and take more advantage of efficient AI-based decision-making. When it comes to the debate about whether AI is fair and whether it can make objective decisions, opinions among the staff differ considerably. Some argue that there should be a gender quota for new hires that reflects the company’s current diversity rate, while others believe the gender quota should apply to the pool of applicants. With such different perceptions of what is fair and what is not, the first step is to agree on a shared definition of fairness. People also often underestimate their own cognitive biases and implicit prejudices – they may even be completely unaware of them, which is how they keep ending up in our data and algorithms.
Solution
Working together with the company, we develop an approach based on multi-stakeholder participation to ensure diversity and inclusion at an early stage and to agree on a suitable definition of fairness. This gives our experts a good basis for workshops to raise awareness about implicit and cognitive biases. We also draft a code of conduct that will limit any potential misuse of the AI. The best way to combat bias is to select appropriate training data. Not only do we ensure the underlying data is representative, but that the team developing the AI is diverse enough to be sensitive to the perils of bias.
Outcome
By including stakeholders that are both directly and indirectly affected by the AI, the company is able to incorporate different views into its definition of fairness. The workshops also revealed that unintended and deliberate biases exist within the company, giving management an opportunity to resolve these issues while also creating the best conditions for fair AI practices.
Build: Fair & Imperial
Challenge
A bank is looking for an algorithm to fine-tune its credit risk decisions. Regulatory authorities expect this algorithm to use only standard criteria (e.g., credit ratings, employment status, disposable income) and to prevent discrimination of specific groups (by gender, ethnic identity or socioeconomic background).
Solution
Deloitte has developed Model Guardian and other tools to identify and measure biases in raw data, training data and model design. Using Model Guardian, we can detect, analyze and assess biases early in the data preparation process and for a wide range of AI models. Doing so ensures that the underlying data for a particular use case is representative and results are not distorted by unintended biases. Model Guardian’s tracking feature monitors predictive power against the degree of perceived bias of successive models.
Outcome
The bank’s credit risk model makes decisions purely on the basis of hard criteria without (potentially illegally) discriminating on protected classes (ethnicity, gender, ...) that could result in prejudiced and unfair decisions. As a result, the bank is not worried about reputational damage or penalties from regulators due to discrimination in the AI.
Integrate: Fair & Imperial
Challenge
A police department makes use of AI to predict criminal activity across neighborhoods it serves. The officers plan to intensify patrols in areas where higher crime rates are expected rather than those classified as safer by the AI. These predictions are designed to help the police department make more efficient use of its resources. However, shortly after launch, a vicious cycle takes shape caused by a self-perpetuating bias unwittingly designed into the AI. The distribution of touchpoints is imbalanced when more officers are sent to patrol one particular neighborhood rather than another – despite both having similar base crime rates. Intensified patrolling results in more arrests and more data points than in other, similar neighborhoods. The AI system is entirely unaware of the selection bias, associating more data points with greater danger.
Solution
To be able to improve the AI through ongoing data collection, Deloitte helps the police department implement control mechanisms to verify that the system is operating entirely without bias, notably by ensuring equal sampling distributions and avoiding selection bias. Should the model accuracy drift, police staff can reevaluate the data and interrupt the biased feedback loop.
Outcome
Through better understanding of how the AI works, the police department can use it the way it is intended and achieve its two primary goals: maximizing safety across all its neighborhoods and efficiently allocating its resources. The AI system has an in-built mechanism that detects potential disparities between neighborhoods and automatically makes recommendations on how to resolve them. This gives police officers a choice when it comes to adapting the way they assign police patrols and ample opportunity to improve the AI with fresh data.
Assure: Fair & Imperial
Challenge
A major telecommunications company would like to further automate its processes and has initiated a joint project between HR and IT to develop an AI system to evaluate job applications. This system aims to screen candidates by assessing various suitability indicators (e.g. open-mindedness, communication skills) on the basis of short video segments submitted by the applicants. Management wants to make sure that the AI system evaluates fairly, both to produce a healthy shortlist as well as to avoid reputational damage on allegations of prejudice.
Solution
Management commissions an independent audit, including a data quality assessment, an analysis of the pre-processing, an evaluation of the metadata model and a black-box test. One such test reveals a general discrimination against women. Deloitte’s investigation of the training data quality traces the bias back to an imbalanced and sparse training set. This is unearthed combining the innovative bias detection tool Model Guardian with an image recognition engine to extract visual features from the videos. The tool’s broad spectrum of metrics aid interpretation and analysis of these features.
Outcome
The test results show that while facial features indicative of gender are indeed removed, the model deduces gender from proxy features, for example clothing such as headscarves. The background imagery in the applicant video also plays a significant role in classifications made by the system. On the basis of our audit findings, the company changes the AI system to reduce the risk of unfair (and inadvertent) discrimination and to ensure all applicants are treated fairly.
Trustworthy AI must be robust & reliable
Employing statistical methods often with random initialization, special care must be taken to assure repeatable, auditable results of AI models. New algorithms and methods may offer great benefits for test cases, yet not be transferrable to others. Without expert knowledge and rigorous checks, models meant to do good can actually cause harm or be compromised.
For AI to achieve widespread adoption, it must be as robust and reliable as the traditional systems, processes and people it is augmenting. Companies need to ensure their AI algorithms produce the expected results for each new data set. They also need established processes for handling issues and inconsistencies if they arise.
Well-founded validation and test procedures are critical. They must extend beyond best-case accuracy to stress-test scenarios. Self-training models require monitoring so as not to be led astray by unintentional or unscrupulous manipulation.
Strategize: Robust & Reliable
Challenge
A utility company wants to use AI to predict cyberattacks and protect its system-critical infrastructure, but the company is totally unaware of the technical challenges, prevailing regulations and potential consequences. With so little experience dealing with AI, management is worried that the company and its infrastructure could face serious consequences if they get the predictions wrong.
Solution
The project team starts by identifying potential risks, providing suitable preventative measures and advising the utility company about the safeguards needed to avoid AI-specific difficulties. Deloitte provides support in building a suitable technology infrastructure that will guarantee stable operation of the AI system. A team of experts inspects the company’s systems to detect any technology or operational gaps that could prevent robust application of AI, testing the company’s processes and infrastructure in a series of gap analyses. It is particularly important to determine whether the existing system architecture is compatible with the technical and regulatory requirements of the planned AI system. Based on the findings of the analysis, the team designs a reliable (enterprise/IT) architecture.
Outcome
Deloitte’s recommendations help the utility company adopt a risk-conscious approach to the AI project, identifying issues early on and taking appropriate precautions. This provides the company with a solid foundation for future-proof implementation and resilient use of AI.
Build: Robust & Reliable
Challenge
An automotive manufacturer is launching a new fleet of cars with an automatic lane detection feature, which is a Level 4 automation based on the SAE automated driving scale. The system is having difficulty detecting roadworks, because there are as yet no definitive, established standards.
Solution
During the development phase, Deloitte already considers multiple design features to reduce risks. The focus is squarely set on developing resilient modeling structures and critical models to protect against adversarial attacks or unforeseen events.Using state-of-the-art techniques, such as randomized smoothing or generative adversarial networks, we train the deep neural networks to be robust against a wide constellation of roadworks. The development process also includes stress tests and adversarial tests to deploy the algorithm in different contexts and retrain the AI for situations where the model may not be sufficiently robust.
Outcome
The lane detection feature delivers reliable results in all the situations specified by the client’s quality assurance department. The automotive manufacturer meets the regulatory standards, and the new fleet operates safely and as intended.
Integrate: Robust & Reliable
Challenge
A home improvement retail chain has launched a new AI system to predict sales figures at the product level more accurately. However, the sales figures vary across stores due to demography, proximity to urban centers and other regional differences. The retailer must be careful not to rely on data from only a subset of stores to train their innovative AI model. Otherwise, the model would fail to generalize and the nationwide roll-out would function only in regions with strongly similar characteristics. The situation is further complicated by the planned integration of the AI into a complex IT landscape with a variety of legacy systems.
Solution
Deloitte deploys the AI model and corresponding data flow into the existing architecture, ensuring the data selected for training is representative of the data application will face in production. This improves accuracy of the AI model decisions with the selected features, both from the outset as well as over time by improving resilience to potential shifts in data. At the same time, Deloitte conducts tests on the CI/CD pipeline for the AI model and on the interfaces to peripheral systems. We provide sufficient documentation of all controls performed in order to guide future testing and updates. In the run-up to the roll-out, we conduct wide-ranging tests of the model itself, which has now been integrated into the overall IT infrastructure.
Outcome
Thanks to our extensive testing and representative data sets, the home improvement retail chain has a reliable AI model in place and can rely on more accurate forecasts of future sales.
Assure: Robust & Reliable
Challenge
A university hospital is collaborating with a tech startup to develop an image recognition software that characterizes a tumor as benign or malignant based solely on the CT scan. This AI system is considered particularly critical for the hospital, as false diagnoses may not only cause harm to patient health, but also lead to lawsuits and reputational damage. After delivering favorable results in a test environment, the hospital management agrees to roll out the software for use with patients. As one last cautionary measure, management needs assurance that predictions made by the system remain within an acceptable margin of error. The system should generally outperform human diagnosis, but in no case exceed a human margin of error. As the first AI-based tool in use at the hospital, the stakes are especially high.
Solution
To address the hospital’s concerns, Deloitte’s experts test the robustness of the AI system, using a collection of techniques ranging from stress-testing to white-box attacks, as well as benchmarking against so-called challenger models. The tests reveal one false assumption affecting the relevance of training images: as the quality of a CT scan varies from machine to machine in practice, the system must scale images in order to allow an AI model trained on them to deliver reliable predictions or any predictions at all. Auditing the testing process itself exposes the quality and thoroughness of model validation, as well as the quality of documentation. The stress tests determine whether the model is able to provide reliable results even in atypical situations.
Outcome
The results of the stress test reveal that the model exhibits insufficient and at times random behavior under stressful conditions. Deloitte communicates this and other findings to the hospital management, who tasks the development team with implementing the recommended adjustments. After another test run, the deficiencies are demonstrably resolved in the new AI model, and the hospital may confidently put the system into operation.
Trustworthy AI must preserve privacy
The volume – depth and breadth – of available data expands rampantly year over year. Tech-savvy organizations employ AI techniques to harvest its value. Yet all data may not be treated the same. Private or sensitive data deserves special treatment – regulation demand it.
We enable trustworthy AI by designing data privacy frameworks to comply with relevant laws and regulations. The GDPR provides a general framework, but other laws may be applicable depending on the scope of the AI. Organizations should ensure that consumer privacy is respected, customer data is not leveraged beyond its intended and stated use, and consumers can opt-in and out of sharing their data.
Strategize: Preserving Privacy
Challenge
A FinTech company is looking to develop AI-enabled products as part of the open banking initiative as well as a greenfield project. With such wide-ranging regulatory requirements and calls for stricter rules in the protection of personally identifiable information (PII), the company is unsure which regulations are currently in effect and how to comply with them for this new venture.
Solution
The project begins with a newly drafted data protection concept for the FinTech company. The first step is to identify the many regulatory requirements for legally compliant use of data and AI in the financial industry. In Deloitte’s RegTech Lab, the FinTech team and two expert teams from the Center for Data Privacy and the Center for Regulatory Strategy come up with a possible solution. This includes compliance checks with important draft legislation and directives, among other measures, in which Germany’s General Data Protection Regulation (GDPR) and the new Payment Services Directive (PSD2) both play an important role. Next, Deloitte develops a comprehensive data governance model that ensures compliance with the regulations we have identified. The project group designs practical/actionable instructions regarding data access (privileges), data storage (data sovereignty) and data processing (data standards, interoperability, anonymization).
Outcome
The resulting data privacy concept allows the FinTech company to develop successful, regulatory-compliant AI tools.
Build: Preserving Privacy
Challenge
A telecommunications company is working on an AI system to predict which product to offer its customers for which price. The Company would also like to determine the price-point that would make individual sales calls economically viable.. During the development phase, the company realizes that the data set they have is too sparse to deliver adequate results, as only a subset of the customers gave consent to use the data under the General Data Protection Regulation (GDPR). The company has additional data in its possession but lacks the authority to use it on the grounds of data protection.
Solution
Deloitte leverages its Anonymization Framework to develop the AI system in a GDPR-compliant manner. There are two main approaches, constructing synthetic training data and anonymizing existing data sets with state-of-the-art methods such as differential privacy or k-anonymity. These methods ensure hackers cannot trace the AI output back to an individual row in the data set, allowing us to mask the influence of any particular individual on the outcome and protect the privacy of each individual with a high degree of certainty.
Outcome
The final model complies with current data privacy regulations and can resist de-identification attacks. The telecommunications company is able to utilize highly-sensitive data as training data while adhering to strict legal restrictions. Knowing the system is robust, the company can now focus all of their effort on achieving the strongest performance from the model.
Integrate: Preserving Privacy
Challenge
With the introduction of the General Data Protection Regulation (GDPR), consciousness for data privacy has grown substantially. Many consider privacy an inalienable human right. This is highly relevant for the healthcare industry, where sensitive patient data requires secure handling and storage. One such example comes in form of the personal information provided to a diagnostic chatbot on a hospital website. Cloud technologies add another layer of complexity and often garner the most media coverage, but hospitals need to ensure this highly personal information is protected from unauthorized eyes from hospital staff or uninvited cyber spies as well.
Solution
Using encryption (client-side or server-side), anonymization and identity and access management (IAM), Deloitte protects the hospital’s sensitive and personally identifiable information (PII) in compliance with GDPR. That means all of the data inflows as well as outflows are under constant supervision and the patients’ privacy is guaranteed.
Outcome
The hospital can offer its patients services that leverage the value of their data for their own benefit, while making sure it is shielded from unauthorized access.
Assure: Preserving Privacy
Challenge
Some of the major shareholders in a medium-sized vacation rental company have become aware of media reports concerning court rulings and penalties against companies that fail to comply with data protection regulations. They ask senior management to verify that the AI systems in use at the company (e.g., for pricing) comply with relevant data protection rules. As the company only has a vague idea about data privacy, the management finds it difficult to assess the risk.
Solution
Deloitte conducts an independent audit of the company’s compliance with the General Data Protection Regulation (GDPR), not only to expose potential deficiencies but also to sensitize the management to the impact these regulations can have on its business. The broad experience of the Deloitte Center for Data Privacy helps tailor the audit specifically to the company’s needs. The audit team identifies which regulations and laws apply and then evaluate the current system on that basis. Both audit access privileges and the underlying data are within the audit scope, to avoid risk of granting access to non-essential employees. The Deloitte team inspects the underlying data for customer consent and for sufficient anonymization where consent is not granted. The team attempts to “de-anonymize” customer data in order to assess re-identification risk. The client is given access to the Deloitte GDPR Navigator in order to increase awareness and strengthen governance around data protection: users search current regulations or requirements for specific aspects, e.g., the relevant regulatory agency, the objective or the date of passage, and obtain the specific information the company needs.
Outcome
Our investigation finds the anonymization process to be inadequate: the audit team is able to identify the names connected to the data with only modest effort. In our audit of the rental process, we find the system does not uniformly fulfill foreign regulations and standards. The management begins to address the issues methodically, aided by its newfound awareness for data protection. GDPR-compliant AI-based solutions are within its reach.
Trustworthy AI must be safe & secure
Trained on data and – in the case of deep learning – expert labeling, AI models enable organizations to apply the hard won expertise at scale. For the multiplier effect to take root, AI models must “break out of the lab” and enter production environments, where they – like any models or any software – are exposed to the outside world, and with that are vulnerable to cyber-attacks.
To be trustworthy, AI must be protected from risks, including cybersecurity risks, that could lead to physical and/or digital harm. Companies need to thoroughly consider and address all kinds of risks and then communicate those risks to users.
Strategize: Safe & Secure
Challenge
A company is launching a line of smart home products such as home assistants to provide customers with a state-of-the-art centralized monitoring system. In order to provide customers the functionality they need and make optimal use of the system, the home assistants have to collect a wide dataset, for example whether the lights in the house are on or off, and store it temporarily. But because the system is always connected to the outside world, it is very vulnerable. Criminal actors could hack the system and use the data to find out whether the homeowner is at home to stage a robbery.
Solution
The Deloitte Cyber Strategy team assists the company in developing preventative measures before taking the home assistant live, using a threat landscape approach to identify all possible cyber risks, including data theft, at an early stage and develop effective countermeasures. Finally, Deloitte hosts a series of workshops and lab sessions to train the company staff in responsible and secure AI practices, helping them develop awareness for system-critical challenges and potential cyberattacks.
Outcome
These well-designed smart home products and the focus on data security give the company a distinct competitive advantage, because the system supports early detection of cyber risks and the staff is well prepared to respond as quickly as possible to cyberattacks.
Build: Safe & Secure
Challenge
A regulatory agency needs to implement a more effective system to detect financial crimes such as insider trading. The agency had successfully applied AI to discover increasingly nuanced patterns. However, criminal actors are testing means to manipulate the agency’s detection algorithm to hide their illegal activity beneath the detection threshold. As soon as the model is available online, money launderers observe how it behaves and attempt to reverse engineer its methodology. Their goal is to trick the model into perceiving their fraudulent transactions as legitimate.
Solution
Deloitte experts specify an environment and the criteria that will allow the model to operate safely, defining potential threat scenarios early in the design process to address possible attack vectors during the modeling phase. The system design considers both traditional cyber threats as well as AI-specific risks, for example if the data or the internal models leak to the public. This dual vulnerability underscores the importance of developing the system within a secure infrastructure where access is restricted, and the model can remain confidential. In addition, we only use secure software to develop the model and state-of-the-art techniques to train the AI, such as adding noise to the training data.
Outcome
Ensuring “Security by Design”, the regulatory agency can deploy its model confident in having addressed potential vulnerabilities and their implications. And there is no need for the agency to spend valuable resources on additional security experts during the post-development phase to close security gaps within the AI system.
Integrate: Safe & Secure
Challenge
A health insurance company is about to go live with an AI system designed to provide better risk coverage for their patients. The insurer intends to deploy into the cloud infrastructure of an external provider, taking advantage of its advanced firewall and other in-built security mechanisms. The insurance company seeks assurance that the system and its IT infrastructure are not vulnerable to cyber-threats that could allow adversaries to mount a denial of service (DoS) attack and shut down the system or use malware to steal sensitive data or manipulate the AI system.
Solution
Deloitte implements a system designed to provide secure identity and access management for the cloud service. At the same time, we check the configurations used in the company’s services and operations for errors. A specially trained Red Team discovers potential vulnerabilities during penetration tests and offensive hacking methods, e.g., through data poisoning, model evasion and DoS attacks. Deloitte also provides protection for the cloud environment with a 24/7 security operations center combined with threat intelligence services.
Outcome
The health insurance company can credibly claim to their policyholders that their cloud solution is well protected against cyber-attacks.
Assure: Safe & Secure
Challenge
A major logistics company uses a proprietary AI-enabled tool to analyze and forecast incoming orders. The tool contains highly sensitive, system-critical data for the company relating to customers, order volume and terms of offer. The security of the system is paramount. It is also showing its age, vulnerable to a new generation of cyber-attacks. The company lacks its own cyber-security team and is therefore unable to fully appreciate the extent of its cyber risk.
Solution
The company commissions Deloitte to audit the tool against the standards of the Trustworthy AI Framework. A penetration test (or “PenTest”) constitutes a major part of the audit, as it scans the company’s entire network. Experienced licensed specialists from Deloitte’s PenTest Lab launch a targeted, simulated attack of pre-defined vulnerabilities. We also run AI-specific tests alongside more generic software-related tests to determine whether the AI system is potentially vulnerable to data-poisoning or backdoor attacks.
Outcome
The audit discovers a vulnerability within one customization of the cloud configuration that provides would-be attackers backdoor access. Once identified, Deloitte experts draft a list of recommendations and a sequential plan to close these vulnerabilities. The investigation and recommendations increase the security of the existing tool, giving the company peace of mind with regard to cyber-attacks and the freedom to once again focus on analyzing and forecasting incoming orders.
Trustworthy AI must be responsible & accountable
The success and availability of open-source methods has generated great interest in AI models – for experts and debutantes alike. Models may be easily updated to new versions on new data. The proliferation of AI models introduces a governance challenge of its own: many models are unaccounted for and of differing levels of quality. Who is responsible if a model fails? The developer, the tester, the product manager, machine learning engineer, or senior management? The person intimately familiar with the model’s inner workings or the person who may have to testify before a government body?
The foundation of trustworthy AI is an end-to-end enterprise governance framework, which focuses on the risks and controls of an AI system. To establish responsible AI the framework does not only have to incorporate all parts of the Trustworthy AI framework but also answers questions of who is responsible and accountable for their output.
Strategize: Responsible & Accountable
Challenge
A private clinic is looking for an AI-enabled decision aid that uses patient data in order to prescribe the most appropriate medication. The aim is to provide more efficient treatment while also reducing personnel costs. However, during the project’s initial development and design phase, several key questions remain unanswered: whether the system is practical or advisable, who will be responsible and/or accountable for the system, and what other potential implications might be.
Solution
The Deloitte team conducts its initial impact/value assessments and acceptance tests in the clinic to determine whether the planned AI system is ethically responsible and whether it is something patients want. At the same time, our subject matter experts work closely with clinic staff to develop a “chain of accountability” that will determine what measures to take and who is responsible if the AI system prescribes the wrong medication, puts certain patients at a disadvantage or prescribes an incorrect dosage. The team takes part in a workshop to assign and define the roles and responsibilities of different stakeholders across all phases of the AI lifecycle. At the same time, project managers assess the digital maturity of the workforce and raise awareness for ethical issues in Deloitte’s Corporate Digital Responsibility Lab. The expert team also advises the clinic as they establish the technical and operational guidelines for later project phases. They support the clinic during the roll-out of the AI governance framework/monitoring mechanisms and test them in a hands-on lab. This enables clinic management to systematically identify, monitor and audit all risks/objectives.
Outcome
The clinic staff is now more aware of what constitutes responsible AI use and of what potential risks and objectives to include in their strategy and review process. In addition, the clinic has a clear idea which stakeholders are accountable for which phases of the AI lifecycle.
Build: Responsible & Accountable
Challenge
A bank is building an AI-enabled robo-advisor designed not only to help customers shape their ideal portfolio, but also to automatically buy and sell assets. The bank sees a potential problem in giving the robo-advisor full responsibility for the proper management of each portfolio. After all, a robot may not fully register all that is happening in its environment, e.g., changes to purchasing behavior of individual stock traders, and may exceed predefined risk thresholds.
Solution
With an area as complex as the stock market, Deloitte added control limits and an “emergency brake” to the robo-advisor software, by which the client can switch off the autonomy if the robot behaves erratically due to significant changes in environmental variables or fails to comply with predefined risk thresholds. The system monitors key indicators that have been set in advance, e.g., daily transaction volume or the current risk of a customer’s portfolio, to monitor changes in the financial markets and the predefined risk thresholds. The system also includes an early warning system that alerts the product owners to potential changes and reminds them to keep the robo-advisor up to date through periodic re-training of the underlying algorithms. Audit trails round out the solution: they capture important regulatory data and changes in the model lifecycle, e.g., volatility changes in active trading, and records them seamlessly and chronologically.
Outcome
Despite the complexity of the stock market environment, the bank can safely operate the robo-advisor. The “emergency brake” and the early warning system limit the downside risk for customers, while the entries in the audit trail log allow the bank to trace back poor decisions made by the system, correct them and prevent them from happening again in the future.
Integrate: Responsible & Accountable
Challenge
A bank is designing an AI-enabled risk model to assess the creditworthiness of loan applicants. Because the bank sees itself as a responsible lender and a customer-focused financial institution, it wants to explain the decision-making process in a way that is understandable for customers.
Solution
Deloitte provides extensive support for the bank’s plan to provide a customer experience fulfilling both the functional needs and high standards that customers expect of the institution. We implement a feedback loop toward customers that makes the decision-making process more transparent and provides instant access to the relevant decision drivers. If customers report a negative experience, the bank can use the integrated feedback loop to establish direct contact with the responsible member of staff.
Outcome
The bank has an enterprise-wide control system in place to ensure error-free operation of all processes and an AI monitoring system to ensure all of the decisions are transparent and documented. For ethical reasons, the bank will also inform loan applicants that their applications have been processed using an AI-enabled system, which will allow the bank to continue to use its innovative risk classification system in good conscience.
Assure: Responsible & Accountable
Challenge
A bank is designing an AI-enabled risk model to assess the creditworthiness of loan applicants. Because the bank sees itself as a responsible lender and a customer-focused financial institution, it wants to explain the decision-making process in a way that is understandable for customers.
Solution
Deloitte provides extensive support for the bank’s plan to provide a customer experience fulfilling both the functional needs and high standards that customers expect of the institution. We implement a feedback loop toward customers that makes the decision-making process more transparent and provides instant access to the relevant decision drivers. If customers report a negative experience, the bank can use the integrated feedback loop to establish direct contact with the responsible member of staff.
Outcome
The bank has an enterprise-wide control system in place to ensure error-free operation of all processes and an AI monitoring system to ensure all of the decisions are transparent and documented. For ethical reasons, the bank will also inform loan applicants that their applications have been processed using an AI-enabled system, which will allow the bank to continue to use its innovative risk classification system in good conscience.
Trustworthy AI must be transparent & explainable
Where AI models may excel in accuracy, they leave much to be desired in terms of transparency. Opacity increases with model complexity, non-linear models may well reflect the world they approximate, but are notoriously difficult to explain. Limited interpretability translates to limited acceptance. For AI to be trustworthy, all participants have a right to understand how their data is being used and how the AI system is making decisions. Organizations should be prepared to build algorithms, attributes and correlations open to inspection.
Strategize: Transparent & Explainable
Challenge
An insurance company wants to use AI to make faster, more precise decisions in calculating insurance premiums. Ideally, the AI-enabled calculator will not only deliver more exact outcomes but will also make the process easier to understand for a variety of stakeholders. When insurance premiums go up, most customers want to know the reason why, and – under Article 22 of the Germany’s General Data Protection Regulation (GDPR) – they have every right to. Developers need to fully understand how their AI works in order to improve transparency in these automated decisions. Ultimately, it is important for the AI solution to calculate insurance premiums as accurately as possible – and to do so in a way that is straightforward and easy to understand.
Solution
Our experts use Deloitte’s Stakeholder Assessment tool to identify who or what is involved in every step of the process from development to AI-enabled decision-making. We collect input from all stakeholders (i.e., developers, insurers, prospective insurance customers) and use their interests and preferences to determine what kind of explanations they need – e.g., a global explanation to help people understand the model itself or a local explanation to clarify one specific automated decision – how to present the explanations, and whether to display them as a visualization or with text. The objective is to ensure that the decision-making process is completely transparent, without jeopardizing the accuracy of the AI-enabled system or violating regulations. At the same time, we establish a systematic framework that makes it easier to document the data we collect and use, which is important if we need to retrace the decision-making process.
Outcome
Identifying stakeholders early on in the process enables the insurance company to draft a few different approaches to the AI solution and compare them, ultimately leading to a solution that provides straightforward explanations for all relevant stakeholders as well as premium calculations that are as accurate as possible.
Build: Transparent & Explainable
Challenge
A hospital plans to launch an AI-enabled virtual assistant to help doctors make faster, better decisions and detect disease with greater accuracy. AI-supported MRI imaging, for example, can automatically detect potentially malignant tumors. Regardless of the system’s benefits, doctors as well as patients expect the algorithm to reveal how it arrives at its diagnosis. This is necessary to prevent doctors from recommending ineffective treatments more likely to put patients at risk than to make them better. When it is a matter of life and death, it is essential for the AI system to fully explain its decision-making process.
Solution
Deloitte implements a state-of-the-art AI model that improves diagnostic accuracy and delivers the desired transparency. Thanks to the range of tools in Deloitte’s Lucid [ML], the system can explain the decision drivers at the global or the local level. Key features/drivers of the expected results are visualized in such a way as to convey which regions within the image determine the diagnosis. Lucid [ML] articulates the drivers in the model in a straightforward way, helping doctors understand and validate the process.
Outcome
The hospital now has a cancer detection system using a high-tech neural network that offers accuracy, transparency and traceability. With the ability to detect even more subtle signs of malignant tumors, doctors have a better chance at saving lives; with the user-friendly transparency dashboard, doctors and patients have more confidence in the quality of the diagnosis.
Integrate: Transparent & Explainable
Challenge
An insurance company notices an increase in complaints from certain customer groups. Particularly, customers reporting claims for minivans are consistently – yet inexplicably – automatically rejected. The claimants are exasperated and demand an explanation. Customer support invests time and effort into a more in-depth investigation of these automated decisions in order to provide a response.
Solution
The explainable AI toolset Lucid [ML] developed at Deloitte’s aiStudio enables companies to display the AI system’s decision-making process in an intuitive way suitable for both technical and non-technical staff. By integrating the explainable AI toolset into the code of the insurer’s AI application, customers have a dedicated field in the online user interface that proactively offers insight into the rationale for a claim denial. This feature is tested first internally with customer service staff to ensure that the underlying AI claims adjudicator operates correctly and then incorporated into the customer-facing web portal. Claimants can now see for themselves why their claims were rejected.
Outcome
Adding an explainability element to the insurance company’s portal increases policyholder satisfaction. Greater transparency in the claims process and instant availability of the reasons for claim denials improve the insurer’s credibility. Policyholders now have faster, higher quality information at the click of a button and the company saves time and effort in customer support.
Assure: Transparent & Explainable
Challenge
As part of its loan business, a large German bank seeks to improve its credit risk forecasts and offer customers more personalized services. The AI system developed by the bank for this purpose (based on an IRB approach) uses customer data to calculate and issue credit ratings as well as a total credit limits for customers. The system relies on an underlying AI model that has been trained with historic data and requires an annual risk-based evaluation of all aspects of the IRB systems. As this is a substantial new system (and underlying technology), the bank engages an external firm to conduct an audit.
Solution
Deloitte has long-standing experience in audits such as these. We expose the drivers behind a model using state-of-the-art techniques and tools, such as the aiStudio’s Lucid [ML] to explain the inner workings of the overall model as well as individual decisions in an intuitive manner. Additional measures form an important part of these audits, such as interviewing a sample of customers to determine whether the decisions made by the AI system were clearly articulated. In order to comply with regulatory standards, the audit must also identify which algorithms the system uses and establish whether the results of the model can be fully disclosed.
Outcome
The explanations constitute multiple components including feature importance analyses, surrogate models and counterfactual explanations. Our audit of the bank’s AI system finds some parameters required to articulate the decisions to be missing, meaning it cannot provide full disclosure. In addition, the parameters being used remain opaque; so even though users receive the correct numbers, they cannot properly interpret them. We present our audit findings to the bank, and they initiate steps to correct any potential deficiencies. After these adjustments, we issue a positive audit report and submit it to the financial authority BaFin. The bank can now rest assured that the results of its AI system will satisfy regulatory transparency requirements.
Relevant for the entire life cycle
One approach to Trustworthy AI could be to leave it to the auditors. "Launch and learn", "Fail fast". Agile processes are highly effective in achieving concrete results quickly. However, we should be careful not to use the trendy terms in the wrong context. What works perfectly well for prototype development or A/B testing is likely not sufficient for solid and reliable implementation into a production environment.
As we know it from the manufacturing process: the quality costs increase exponentially the later an error has been identified. This is no different with AI. Failed AI models can not only incur economic costs, but also reputational damage. This can affect all developers – even the tech giants who owe some of their spectacular growth to AI. Yet true disaster cases have so far remained thankfully few. We attribute this to two dynamics:
- The tech giants are investing heavily in Trustworthy AI
- Other companies have little exposure as yet, only now emerging from the proof-of-concept (PoC) phase, when AI models have been shielded from the dangers of the outside world
Tight deadlines, limited budgets and other pressures increase the risk of errors. Clear priorities and a rigorous approach are needed to ensure trustworthy by design: from conceptualization to prototyping, integration to testing, and ultimately monitoring and general governance.
Trustworthy AI Framework | How can it help you?
Find more relevant cases and information about trustworthy AI in you industry or sector.
Ihre Ansprechpartner
