Managing the black box of artificial intelligence (AI)

Why is this trend important today?

This article is one of nine trends outlined in Deloitte's Future of risk in the digital era report. 

Organizations have begun using artificial intelligence (AI) techniques and solutions to affect outcomes across a broad range of purposes, such as approving loans, identifying fraudulent transactions, and performing surveillance.

These applications often operate like “black boxes” for decision making. If they produce results without explanation, they make detection of inappropriate decisions difficult. This exposes the organization to vulnerabilities, such as biased data, unsuitable modeling techniques, and incorrect decision making, in the algorithm lifecycle.

As algorithms become more powerful, pervasive, and sophisticated, the methods for monitoring and troubleshooting them lag behind adoption. Organizations should consider seeking transparency and accountability in how decisions are made by algorithms; consider ethics, fairness, and safety in how algorithms are used; and adopt new approaches to effectively manage the novel risks introduced by complex AI algorithms.

Back to top

Where has this trend had an impact?

• A health care program witnessed backlash from patients when it replaced a manual process with an algorithm that started reducing patients’ medical benefits leaving some patients unclear about the rationale behind the decisions.
  
• A crime-predicting algorithm that judges consulted when making sentencing determinations of convicted criminals came under scrutiny when it was found to be no better at predicting an individual’s risk of recidivism than random volunteers recruited from the internet.
• Many organizations pulled their advertisements from a website after its programmatic advertising algorithm placed ads next to inappropriate content.
  

Back to top

What does this mean for organizations?

• Erroneous decisions made by algorithms trained on biased data leading to poor product performance and even negative impacts on the health and safety of consumers.
• Overlooked vulnerabilities if traditional risk management processes, such as IT general controls and internal audits, fail to consider algorithmic risks.
• Increased scrutiny and higher expectations from consumers for algorithm-driven products to perform without errors.
• Reputational, legal, and regulatory consequences as a result of algorithms making decisions that don’t align with social, ethical, cultural, or legal norms.
• Delays in proper redress of business issues in the absence of clear accountability guidelines for managing malfunctioning algorithm-dependent systems.
• Third-party induced risks due to limited visibility into algorithm design and underlying training data when using commercial algorithm-based solutions from external vendors.

Back to top

How can organizations respond?

• Assess the AI applications inventory and develop AI risk management strategy and governance structure, covering areas such as policies, training, roles, and responsibilities.
• Review black box algorithms (internal and external) and establish controls across the algorithm life cycle, including data gathering, preparation, model selection, training, evaluation, validation, and deployment.
• Institute standardized disclosure practices to inform relevant stakeholders when decisions affecting them are being made using algorithms.
• Develop and maintain surveillance processes to help monitor outcomes from algorithms.
• Conduct periodic independent auditing or validation of algorithms based on established baseline parameters to test the validity of the training data, assess security against manipulation, and optimize model performance.
  
• Engage with researchers, think tanks, and innovators to adopt leading practices and cutting-edge tools to address algorithmic risks.

Back to top