Dynamic AI governance: A recipe for crafting trustworthy AI
What does it involve? Fusing diverse governance strategies, evolving with technological and policy shifts, and incorporating a dash of crucial human touch.
Bruce Chew
Sarah Milsom
Aaron Roberts
Carolyn West
Angela Huang
Introduction
Impactful innovations generally require some oversight to be effective. Just as artificial intelligence is entering the public consciousness as something requiring regulation, car design and safety also underwent a regulatory transformation. In the years since the first Ford Model T, governments developed guidelines like traffic laws and licensing requirements, manufacturers shifted focus to safety and reliability, and designers developed new safety features such as rear-view mirrors and seat belts. None of these actions on their own are sufficient to make driving safe—they need to work together. Similarly, governing AI for government requires an adaptable approach that integrates common governance approaches to form one cohesive strategy.
In the past five years, the federal government has promoted trustworthy AI1—that is, AI systems that are safe, secure, reliable, and transparent—and set forth guiding principles, such as those outlined in the Office of Management and Budget’s (OMB) Memorandum M-21-06.2 Focus is now turning to implementation. Nearly half of all enterprise leaders surveyed say they are investing more in responsible AI in 2024 than ever before.3 These investments are coinciding with increased legislative and executive actions, such as the White House Executive Order 14110 on “safe, secure, and trustworthy development and use of AI,” which prioritizes the implementation of trustworthy AI practices across all American government and commercial enterprises.4
For government leaders tasked with implementing responsible and effective AI solutions, prioritizing a cohesive AI governance strategy that works with its people’s existing tendencies for governance is likely critical to success. Currently, three personas represent the three common approaches to effective AI governance: Guides, who focus on enterprise policymaking like Executive Order 14110; Guards, who focus on standardized quality assurance checkpoints; and Gadgeteers, who focus on tooling and feedback mechanisms.
While these approaches can provide a strong foundation, the exponential pace of AI innovation should have dynamic AI governance that can integrate all three approaches and adapt as AI continues to evolve.
Core challenges with AI governance
Governance involves the management of business processes to promote efficacy and efficiency while mitigating risks and liabilities. Cars, computers, mobile phones, and other transformative innovations of the 20th and 21st centuries spawned their own governance bodies like Federal Communications Commission or National Highway Traffic Safety Administration. So too does AI call for its own governance approaches. But given the fast-moving nature of AI, these approaches should be even more dynamic to accommodate the unique AI challenges, such as:
- AI models are built to continually change over time: AI models are often built with the inherent ability to adapt, based on new data. This characteristic makes the technology powerful and resilient. However, without proper governance at critical junctions, the accuracy and reliability of the model’s outputs may not align with the model’s intended use over time.
- AI products vary based on the context and type of AI: AI users and stakeholders may have different risk thresholds depending on the context in which an AI product operates. For example, an AI product that performs a mission-critical function like fraud detection in medical billing should have more accuracy and reliability than an AI product that provides personalized recommendations on a webpage. Based on the type of AI, the risks and consequences may look different as well. For example, generative AI poses unique risks of producing deepfakes—false videos that impersonate individuals via video or audio.5
- AI technology is advancing rapidly: Twenty-five percent of companies are turning to AI adoption to address labor-shortage concerns, marking a future where a slowdown of AI adoption is highly unlikely.6 Combined with constant advancements in AI systems and tools, it can be difficult to maintain effective governance, as policies and standards quickly become outdated.
- AI products are subject to human bias and error: Improperly supervised training for AI models can lead to outputs that perpetuate biases in society. Without proper quality controls, human error can also pose risks to systems, solutions, and the public. For example, research shows that injecting 8% of “poisonous” or erroneous data, whether done intentionally or by accident, can decrease an AI system’s accuracy by 75%.7
AI governance personas and approaches
Exemplifying a persona, government agencies typically follow one of three approaches to AI governance.8 Success often comes from fusing these three AI governance personas or approaches.
Guides focus on establishing clear policies and guidelines to promote safe, secure, and transparent outcomes. Guards focus on establishing standardized checklists and stage gate reviews during the AI product life cycle to facilitate quality assurance. Gadgeteers focus on embedding tools in the AI product life cycle to help evaluate the trustworthiness of model outputs. Although agencies may consider all three approaches, they often frontload effort in one area. Consider taking a dynamic approach to AI governance that integrates all three approaches and adapts over time to help achieve trustworthy AI goals.
Guides
Guides focus on establishing enterprise policies to document point-in-time standards with the expectation that others in the organization will follow suit. This approach typically involves policy-setting at different levels of governance, such as the federal, state, or enterprise levels. Examples include:
- Federal and executive office direction: Federal agencies and executive offices set national standards around AI adoption, responsibilities, and future alignment. For example, the White House released a Blueprint for an AI Bill of Rights (2022), which highlights a broad set of policies to guide AI adoption.9 The National Institute for Standards and Technology subsequently released an AI Risk Management Framework Playbook (2023), which provides suggested actions to mitigate AI risk in design, development, deployment, and use.10 Most recently, OMB released memorandum M-24-10 to strengthen governance of AI products that impact the rights and safety of the public. The memorandum includes detailed guidance and requirements, such as designating a chief AI officer, developing an AI strategy, and inventorying AI use cases.11
- Internal policies for relevant stakeholders: Government agencies often use national standards to develop policies tailored to the needs of their organization and partners. For example, following the boom of gen AI, Governor Newsom of California issued an executive order in September 2023 that directed state agencies to develop guidelines for gen AI adoption.12
Guides can lay an important foundation but may set policies that are resource-intensive and can potentially cause ineffective operational processes. On their own and without supplementary activities, Guide-driven approaches may fall short because:
- The pace of AI evolution quickly outdates policy: The fast pace of AI growth, as with the democratization and rapid expansion of gen AI, demonstrates how sudden AI advancements can outdate existing policies. As a result, static policies may limit or delay the adoption of innovative AI technologies that could drive better solutions and outcomes.
- Guides can become a bottleneck for innovation and effective implementation: Overemphasizing policy-writing can create a resource trap for talent that could deliver more value through effective implementation. Cumbersome policies can also be a barrier to innovation and experimentation that organizations want to move forward.
- Inflexible policy can create standards gaps that can create additional risk: Organizations often tailor overly prescriptive policies to fit their own needs, which can create inconsistent standards and additional risks. On the other hand, it can be difficult to measure compliance if policy is too broad. Contractors and other third-party entities may have a different set of organizational policies that could affect project compliance. Policy compliance may also create a false sense of security when other risks not covered by policies are still present in the AI product.
Guards
Guards focus on managing AI risks through quality assurance checkpoints during the AI product life cycle. Similar to the Federal Information Security Modernization Act requirements for information systems, Guards aim to establish an approach typically involving two components for AI development, operations, and maintenance:
- Standardized AI product requirements: Guards often establish detailed requirements that AI products must meet prior to the next phase of development, such as specific evaluation metrics and thresholds. For example, OMB memorandum M-24-10 establishes minimum risk management requirements for rights- and safety-impacting AI prior to deployment, such as completing an AI impact assessment and testing performance in a real-world context.13 These checklist-style stage gates aim to enhance quality assurance during each phase of the AI product life cycle.
- Standardized procurement requirements: Guards often focus on codifying requirements in solicitation language and contract clauses to ensure third-party vendors comply with the same quality assurance standards. For example, a recently proposed bill—the Federal Artificial Intelligence Risk Management Act of 2023—would require the Federal Acquisition Regulatory (FAR) Council to establish risk-based compliance requirements for the acquisition of AI products.14 Executive Order 14110 similarly directs the FAR Council to consider amending the Federal Acquisition Regulation as appropriate.15
Standardized checkpoints and templatized procurement requirements can improve consistency in AI governance by holding both internal and external parties accountable to baseline expectations. Taken on its own, however, the Guards’ approach often faces three primary challenges:
- Standardized checkpoints may not account for differences in AI products: The use case and technical components of AI products require different methods for evaluating trustworthiness. Safe and rights-respecting AI is particularly important for use cases that directly impact individuals.16 For example, an AI product that recommends eligibility determinations for human services programs like income assistance may require more rigorous quality assurance checkpoints than an AI product that performs a back-office administrative task like automated trends analysis in expense reports. These nuances in AI products require product-specific metrics and checkpoints.
- Standardized checkpoints may not account for end users of AI products: While checkpoints can help build quality assurance into the AI product life cycle, they rarely address the legal and reputational risks associated with end users of AI. For example, users may apply AI outputs in unethical ways, such as using inaccurate outputs to spread misinformation. These user-based risks require additional layers of AI governance beyond predefined checkpoints.
- Standardized checkpoints may not account for changes in AI over time: Like enterprise policies, checkpoints are point-in-time reviews that may miss significant AI risks that develop over time. For example, a model may pass a deployment checkpoint but later suffer from model drift, which refers to a decrease in performance due to environmental changes. These evolutions in AI models require more continuous monitoring than checkpoints alone can provide.
Gadgeteers
Gadgeteers focus on governing AI with platforms and tools that automate governance and management of AI products. This approach typically involves the following components:
- Tools to enhance performance: Gadgeteers often implement tools that improve the quality of AI model outputs. For example, Gadgeteers may use bias detection tools to identify and mitigate sources of inequity in AI products or monitoring tools to detect model drift following deployment.
- Tools to strengthen security: Gadgeteers often employ tools to improve the privacy and security of AI models and their data. For example, red-teaming tools, which simulate real-world adversarial attacks on an AI product, are a common strategy to test the security of machine learning pipelines.
- Tools to promote accountability: Gadgeteers may use tools to help improve accountability and transparency in AI, such as watermarking tools that can enable end users to track AI-generated content or auditing tools that enable agencies to assess compliance with legal and ethical requirements.
Tools can be an effective method for tracking and improving model performance, but the Gadgeteer-focused approach can experience two challenges when implemented without other activities:
- Tools cannot evaluate the trustworthiness of an AI use case: Prior to development, AI product managers should consider if the intended use of the AI product is a responsible one. To do so, AI product managers should understand the organization’s mission and consider any unintended consequences on the organization’s staff, partners, and beneficiaries.
- Tools cannot evaluate compliance with all AI policies: Tools can only test and evaluate AI models and their outputs. AI product managers should consider other applicable policies, such as data privacy policies, to confirm that model inputs and uses are also compliant.
Dynamic AI governance
Consider the important aspects of AI governance
Guides, Guards, and Gadgeteers address important aspects of AI governance, but none on their own are sufficient to achieve trustworthy AI goals. Imagine you are embarking on a backpacking expedition. A Guide may help you set a path but may not help you re-route if weather affects your course. A Guard may provide checklists to help you pack, set up a campsite, and complete other milestones on your journey but may not help you face unforeseen obstacles along the way. A Gadgeteer may help you choose durable tents, hiking boots, and other gear, but may not help you evaluate the expedition itself.
To achieve trustworthy AI goals, agencies should integrate all three approaches and incorporate mechanisms for short-term and long-term adaptations in AI governance. Together, the five key features of dynamic AI governance can help form a well-rounded and adaptable governance strategy.
- A clear framework for trustworthy AI to inform governance activities: A dynamic approach to AI governance can start by adopting the Guide mindset and outlining a set of principles that defines trustworthy AI in the context of an agency’s mission and goals.17 To do so, organizations should consider what needs to be true about their AI products to maintain the trust of staff, partners, and the public. Organizations may choose to adapt principles outlined by the Executive Office of the President or other government agencies. For example, the Blueprint for an AI Bill of Rights includes principles like safe and effective systems and data privacy, while OMB Memorandum M-21-06 includes principles like flexibility and interagency coordination.18 An effective framework should address both technical considerations (for example, the need for reliable, secure AI model outputs) and functional considerations (for example, the ability for end users to interpret AI model outputs). This type of framework can provide members of an agency with a shared understanding of what trustworthy AI means for their organization and a reference point for future governance decisions.
- Measurable outcomes to track progress and protect against quality issues: A dynamic approach can incorporate the Guards’ mindset by establishing a clear list of expected outcomes for AI governance tied to the agency’s mission, vision, and trustworthy AI principles. By outlining requirements, enablers, and barriers to success, these outcomes help agencies identify and prioritize AI governance activities that can have the greatest impact on AI trustworthiness. For example, if non-discrimination is a core component of the agency’s trustworthy AI framework, the agency may include a percentage increase in model fairness in its list of expected outcomes and use metrics like equalized odds to measure progress.19 When establishing measurable outcomes for AI governance, agencies should consider trade-offs between different goals. For example, AI models that are easier to interpret may have less predictive accuracy.20
- Tools and feedback mechanisms to support continuous improvement: A dynamic approach incorporates the Gadgeteer mindset by building in tools to accelerate AI governance processes like performance monitoring. In particular, a dynamic approach creates avenues by which staff, partners, and end users can consistently share feedback on AI products in a timely fashion. Government agencies may consider traditional avenues for public input, such as the White House’s public listening sessions, panel discussions, and formal request for information to inform the Blueprint for the AI Bill of Rights.21 Government agencies may also consider nontraditional avenues to gather live input, such as public red-teaming competitions like those conducted at the 2023 DEF CON convention in partnership with the White House Office of Science and Technology Policy.22 For more frequent feedback, agencies can also turn to automatic feedback mechanisms, such as gathering citizen feedback directly from websites, as frequent feedback mechanisms provide critical information about AI risks and opportunities, helping agencies to improve AI products over time.
- Continuous adaptations to stay abreast of new challenges: A dynamic approach is inherently one that adapts to changes in AI technology, such as recent advancements in gen AI. ChatGPT became the fastest-growing consumer application in history when it reached 100 million monthly active users two months after launch.23 Moreover, 31% of the 2,835 business and technology leaders who participated in Deloitte’s State of Generative AI in the Enterprise Survey expect gen AI to substantially transform their organization and industry in less than one year.24 To keep pace with these and other evolutions in AI, government agencies should regularly revisit and enhance policies, checklists, and tools to ensure they continue to meet the agency’s trustworthy AI goals.
- Humans in the loop to tailor AI governance to mission needs: A dynamic approach should rely on maintaining a minimum number of individuals at the enterprise, business unit, and product level to design AI governance to meet the specific, nuanced risks of AI products and the environments in which they operate. While some AI products may not require human oversight for all model actions and outputs, humans in the loop provide an important layer of governance for higher-risk AI products that impact the safety and rights of the public.25 Teams with a combined focus on AI technology, the agency’s mission, and the individuals and communities the agency serves are often well-equipped to play the human-in-the-loop role.
What next?
As AI adoption spreads and AI technology grows in sophistication, the need for effective governance is expected to increase, and in turn, so would a need for those who can oversee the integration of dynamic AI governance. Agencies with the greatest success will likely require individuals with the instincts of Guides, the sensibilities of Guards, and the savviness of Gadgeteers to drive dynamic AI governance. To do so, agencies should bring together individuals who understand the potential benefits of AI technology, can identify current and future challenges with AI governance, and are intimately familiar with the agency’s mission. These cross-functional teams can strategically link AI governance activities to the agency’s mission, vision, and desired outcomes.
In the coming years, there will likely continue to be an increasingly systematic focus on building and maintaining responsible and effective AI solutions. Just as safety features in cars work together and evolve over time to effectively protect drivers and passengers, government agencies should integrate and continuously adapt AI governance approaches to proactively instill trustworthy AI principles across the enterprise. By leaving room for future changes, this dynamic approach to AI governance can set the foundation for truly trustworthy AI practices.