As Artificial Intelligence’s (AI) use in business and government increases by leaps and bounds, the need for transparent, fair, and safe governance standards has moved off the planning list into action. In a vacuum of universal guidelines, many corporations and governments have already begun to create their own standards to address the key topics of model transparency, explainability, and fairness. While these guidelines have been an excellent start, there is still a need for organizations to control and guide their AI development while also keeping the current and emerging regulatory environment in mind.
Due to the nascent nature of the field, there are few widely implemented and agreed upon best practices. The 2023 World Privacy Forum (WPF) report on Assessing and Improving AI Governance Tools[1] attempts to remedy this gap by highlighting examples across 6 categories:
WPF are using the following definition of AI governance tools to shape their work: “socio-technical tools for mapping, measuring, or managing AI systems and their risks in a manner that operationalizes or implements trustworthy AI”. In other words, when making decisions with AI, it is necessary to account for the system's transparency, explainability, fairness and potential societal impact.
| Category | Tools |
|---|---|
| Practical Guidance | Includes general educational information, practical guidance, or other consideration factors |
| Self-assessment Questions | Includes assessment questions or detailed questionnaire |
| Procedural Framework | Includes process steps or suggested workflow for AI system assessments and/or improvements |
| Technical Framework | Includes technical methods or detailed technical process guidance or steps |
| Technical Code or Software | Includes technical methods, including use of specific code or software |
| Scoring or Classification Output | Includes criteria for determining a classification, or a mechanism for producing a quantifiable score or rating reflecting a particular aspect of an AI system |
These tools mentioned are in active use worldwide, making them an excellent starting point to ensure compliance with present and future regulations. Without the appropriate tools in place, it is nearly impossible to ensure a trustworthy AI future. A survey conducted by the AI Risk and Security (AIRS) group evaluated the current status of AI governance across their members (Figure 1). The survey revealed that there is room to improve, with only 30% of enterprises have established roles or responsibilities for AI systems, and just 20% have a centrally managed and budgeted department dedicated to AI governance.
“Global spending on AI is expected to increase from $150 billion in 2023 to $300 billion by 2026. The use of AI is moving at a rapid pace with regulators’ eyes keeping a close watch, and we’re seeing leaders in the TMT industry create their own governance tools as a commercial and operational imperative.” George Haitsch, WTW Technology, Media and Telecom Industry Leader
“Global spending on AI is expected to increase from $150 billion in 2023 to $300 billion by 2026.”
George Haitsch | WTW Technology, Media and Telecom Industry Leader
Implementing several AI governance tools fosters a collective understanding and assessment of possible risks and limitations in AI explainability, transparency, and fairness across different departments. All personnel engaging with AI models should possess awareness regarding its capabilities and shortcomings, emphasizing the importance of maintaining a skeptical perspective toward AI outputs.
Notes: Respondents included professionals from technology risk, information security, legal, privacy, architecture, model risk management, and other fields, working in financial and technology organizations, as well as academic institutions.
Regulation is evolving in different pathways around the world. Policy makers across the globe are hurriedly addressing the issues, gaps, and limitations of AI driven decisions.
This has prompted a surge of literature and research into the subject. In July 2023, the U.S. Biden administration announced[3] that Amazon, Anthropic, Google, Inflection, Meta, Microsoft and OpenAI will self-regulate their AI development.
The EU produced a similar set of ethical guidelines to the US with 7 key requirements including transparency, accountability, and respect for privacy and data protection[6]. They also proposed a classification scoring system. In the European Union’s A.I. Act, regulations are assigned proportionate to the level of risk posed by an AI tool: unacceptable, high, limited, and minimal risk.
Canada produced a mandatory risk assessment tool in 2020 named as the Algorithmic Impact Assessment (AIA).[7]
Singapore launched a technical framework and software with AI Verify[8] in partnership with companies from different sectors and scale including AWS, DBS Bank, Google, Meta, Microsoft, Singapore Airlines, and Standard Chartered Bank. It became open source in June 2023.
In June 2023, China revealed that it was developing an “Artificial Intelligence Law” that could offer practical guidance, a technical framework, and a classification scoring system. Rather than devising a comprehensive regulatory plan, China has chosen to regulate AI through individual legislation, with specific laws addressing issues such as algorithms, generative AI, and deepfakes.
Global companies will be facing increasing pressure to comply with regulatory standards in AI governance. This often presents a challenge as they must also comply with other regulatory measures, such as ESG. As a result, companies may spend more time disclosing information about their algorithms rather than making tangible progress.
Several AI governance tools, including fairness AI auditing software, may be limited in their application to specific phases of the AI life cycle. AI fairness tools might solely address fairness concerns during the model training stage of AI development. However, ensuring fairness at one stage does not guarantee its persistence throughout the entire AI life cycle.
Source: NHS England “Creating an international approach to AI for healthcare“[9]
Addressing bias in AI models through technical code or software is just the beginning for AI governance. Companies must extend their focus beyond the technical developers’ Proof Of Value. AI fairness auditing implemented across the entire AI life cycle along with comprehensive documentation is key. The documentation should incorporate various AI governance tools understandable to both technical and non-technical audiences.
Technical developers are focused on achieving explainability by describing the mechanisms of an AI system or algorithm through software/code. One of the major problems for evaluating explainability from the technical developer’s point of view is oversimplification.
In a typical scenario, a data scientist may opt for SHAP or LIME to show how an output was determined for a single instance of a model input, such as a specific decision or prediction, rather than the entire model. In other words, both methods operate by approximating more intricate, non-linear models (often referred to as "black-box" models) with simpler linear models, potentially resulting in misleading outcomes.
Oversimplification and the potential lack of critical context poses a significant challenge. Some algorithms designed for specific social settings may be inaccurately applied in different contexts, resulting in technical inaccuracies and misguided outcomes. The NIST AI Risk Management Framework[12], examined in Part II of the report, acknowledges the shortcomings of metrics used to measure AI risk, citing the risk of oversimplification, gaming, lack of critical nuance, and unexpected reliance.
This deficiency in contextual understanding could lead to unfair resource allocation or access, with implications across various areas such as mortgage lending, employment screening, college admissions, child welfare, and medical diagnoses.
An AI governance framework should encompass internal governance structures and measures, define the extent of human involvement in AI decision-making, address operations management, facilitate stakeholder interaction and communication, and illustrate de-risking strategies for AI across the entire business to mitigate large-scale failures.
01
Internal governance structures and measures play a crucial role in ensuring robust oversight of an organization's use of AI. They should be addressing risks and integrating ethical considerations through ethics review boards.
Organizations may explore features for their internal governance structures, such as clear roles and responsibilities for ethically deploying AI. A decentralized governance mechanism might be considered when a centralized approach is suboptimal, bringing ethical considerations into day-to-day decision-making.
The involvement and support of top management and the board of directors are pivotal. Key tasks include defining roles, responsibilities, and training for personnel involved in AI governance, using risk management frameworks for assessing and managing risks, and establishing monitoring and reporting systems. Regular reviews ensure the continued relevance and effectiveness of internal governance structures.
02
Before deploying AI solutions, organizations are advised to determine how much influence humans have in the process. The levels addressing human oversight with AI are listed below:
Human-in-the-loop: In this model, human oversight is active, with people retaining full control. The AI provides recommendations or input to the humans driving the process.
Human-out-of-the-loop: In this model, there is no human oversight, and the AI system has full control without the option of human override.
Human-over-the-loop (or human-on-the-loop): This model involves human oversight to the extent that the human is in a supervisory role, with the ability to take control in the face of unexpected events. Humans can adjust parameters during the algorithm's operation. Examples include: AI assisted medical diagnoses, product recommendations, or GPS navigation systems.
03
The individuals involved in model training and selection, whether internal staff or external providers, should work collaboratively. Data accountability practices, including understanding data lineage, ensuring data quality, and minimizing inherent bias, are essential. Organizations must understand the lineage of data and address factors affecting data quality.
Minimizing inherent bias involves being aware of biases in datasets, using heterogeneous datasets, and employing different datasets for training, testing, and validation. Regular review and updating of datasets, even if non-personal, are recommended for accuracy, quality, and reliability. Good data accountability practices apply even when using non-personal data or anonymized personal data in AI model training.
In deploying AI algorithms, organizations must iterate through model development until achieving the most suitable results for their use case. The interaction between data and algorithms/models is vital. Datasets, sourced from various places, both personal and non-personal, are integral to the AI solution's success.
04
Effective AI governance relies on clear end-to-end communication with various stakeholders such as developers, executives, regulators, external AI tool customers, internal business users, and more. Achieving this involves concise and accessible AI documentation, addressing model gaps and biases, and specifying appropriate use cases.
Transparent communication, simple user interfaces, opt-out mechanisms, and feedback channels are paramount for user understanding and interaction. Organizations should regularly assess their AI governance against evolving ethical standards and share the results with relevant stakeholders. Meta's "Why am I seeing this?"[13] feature exemplifies transparency in advertising and machine learning model training. As AI progresses, AI governance evolves. WTW will continue to keep pace with that change and explore collaboration amongst employees, executives, users, and researchers on examining and enhancing AI for optimal governance.
