How to Audit AI

An AI audit often starts with a simple request: a list of all AI tools currently in use across the organization. Management provides a list of tools that went through the formal approval and implementation process. But when the audit team reconciles that list against procurement records, accounts payable data, and software monitoring tools, the lists do not match. Employees are using AI tools that IT does not know about, and protected data and internal documents are being uploaded to these tools without restriction.

This situation is increasingly common in today’s cybersecurity landscape. The gap between the AI tools an organization knows about and those that are being used is often where the audit begins. Given the prevalence and accessibility of AI tools, inquiry alone is not enough to define the scope. That means auditors need to look beyond the approved inventory and understand how each tool is being used in practice.

Testing and evidence gathering for AI are not fundamentally different from other Information Technology General Controls reviews, but there are some important distinctions. AI systems can change their behavior over time, often referred to as drift, and the access and data available to an AI tool can extend beyond a single system. The auditor’s job is to determine which tools are being used, whether they are internally developed and trained or provided by a third party, what data they can access, how they are being used, and how the organization is ensuring outputs are valid, fair, reliable, and compliant with applicable legal and regulatory requirements.

The Tools Nobody Approved

Shadow AI is AI operating outside the organization’s governance program. It typically takes one of two forms:

• Employees use SaaS tools that have not been approved by the organization 
• An approved tool or platform introduces new AI functionality through an update, but the new capability is enabled without management review or approval 

Neither situation is likely to appear in an inventory built only from approved projects, and IT may not know the tools are in use. Finding them requires looking beyond the inventory itself. Audit teams can start with the following checks: 

• Identity provider data, including OAuth grants and application authorizations
• Network, DNS, or cloud access security logs showing traffic to AI services
• Expense reports and card activity for individual subscriptions
• Browser extension inventories on managed devices
• Vendor release notes and renewal terms for platforms already in use

Risk-Rank What You Find

For each AI use case identified, audit should understand how the tool is being used and who owns it. That includes the data it relies on, the model or vendor behind it, the level of automation, where human review occurs, and what downstream systems or decisions could be affected if something goes wrong. From there, the auditor can rank the use case based on its potential impact.

Not every use case carries the same weight. AI tools that summarize internal notes or transcribe meetings still warrant oversight, but they generally carry less risk than tools that directly influence hiring or other customer and constituent facing outcomes, whether that’s underwriting and claims decisions for an insurer or grant and donor eligibility determinations for a nonprofit.

Determine the Audit Criteria First

A strong AI audit program should start with established industry frameworks. NIST’s AI Risk Management Framework organizes the work around Govern, Map, Measure, and Manage. ISO/IEC 42001 provides requirements for an AI management system that can be mapped to control objectives. The IIA’s Artificial Intelligence Auditing Framework addresses responsibilities through its Three Lines Model, and ISACA’s COBIT-based guidance puts AI within technology governance audit areas that are already well established. 

These frameworks give audit teams a basis for developing the audit program and evidence requests before fieldwork begins. Testing criteria should be specific, with defined metrics and documentation requirements. Each control should have a measurable test, a defined owner, and a clear pass-or-fail threshold.

Six Key Areas to Test

AI Implementations. Trace a sample AI implementation from the original business request through deployment. Review meeting minutes or other evidence that legal, compliance, business, management, and IT stakeholders were involved in the discussions. Confirm that accountability for the AI tool has been established. Document project tracking, approvals, testing, go-live, and ongoing monitoring controls.

The Data and the Model. Management should be able to explain where the data used by the tool came from, why it is appropriate for the use case, and how data quality, quantity, access, and permitted use of the data are controlled. Predictive AI systems may require independent validation, performance testing, and bias analysis. Generative systems require a different test set, including factual accuracy, source validation, privacy leakage, harmful content, and behavior outside the approved use.

The Human in the Loop. Organizations should have a defined review process for human validation of AI outputs. Audit should identify who performs the review, the criteria they use, what the output is validated against, what percentage of outputs are reviewed, and the pass rate. This gives the auditor a better view of both model performance and the effectiveness of the review process. An unusually high pass rate may also warrant a closer look at how consistently the review criteria are being applied. 

Security, Privacy, and Changes. Start with familiar technology controls such as authentication, authorization, change management, and cybersecurity, then extend testing to areas specific to AI, including prompt and output logging, sensitive data restrictions, and output validation. For generative AI, audit should also consider controls addressing prompt injection, data exfiltration, poisoned retrieval sources, internal document access, and unauthorized use of connected tools. Change testing should extend beyond code. A vendor model update, a new prompt, a different data source, or a shifted threshold can all change how the system functions. 

Third-Party Management. Management and monitoring of a third-party AI tool should not stop at the SOC report. A SOC report, vendor attestation, or a completed questionnaire can provide evidence, but auditors should also review the contract or vendor agreement. Look for provisions covering due diligence, data rights, model security, cybersecurity and privacy terms, update practices, incident notification, subcontractors, and regulatory cooperation.

Monitoring and Outcomes. Pre-deployment testing tells you how a system behaved in a controlled setting. Ongoing monitoring tells you what happens once employees begin relying on an AI tool. Audit should look at accuracy, drift, fairness, complaints, appeals, overrides, incidents, and outcomes. When outputs deviate from expected behavior, hallucinations occur, or the model begins to drift, management should be able to demonstrate that the issue was detected and addressed through established controls. 

Walkthroughs Alone Are Not Sufficient

AI governance can appear to be mature on paper but operate differently in day-to-day use. Inquiry alone is not enough. Auditors should pair walkthroughs with testing, where appropriate, to determine whether controls are operating effectively. 

Auditing AI does not require becoming a model developer. Auditors do, however, need enough technical and business knowledge to challenge the evidence and recognize when a risk requires deeper evaluation or additional expertise. Experience with ITGC audits already gives auditors a useful foundation in data security, cybersecurity, change management, monitoring, and incident response. Still, a high-impact underwriting model may need actuarial or data science validation. 

The auditor should be able to recognize the sophistication of the AI tool and its use case, then determine whether the controls in place appropriately address the risks involved. 

As that approach is translated into the audit plan, the following areas should be covered:

Four Questions Your Audit Committee Should Ask Before the Audit

• Do we know where AI is being used? That includes vendor features enabled within platforms that are already licensed, as well as tools employees have adopted on their own.
• Which AI-supported decisions carry the most risk? The audit scope should follow the people and systems affected by the inputs and outputs of those tools. 
• Can we reconstruct a decision made with an AI tool? Do we have the input data, model version, expected output, human review, and final action taken? Can we validate how the human decision compared with the AI-supported result? The goal is to understand how auditable the system really is.
• Can we verify what our vendors tell us? Contracts and agreements alone do not provide enough evidence. Can we validate the design and operating effectiveness of vendor controls, and do our own internal controls address any gaps or Complementary User Entity Controls (CUECs)?  

How Johnson Lambert Helps

Our Business Advisory team works with organizations in several areas:

• AI Governance Assessment: Evaluating programs against the NAIC Model Bulletin, NIST AI RMF, and ISO 42001.
• Risk-Based Scoping: Identifying which AI use cases carry real consumer, financial, and regulatory exposure.
• Independent Testing: Testing controls over data, validation, human oversight, vendor management, model security, AI monitoring, and reporting.

If AI is already influencing decisions at your company, the evidence trail should be in place before an examiner or regulator asks for it. Speak to a Johnson Lambert specialist about assessing the controls around your organization’s AI use.

Frequently Asked Questions About AI Audits

• How do you audit AI? An AI audit starts by identifying which tools are in use, including tools that may not appear on the organization’s approved inventory. From there, auditors can assess how each tool is used, what data it accesses, who owns it, where human review occurs, and what systems or decisions could be affected. The audit should then test whether the relevant controls are designed appropriately and operating as intended.
• What should be included in the scope of an AI audit? The scope should extend beyond formally approved AI tools. Auditors should consider internally developed models, third-party tools, AI features embedded within existing software, and tools employees may have adopted independently. The scope should also account for the data each tool can access, the level of automation, downstream impacts, and applicable legal or regulatory requirements.
• What is shadow AI, and how can auditors identify it? Shadow AI refers to AI tools or capabilities operating outside the organization’s governance program. Auditors can look for signs of unapproved AI use in identity provider data, network and cloud access logs, expense reports, browser extension inventories, and vendor release notes for software already in use.
• How should AI use cases be ranked by risk? An AI risk assessment should consider how the tool is being used, who or what may be affected, the data involved, the level of automation, the role of human review, and the potential impact of a failure. A meeting transcription tool, for example, generally presents a different level of risk than AI used in underwriting, pricing, claims, hiring, or other decisions with direct consequences.
• What AI controls should auditors test? An AI audit should examine controls across implementation, data and model management, human review, security and privacy, third-party oversight, and ongoing monitoring. Depending on the use case, testing may also address prompt and output logging, sensitive data restrictions, model validation, drift, output accuracy, and changes to prompts, models, data sources, or thresholds.
• How is auditing generative AI different from a traditional ITGC review? Many familiar ITGC concepts still apply, including authentication, authorization, change management, cybersecurity, and monitoring. AI introduces additional considerations because systems can change behavior over time and may have broader access to data or connected systems. Generative AI may also require testing for issues such as prompt injection, data exfiltration, privacy leakage, harmful content, and outputs outside the approved use.
• What frameworks can guide an AI governance audit? Several established frameworks can help organizations define audit criteria. The article points to the NIST AI Risk Management Framework, ISO/IEC 42001, the IIA’s Artificial Intelligence Auditing Framework, and ISACA’s COBIT-based guidance. These frameworks can help shape control objectives, testing criteria, evidence requests, and ownership before fieldwork begins.
• Why are walkthroughs alone not enough for an AI audit? Walkthroughs and inquiry can help auditors understand how an AI process is intended to work, but they do not demonstrate that controls are operating effectively. Auditors should perform testing where appropriate and review evidence that supports how the system performs in day-to-day use, including monitoring results, overrides, incidents, complaints, and other outcomes.