- The word audit hides three different jobs. A conformity audit against a management standard, a risk assessment against a framework such as the NIST AI RMF, and a technical assurance audit that actually runs the system are not interchangeable, and each wants different evidence.
- An audit becomes tractable the moment you stop asking for assurances and start naming artefacts. Ask for the document, not the opinion.
- Most AI audits stall at step one, because no one can produce a complete list of the AI systems in use. Governance that starts anywhere else is decorative.
- NIST's framework is explicit that its actions "do not constitute a checklist, nor are they necessarily an ordered set of steps", so citing it as a pass or fail standard misreads it.
- The EU AI Act already names the records a high risk system has to keep, with Article 12 requiring automatic logging over the lifetime of the system, and those obligations bite from 2 December 2027 for Annex III use cases.
- Two questions do more work than any checklist: when was the evaluation last re-run and on what data, and who can change the production prompt without a code review.
What do you actually ask for? That is the question every controls person hits in the first hour of an AI audit, and the reason it is hard has nothing to do with machine learning. You know how to audit a payment process because you know what a payment leaves behind. An AI system leaves behind artefacts you have never requested, held by people who have never been audited, under names that mean something different inside the team than they do in your workpapers.
So this is not a conceptual piece about trustworthy AI. It is a list of things to put in an email, with the name of the document, the question it answers, and the person who is supposed to own it.
What does an AI audit actually mean?
Three different engagements wear the same word, and conflating them is how an audit ends up producing a report nobody can use. Decide which one you are running before you send a single request.
| Type | What it tests | What it produces | Who accepts the result |
|---|---|---|---|
| Conformity audit | Whether a management system meets a published standard, typically ISO/IEC 42001 for AI management systems | A certificate, issued by an accredited body, with a scope statement | Customers and procurement teams asking for a badge |
| Risk assessment | Whether identified risks have been governed, mapped, measured and managed against a framework | A risk register with owners, plus gaps against the framework's categories | Your own board and risk committee |
| Technical assurance | Whether the system behaves as documented when you run it yourself | Test results on data the team did not choose, with a date | Anyone who has to defend a decision the system made |
The second row deserves care, because it is the one most often misquoted. The NIST AI Risk Management Framework Core is built from four functions named govern, map, measure and manage, each broken into categories and subcategories. Govern alone carries six categories and 19 subcategories covering policies, accountability, organisational culture and third party risk, and NIST describes it as a cross cutting function infused throughout the rest.
What the same page says next is the sentence auditors skip: the actions "do not constitute a checklist, nor are they necessarily an ordered set of steps", and organisations may select from among the categories. NIST released AI RMF 1.0 on 26 January 2023 for voluntary use, catalogued as NIST AI 100-1, and it is currently being revised under the White House AI Action Plan. A supplier claiming to be NIST compliant is therefore claiming something the document does not offer. Ask which subcategories they selected, and why they left the others out.
Why does the inventory step sink most of these?
Because nobody can list the systems. Not out of obstruction, but because the question is genuinely hard once AI stopped arriving as a project and started arriving as a feature of software you already bought. The model inside your help desk tool was switched on by the vendor. The summariser in your meeting recorder was enabled by default. A team somewhere is calling an API with a company card.
An inventory built from the project register will find the first category and miss the other two entirely, which is why we argued that governance without an inventory is theatre. The practical fix is to triangulate: the project register, the expense ledger filtered for AI vendors, and an interview question asked of every team lead about which of their tools gained a new assistant this year. Each source finds systems the others miss.
Record the completeness of the inventory itself as a finding. An audit that silently scoped to the three systems someone remembered is worse than no audit, because it produces a clean report over an unknown population.
What should the evidence request actually say?
Name the artefact. This is the table to paste into the email, with the right hand column filled in before you send it, so that an unanswered row is visibly somebody's unanswered row rather than a general absence.
| Audit question | Artefact that answers it | Who normally produces it |
|---|---|---|
| What is this system for, and what is it not for | Model card or system card, with an intended use section that names excluded uses | The team that shipped it |
| Where did the training or tuning data come from | Data provenance record, with licence terms and the consent basis for personal data | Data engineering, with legal countersigning |
| How well does it work | Evaluation results on a held out set, carrying the date it was run and the commit it was run against | The modelling owner |
| Does it work equally well for everyone | Disaggregated performance by subgroup, not a single headline accuracy | The modelling owner |
| Who checks the output | Human oversight design: who reviews what, with what authority to override, under what service level | The operations owner |
| Can you reconstruct a past decision | Logs tying an output to its input, its model version and its prompt version, with a stated retention period | Platform or SRE |
| What has gone wrong before | Incident register entries, including near misses and the rollbacks that followed | Operations |
| Whose model is it anyway | Third party model contract, naming the provider's terms on training with your inputs and on service changes | Procurement |
| How do you turn it off | Rollback plan naming the previous version, the trigger for using it, and who may pull it | Engineering |
| Who may change it in production | Change control record covering prompts and configuration, not only code | Engineering |
Two rows in that table are doing most of the work, and they are the two an ML team will not have thought of as audit evidence. The evaluation row needs a date because an evaluation run against last quarter's data describes last quarter's system. The change control row needs to cover prompts because in most deployments the prompt is the behaviour, and in a surprising number of them it lives in a configuration field that any team member can edit without review.
On the fourth row, the original research is worth knowing by name. Mitchell and colleagues proposed model cards in October 2018, and their central argument was exactly this: a released model should carry benchmarked evaluation disaggregated across cultural, demographic or phenotypic groups, plus the context in which the model is meant to be used. A single accuracy figure is not an answer to the question you asked.
Which questions reliably surface trouble?
These five, asked verbally, in the room, before any document arrives. Each has a short true answer and a long evasive one, and the length of the answer is itself the finding.
- When was the evaluation last re-run, and against what data? If the answer is the launch evaluation, the system has been unmeasured for however long it has been live. If the data is the same set used to tune it, the number means less than it looks. Our note on a method rather than a benchmark for telling whether a change helped is the version of this question an engineering team will accept as reasonable.
- Who can change the production prompt, and is that change versioned? Ask to see the last change. If it cannot be shown, there is no change control over the system's behaviour regardless of what the code pipeline looks like.
- Show me a decision from three months ago and tell me which model version produced it. This single request tests logging, retention and model versioning at once. Most teams discover the gap while answering.
- What does your provider's contract say about your inputs? Third party dependency is the most under audited area because the system looks internal. The terms are not.
- What is the documented procedure for switching this off? Not whether one exists. Ask for the document and the name of the person authorised to invoke it.
Resist the temptation to accept a dashboard as evidence for the logging question. A dashboard shows aggregates and is usually built on sampled data with a short retention. The question is whether a single historic decision can be reconstructed, which is a different technical property.
Where does the law already tell you what to keep?
More precisely than most internal policies do. Article 12 of Regulation (EU) 2024/1689 requires that high risk AI systems technically allow the automatic recording of events over the lifetime of the system, and it names the purposes those logs must serve: identifying situations where the system may present a risk or undergo a substantial modification, supporting post market monitoring under Article 72, and monitoring operation under Article 26(5). For remote biometric identification under Annex III point 1(a) it goes further, requiring at minimum the start and end time of each use, the reference database checked, the input data that produced a match, and the people who verified the result.
The surrounding obligations are set out on the Commission's own regulatory framework page: adequate risk assessment and mitigation, logging for traceability, detailed documentation sufficient for authorities to assess compliance, high quality datasets, clear information to the deployer, plus appropriate human oversight measures. The timeline matters as much as the list. Prohibitions applied from 2 February 2025, governance and general purpose model obligations from 2 August 2025, with the Act becoming applicable on 2 August 2026. High risk obligations under Annex III apply from 2 December 2027, and for systems embedded in regulated products from 2 August 2028.
Even where none of this binds you, the list is free. A regulator has already done the work of deciding what evidence a serious AI deployment produces, and an internal audit that asks for the same artefacts is asking for things that will exist anyway. We mapped the tiers and the deadlines in more detail in our obligation by obligation map of AI Act compliance, and the shorter decision tree version of the tiers and timeline is the one to hand a product owner.
What can a certificate not tell you?
Whether the system works. This is the gap that catches organisations who commissioned the wrong audit type and believed they had covered the risk.
A conformity audit against a management standard examines whether you have processes, whether you follow them and whether you review them. It is genuinely valuable and it is what a customer's procurement team usually wants. It says nothing about the error rate of the model, nothing about which subgroup it fails for, and nothing about whether yesterday's deployment changed the behaviour. An organisation can hold a valid certificate while running a system that has not been evaluated since launch, because keeping an evaluation schedule is a process question and the evaluation result is not.
The inverse is also true and less often said. A technical assurance audit that reproduces the vendor's numbers tells you nothing about whether anyone will notice when those numbers drift. Pairing the two is the only honest coverage, and the cheapest version of that pairing is a process certificate plus one re-run evaluation per quarter on data the team did not pick.
What if the whole system belongs to a vendor?
Then your evidence list shortens and your contract becomes the audit. You cannot request a held out evaluation from a provider who will not give you one, and pretending otherwise produces a workpaper full of marketing claims. What you can request is narrower and still useful: the published model or system card, the documented intended use with its exclusions, the provider's own statement on what it does with your inputs, the change notification terms, and the service level attached to availability rather than to quality.
Three substitutions make a vendor system auditable without the provider's cooperation. First, test the output rather than the model, by running a fixed set of your own cases through it each quarter and keeping the answers. That is a real evaluation of the thing you actually depend on, and it belongs to you. Second, audit the human control around it, since the oversight design, the override rate and the escalation path are entirely yours. Third, audit the exit, because the question of how long it would take to stop using this and what you would lose is answerable today and will not be answerable in a crisis.
Record the dependency itself as a risk with an owner. The common failure is to treat a purchased capability as out of scope because no one internal built it, which removes from the register the system with the least visibility and the fewest controls. The register should be heavier on bought AI than on built AI, not lighter.
How much testing is enough?
Less than people fear, as long as it is fixed and dated. A standing set of fifty to a hundred representative cases, written once with the correct answer agreed by someone who is not the model owner, re-run each quarter, produces a trend line that is worth more than a single large evaluation nobody repeats. The value is in the repetition and in the fact that you chose the cases, because a system drifting away from the task will show up as a falling score on cases that never changed.
What good looks like at the end
A report where every row of the evidence table has either a document reference or a named gap with an owner and a date. Not a maturity score. Scores compress exactly the information an audit exists to surface, and a system with an excellent score and no retained logs is a system you cannot defend.
Two habits make the next audit cheaper. Keep the inventory as a living list rather than a point in time artefact, so the first week of the next engagement is not spent rediscovering it. And push the logging requirement into the build stage, because retrofitting traceability onto a live system is the single most expensive remediation on the list. What belongs in those records is a design decision rather than an afterthought, which is why the trace that explains a bad answer is worth settling before the system ships rather than after an auditor asks.
If you are writing the policy that sits above all this, our AI policy page sets out the commitments we hold ourselves to, and the security page covers the control side. Neither is a template for your organisation, but both show the shape of the document an auditor expects to be handed on day one.