Reading AI Evaluation Reports: A Practitioner’s Filter for the New Procurement Reality

AI evaluation reports are on their way to your procurement inbox. They will not all look the same. Some will be vendor-produced safety summaries with a few benchmark numbers and a glossy cover. Others will be third-party assessments commissioned by the vendor and run against pre-agreed criteria. A smaller set will be independent red-teaming write-ups that the vendor did not control. Each AI evaluation report tells you something different, and each is useful for a different question. Procurement leads, AI champions and compliance officers all need to read them. Most of the existing templates do not flag what to look for. This is the practitioner filter: what to require, what to read, what to ignore.

What an AI evaluation report actually is

An AI evaluation report is not a compliance certificate, an audit opinion or a sales deck, although it often gets confused with all three. It is a structured account of what was tested, against what criteria, with what result. Sources vary: the vendor, a contracted third party, a government body such as the Center for AI Standards and Innovation or an academic red team. The format varies; the reading discipline does not. Always look first at what was tested, then at what was claimed and then at the gap between the two. That gap is where most of the procurement risk lives.

Three types of evaluation evidence, three different jobs

The first practical move when reading an AI evaluation report is to know which type of evidence sits in front of you. Each answers a different question. Confusing them is the most common error in vendor reviews this year.

Vendor-supplied tests

The vendor ran its own tests, designed its own benchmarks and chose what to include. This is useful evidence about what the vendor wants you to see. It is rarely sufficient evidence for procurement on its own. Treat vendor-supplied tests as a starting point: the questions they answer and the questions they conspicuously avoid both inform follow-up due diligence. When the only safety evidence is a self-test against vendor-selected benchmarks, the vendor due diligence question set needs to ask why.

Third-party evaluations

A contracted external party ran the tests, but the vendor commissioned and scoped the work. The methodology is usually better than self-testing. Scope, though, was negotiated. Read the methodology section carefully: what did the third party assess, what did they not, and what were the agreed limits of their access? A third-party evaluation that did not access the production model, or that excluded high-risk use cases, is useful but partial. It is not equivalent to independent assurance.

Independent red-teaming

A red team with adversarial intent and no commercial relationship probed the model for failure modes. The vendor did not control the scope, the criteria or the disclosure. This is the most useful evidence of the three, and the rarest. CAISI’s published evaluations, academic red-team write-ups and post-incident security research all fall in this category. When you see one, read the limitations section first: even independent red-teams operate under access constraints, and those constraints define the evidence value.

The reading checklist that separates substance from theatre

When the report lands, four questions decide whether it tells you anything useful.

• What was tested? Specifically: which model version, on which infrastructure, against which inputs. A general claim about “the model family” is not the same as evidence about the version you would deploy.
• What did the vendor claim? Read the executive summary against the methodology. Claims made in marketing language that the methodology cannot support are common.
• What did the report actually test? Compare the claim to the test design. The two often do not match, even in good-faith reports.
• What was neither claimed nor tested? This is the most useful column. A report that quietly does not cover your use case has told you something important.

Build these four questions into your AI vendor intake template. Returning them to the vendor in writing turns a passive document into an evidence trail.

Where AI Act conformity routes fit in

Under the EU AI Act, high-risk AI systems must pass a conformity assessment before they are placed on the market. Two routes exist, and they produce different kinds of evidence. As a deployer, knowing which route a system has followed tells you how much external scrutiny is in the AI evaluation report you are reading.

Annex VI: internal control

Most high-risk systems follow Annex VI, the internal-control route. The provider assesses conformity itself, applies harmonised standards where they exist and produces the technical documentation. There is no external evaluator. As a deployer, an Annex VI declaration tells you the provider believes the system meets the requirements. It does not, by itself, tell you a third party agreed.

Annex VII: third-party assessment

A narrower category of high-risk systems, primarily biometric identification, must follow Annex VII, which requires notified-body involvement. The notified body reviews the quality management system, the technical documentation and the conformity assessment itself. This is closer to independent assurance, although the notified body is still appointed by the provider. The AI Act text sets out which systems need which route. As a procurement input, an Annex VII path is a meaningfully stronger signal than Annex VI.

When the evidence expires: re-evaluation triggers

An AI evaluation report is a snapshot. Four events should trigger you to refresh it.

Model updates are the most obvious trigger. Vendors push updates to deployed models silently and often. A fine-tune, a system-prompt change or a new tool-use capability can shift the safety profile without external notice. Your contract should require notification. The model provenance question set is the practical place this lives.

Scope change on your side has the same effect. If you start using the model for a use case the evaluation did not cover, the original report no longer answers your question.

Incidents, whether at your organisation or at another deployer, reset the evidence base. Vendor responses to incidents are themselves procurement evidence: what changed, how was it tested and when can you see the new report?

Regulator requests are the fourth trigger. By 2027, EU market surveillance authorities will be requesting evaluation documentation as part of routine oversight. The pattern of government-to-provider data demands is already established outside the EU. Receiving a request and discovering that your AI evaluation report is two model versions out of date is a poor day.

Where AI evaluation reports live in your AI inventory

Treat AI evaluation reports as a structured field in your AI inventory, not as PDF attachments in a shared drive. Each entry should carry the report type (vendor, third-party, independent), the model version tested and the date. It should also carry the AI Act conformity route, the next re-evaluation trigger and the named owner who reads it. This is the level of structure that holds up under a regulator request, an audit or a board question about AI assurance. The NIST AI RMF Generative AI Profile sets out a comparable structure for managing model-related risk through the lifecycle, and is the cleanest external reference for organisations that have not yet built their own.

An AI evaluation report is not new paperwork to file. It is a procurement input that, read properly, separates vendors that can defend their work from vendors that can only describe it.

Audit the last three vendor packages you received. Apply the four-question checklist to each. Those that survive are worth keeping in your roadmap. The ones that do not have just told you what to write into the next vendor agreement.