Third-party AI assurance · evidence practice
Vendor AI Risk Evidence
Vendor AI risk evidence is the documentation, test results, contractual commitments, system records, and review decisions used to assess claims about a third-party AI product for a defined use. The goal is not document volume. It is enough relevant evidence to proceed, restrict, defer, or decline a use—and to revisit that decision when the product changes.
For technology risk, AI governance, procurement, security, privacy, compliance, model governance, internal audit, and business owners reviewing external AI products and embedded AI features.
Answer first
What counts as vendor AI risk evidence?
Evidence is information that helps evaluate a defined claim about the vendor AI system you actually intend to use. A questionnaire answer can record a claim. A model card can describe intended behavior. A contract can create obligations. A test can show observed behavior. None of those artifacts automatically proves every other claim about the product.
The evidence file should preserve scope: vendor, product, model or service version where known, enabled features, configuration, data boundary, intended use, review date, source, limitations, and the claim each artifact supports.
The broader risk review determines which uses may proceed; this evidence file records what supports or limits those decisions.
- Supplier claim
- “Customer data is not used for training.” Useful as a statement to test, not as self-validating proof.
- Supporting evidence
- Plan-specific terms, data-flow documentation, configuration, subprocessors, retention terms, and relevant assurance records.
- Observed evidence
- Customer-side tests, logs, configuration captures, incident records, or behavior observed in the purchased environment.
- Decision evidence
- The dated record connecting the claim, evidence reviewed, unresolved gaps, owner, conditions, and reassessment trigger.
Evidence file anatomy
Build the file around six questions.
A useful vendor AI evidence file makes the use, dependency, proof, and decision reconstructable. Increase depth as data sensitivity, permissions, consequence, opacity, operational dependency, or regulatory exposure increases.
What exactly is being approved?
Record the vendor, product, AI feature, relevant version or service state, enabled configuration, workflow, users, owner, and intended decision.
Useful records: use-case brief, service description, identifiers, enabled features, approved and excluded uses.
What enters the service, and what can the AI reach?
Document data inputs, retention, training use, subprocessors, user permissions, connected systems, and any tool or retrieval access.
Useful records: data-flow diagram, DPA, retention terms, training-use statement, subprocessors, permission map.
What does the supplier say the AI can and cannot do?
Capture intended behavior, known limitations, prohibited uses, failure modes, and conditions that constrain reliable operation.
Useful records: model card, transparency note, intended-use statement, limitation register, reviewer guidance.
What evidence shows the capability is adequate for this use?
Use evaluations and local observations that match the workflow, population, configuration, permissions, and decision being reviewed.
Useful records: evaluation results, pilot results, samples, access-control tests, output review, independent testing.
What could invalidate the earlier review?
Track material model, retrieval, data, feature, integration, upstream-provider, and incident changes that can alter the evidence basis.
Useful records: release notes, change notices, incident terms, impact assessment, rollback path, retest record.
Who owns the controls, and what can the organization enforce?
Make evidence rights, shared responsibilities, support obligations, escalation, export, deletion, contingency, and exit conditions explicit.
Useful records: contract clauses, responsibility matrix, support terms, export/deletion process, contingency plan.
Minimum review record: Claim and acceptance criterion · product, version, and use · evidence source and date · observed result · limitations and shared evidence roots · reviewer, decision, owner, and reassessment trigger. The Evidence Review Kit provides working fields for these records.
Evidence strength
Judge evidence by its connection to the claim.
These are five forms of evidence, not a universal ranking. Judge each by its directness to the claim, product and configuration fit, method, freshness, and independence. Higher-risk decisions often need testing or corroboration.
Assertion
A vendor statement, sales response, questionnaire answer, or policy summary records what is claimed. It identifies what needs support; it is not self-validating proof.
Documented support
Contract terms, technical documentation, model cards, architecture diagrams, and control descriptions add scope and specificity to the claim.
Observed evidence
Logs, configuration captures, release records, samples, incident records, and customer-side observations show what actually occurred.
Tested evidence
Relevant evaluations, control tests, pilot results, permission tests, and output testing directly challenge consequential claims.
Corroborated evidence
Relevant independent assurance, external validation, and separate tests can corroborate a claim. Reports derived from the same vendor test still share one evidence root.
Third-party audited does not automatically mean stronger. For a configuration claim, a dated customer-side capture may provide more direct evidence than a broad certification. Examine the claim covered and the underlying evidence, not just the report label.
Artifact limits
What does each vendor document actually prove?
Common vendor artifacts are useful when the claim matches their scope. Treat each artifact as evidence for a bounded purpose, not as a universal assurance stamp.
| Artifact | Can support | Does not establish by itself |
|---|---|---|
| Model card / transparency note | Intended use, model or feature description, evaluation summaries, known limitations, safety guidance. | How your specific configuration behaves, whether your controls operate, or whether contractual promises are enforceable. |
| SOC 2 / assurance report | Controls and testing within the report’s system boundary, period, criteria, exceptions, and auditor scope. | AI quality, fairness, factuality, suitability for your use case, or controls outside the examined boundary. |
| Vendor questionnaire | Structured supplier assertions, ownership statements, process descriptions, and pointers to supporting records. | Independent verification. Answers remain assertions until supported where the risk requires support. |
| Evaluation / benchmark | Observed results for the tested model, data, tasks, metrics, conditions, and period. | Performance on untested workflows, populations, versions, configurations, or future releases. |
| DPA / contract terms | Enforceable obligations when the executed agreement clearly covers the service and use. | Whether technical behavior actually conforms to the terms or whether local configuration creates additional exposure. |
| Customer pilot / local test | Observed behavior in a bounded customer workflow using selected data, settings, permissions, and evaluation criteria. | General performance outside the test boundary or behavior after a material vendor change. |
Evidence quality is claim-specific. Record what an artifact covers, what it excludes, how current it is, and whether another artifact depends on the same underlying analysis.
Worked review
Can a material AI change occur without a usable review window?
Consider the internal policy assistant in the broader vendor review. It retrieves company policies and returns source references. A material change to its model, retrieval, or permissions could undermine earlier tests. Can the organization assess that change before wider use?
- Request
- Executed terms, change-notification language, named notice channel, release history, version policy, and examples of prior notices.
- Check scope
- Does “material change” include upstream model substitutions, retrieval changes, permission changes, data-use changes, and features enabled by default?
- Check timing
- Is notice provided before deployment, after deployment, or only when the vendor decides customer action is required?
- Check actionability
- Can the customer delay adoption, disable the feature, restrict affected workflows, retest, or roll back?
- Gap
- Notice is not the same as a review window. Even a contractual notice obligation may not cover upstream changes, give sufficient lead time, or identify someone authorized to suspend use.
- Required record
- Keep the applicable notice clause, named recipient and decision owner, material-change criteria, change log, test plan, and a method to pause or limit affected functions.
Example decision
Restrict to a non-sensitive pilot; defer production use. Before expansion, confirm which changes require advance notice, who receives it, and how the team can restrict or retest affected workflows. Reassess when the model, retrieval, permissions, or data handling changes materially.
Primary sources
Use current guidance within its scope.
The sources below support the third-party-risk, AI supply-chain, documentation, change, monitoring, and model-risk concepts used here. They do not create one universal vendor-evidence checklist for every organization.
Sources and regulatory context (7)
The 2023 U.S. interagency third-party risk guidance remains the existing general guidance. Federal banking agencies issued a proposed replacement on September 11, 2026; it is not final guidance as of this review.
- U.S. banking · current
Interagency Guidance on Third-Party Relationships: Risk Management
Federal Reserve / FDIC / OCC, 2023. Covers risk-based third-party relationship management across planning, due diligence, contracting, monitoring, and termination.
- U.S. banking · proposed
Proposed Third-Party Risk Management Guidance
OCC with FRB, FDIC, NCUA, Sept. 11, 2026. Proposed replacement guidance emphasizes tailoring third-party risk management to reasonably assessed risk.
- SOC 2 · criteria
AICPA Trust Services Criteria (revised points of focus, 2022)
Criteria for controls over security, availability, processing integrity, confidentiality, and privacy. Review the scope and tests of the actual SOC 2 report; the criteria are not a direct evaluation of AI factual accuracy or suitability.
- NIST AI RMF
NIST AI RMF 1.0 — GOVERN 6
Addresses AI risks arising from third-party software, data, and supply-chain issues, including contingency processes for failures or incidents.
- Generative AI
NIST AI 600-1: Generative AI Profile
2024. Extends AI RMF practices for generative AI, including third-party assessment, supplier agreements, monitoring, incident coordination, and supply-chain considerations.
- Model risk · scope
OCC Bulletin 2026-13 — Revised Model Risk Management Guidance
Discusses vendor and other third-party products for in-scope models while excluding generative AI and agentic AI from that guidance’s scope.
- EU · where applicable
EU AI Act Article 25 — Responsibilities along the AI value chain
Use with related provider/deployer provisions. Applicability depends on the system, organizational role, and facts.
Evidence depth should be proportionate to the use, data, permissions, consequence, criticality, opacity, and dependency being reviewed. Apply the laws, policies, supervisory expectations, contractual obligations, and technical criteria that actually govern the organization and use case.