Governing AI APIs that can change production systems

Executive summary

AI risk is usually reduced to model behaviour. That is too narrow. A production AI capability is an assembly — identities, prompts, retrieval data, model endpoints; libraries, tools, logs, and everywhere its output can land. The attack surface is in the relationships between those parts.

NIST AI 600-1 provides a cross-sector profile for generative-AI risk. NIST's Cyber AI Profile is currently preliminary and is organized around securing AI components, using AI for defense, and defending against AI-enabled attacks. This paper translates those directions into a system architecture centered on six controls:

1. Complete AI use-case and dependency inventory.
2. Separate identities for reading, proposing, and acting.
3. Untrusted-content treatment for prompts and retrieval.
4. Schema and destination boundaries around every tool.
5. Evidence and approval outside the model.
6. Revocation, reconstruction, and containment designed in advance.

Threat context for this paper is in Whitepaper 02.

1. The system is bigger than the model

An AI system record should include:

• Business use case, owner, users, affected decisions, and failure consequence.
• Model provider, model, and version policy, hosting mode, region; contractual terms, retention, and training-use settings.
• Prompt templates, system instructions, retrieval sources, vector stores; memory, fine-tuning data, evaluations, and output destinations.
• User identities, application identities, API keys, roles; delegated access, tenant boundaries, and emergency access.
• Orchestration frameworks, packages, containers, plugins; tool servers, browser surfaces, and build pipeline.
• Every tool's permitted operations, destination allowlist, transaction limit, approval condition, and audit output.
• Monitoring, incident contacts, stop mechanism, credential-revocation path, and evidence-retention rule.

Inventory quality should be measured. An unknown owner, an unknown retention setting, an unknown tool permission — each is a control gap. Treat a blank field as a finding.

2. Threat model

2.1 Instruction injection

Untrusted text can try to override system intent, extract context, redirect tools or suppress safeguards. It can arrive almost anywhere the system reads — a web page, a ticket, a source repository, even text inside an image.

2.2 Excessive agency

A broadly privileged tool turns an incorrect or manipulated model output into a side effect. High-risk effects include sending externally, changing access, deleting state, publishing; purchasing, moving funds, executing code, or touching customer systems.

2.3 Data disclosure and cross-boundary mixing

Data leaves its tenant through the obvious routes — prompts, retrieval, logs — and the less visible one: the vendor's support channel. Output can also give up system prompts, secrets, or personal data.

2.4 Identity confusion

An agent can act under the wrong customer, subscription, operator, or service identity. Shared sessions and ambient credentials make it difficult to attribute or revoke the action.

2.5 Software and model supply chain

Libraries, containers, model files; plugins, tools, and update channels can be malicious, compromised, stale, or incompatible. A model name without a version policy reduces reproducibility.

2.6 Unreliable inference

Confident output can be unsupported, incomplete, stale, or sensitive to small input changes. For security work, an invented asset, vulnerability, scope rule, or remediation state can be operationally dangerous.

3. Reference control architecture

3.1 Provider-neutral capability boundary

The agent should call a stable capability interface rather than receive a shell and a repository path. Each capability declares input schema, output schema, maturity; side effects, provenance, and execution boundary. Provider updates can change the conversational shell without bypassing the control plane.

3.2 Every run carries its own identity

Every run binds operator, provider account, customer or project, task, and authorization context. Credentials are selected from that identity rather than inherited from an ambient environment. Mismatch fails before a request leaves the host.

3.3 Read, propose, act separation

Read-only discovery produces evidence. A proposal produces a reviewable plan or draft. State-changing action requires a separate capability and gate. The most privileged step should not share credentials or automatic retry behavior with the least privileged step.

3.4 Untrusted content boundary

Retrieved content cannot grant itself authority. The orchestrator strips or isolates instructions from data, labels source and trust, constrains context, and validates tool arguments against external policy. The model never decides whether the content is allowed to expand scope.

3.5 Network and destination control

Network tools enforce exact-origin or destination allowlists, resolve hostnames under policy, bind resolved addresses to pacing, revalidate redirects, limit response bodies, reject credentials in URLs, and audit direct egress paths.

3.6 Evidence store

Results land in a normalized store recording the asset, source, time, and model or tool version. A model's summary can help you find the underlying record. It is not the record.

4. Write gates for tool-enabled agents

A write gate should verify facts outside the model:

1. The caller identity and active project match.
2. The requested target is in the current written scope.
3. The capability is allowed for the role and current task.
4. Required deduplication or conflict checks passed.
5. The payload matches an exact schema and size ceiling.
6. Secrets are absent from fields that will be logged or returned.
7. The destination, recipient, or resource is fixed or allowlisted.
8. The operator confirms one action; future actions need a new confirmation.
9. The result can be reconciled through a receipt or state readback.

Retry logic must understand side effects. A timeout after submission is ambiguous; repeating the action can create duplicate messages, tickets, payments, changes, or reports. Idempotency keys, receipts, and read-after-write checks are control requirements.

5. Provenance and evidence

A useful AI-assisted security record identifies:

• Provider, model, and relevant version or alias at execution time.
• Capability code identity and manifest.
• Prompt or instruction template version.
• Source records and retrieval references used.
• Tool arguments after validation and redaction.
• Authorization, approval, and policy decision.
• Tool result, receipt, and resulting state where applicable.
• Analyst review and corrections.

...and set retention limits by data class.

6. Evaluation program

Evaluation should test the assembled system, not only the base model:

• Direct and indirect prompt injection from every retrieval source.
• Cross-tenant and wrong-project identity attempts.
• Tool-argument smuggling, duplicate keys, oversized values, unusual Unicode spacing or separator characters, and schema confusion.
• Redirect, DNS rebinding, private-address, and response-size behavior for network tools.
• Ambiguous write results, retries, stale receipts, and duplicate action prevention.
• Secret leakage through output, error, trace, summary, and report fields.
• Revocation during a running job and cleanup of descendants.
• Unsupported claims, missing sources, stale context, and disagreement between models or tools.

Each control needs two tests: one that passes, and one that changes a single authorisation or input and proves the control was doing the work. Passing a happy path is not evidence that the boundary is load-bearing.

7. Incident response for AI systems

1. Stop: disable the agent route and revoke tool credentials, then stop queued work and isolate affected connectors.
2. Preserve: retain bounded prompts and sources; tool calls and approvals; receipts and state changes, without spreading secrets.
3. Scope: identify users and projects; models and tools; destinations and the data classes involved.
4. Reconcile: determine which proposed actions became real side effects and which remained drafts.
5. Correct: fix policy and identity controls; parser and prompt handling; or the tool and workflow control. A wording change alone cannot close the issue.
6. Verify: rerun the failing control and mutants, review adjacent paths, rotate affected secrets, and monitor recurrence.

8. Governance metrics

• AI use cases with named owner, approved provider, data classification, and dependency inventory.
• Tool-enabled workflows with separate read and write identities.
• High-impact actions protected by deterministic gates and readback.
• Retrieval sources covered by injection tests.
• Capabilities with explicit input, output, side effect, provenance, and maturity contracts.
• Percentage of AI-supported security claims linked to independent evidence.
• Time to revoke all credentials for one agent or project.
• Time to reconstruct state-changing actions after an incident.
• Evaluation coverage by threat class and last successful negative control.

9. How KernelHawk does this

KernelHawk separates read-only analysis from actions that change systems. High-impact actions require a named owner, narrow permissions, explicit approval, and a record of the result.

Provider changes must not silently change who can act, what they can change, or which evidence is retained.

Conclusion

AI security becomes manageable when the model sits inside an ordinary security architecture — inventory, identity, least privilege, gated writes, and rehearsed response. Nothing about the interface is new enough to change that. What is new is how quickly a mistake becomes a side effect.

Edition history: September 1, 2026, version 1.1. September 8, 2026, version 1.2: NIST links added; capability boundaries, write gates, and evidence controls revised.