Don't Only Audit the AI

We spend a lot of time asking whether an AI model is safe.

Does it give dangerous answers? Can it be manipulated? Does it invent facts? Can it be trusted to follow instructions?

Those are real questions. They are no longer enough.

A model that can only produce text is one kind of risk. The same model connected to a database, an email account, a payment system or a code runner is something else. If it is allowed to change files, send messages, move money or execute commands, a bad answer can stop being an answer and become an action.

That means part of AI safety sits outside the AI.

AI safety cannot stop at the model. Once an AI is connected to tools and given authority to act, the system humans build around it becomes part of the safety problem.

One model failure, several possible consequences

A vulnerability in the pgAdmin 4 database-management software shows the distinction unusually clearly.

Versions 9.13 to 9.15 included an AI Assistant with a tool that could run SQL generated by a language model. The tool was intended to be safe for reading data. The generated query was placed inside a PostgreSQL read-only transaction so the assistant should not have been able to alter the database.

But the boundary was incomplete.

According to the National Vulnerability Database record for CVE-2026-12045, an attacker who could place malicious instructions inside database content that the assistant later read could use prompt injection to influence the model. The model could then produce a multi-statement SQL tool call.

The problem was not simply that the model could be manipulated. The surrounding tool accepted the generated SQL without first restricting it to one genuinely read-only statement. A crafted first command could end the read-only transaction. Later commands could then run outside the protection that was supposed to contain them.

What happened next depended on permissions.

If the database role used by pgAdmin had ordinary write access, the vulnerable chain could allow unauthorised data modification. If that role had much stronger PostgreSQL privileges, the possible consequence could extend to operating-system command execution on the database server.

Same manipulated model. Same bad generated query. Different authority. Different maximum harm.

That distinction matters.

The vulnerability record does not show that somebody actually used this chain against a victim in the wild. It establishes that the route existed and what it could do under different privilege levels.

The fix is just as important as the flaw. pgAdmin 4 version 9.16 did not solve the problem by proving that the language model could never again be prompt-injected. Instead, the software tightened the system around it. AI-generated SQL now has to pass validation before it reaches the database: one statement, from a limited set of permitted read-only forms. Transaction-control commands and data-changing operations are rejected before execution.

The model could still be wrong. The surrounding system became less willing to let that wrongness become an action.

Permissions are part of capability

This sounds obvious when described as database security. Software engineers have used ideas such as least privilege, read-only access and defence in depth for decades.

That is an important correction. Permissions are not a new problem invented by AI.

What changes with modern AI agents is the route between untrusted information and action.

A traditional program is usually written to follow defined paths: when this happens, execute that code. A language-model agent can interpret open-ended natural language, read external material, choose among tools and decide how to sequence them. The content it reads can be both information and, through prompt injection, an attempt to influence what it does next.

NIST’s work on agent systems therefore treats the tool layer as part of the thing that needs describing. It distinguishes read-only from write access, trusted from untrusted environments, reversible from persistent actions, and low-autonomy from more autonomous tool use.

These are not details added after the model has been declared safe. They help define what the complete system is capable of doing.

A model allowed to search the web has one consequence boundary. Give it write access to a company drive and the boundary moves. Give it permission to send email and it moves again. Connect it to financial transactions, production infrastructure or physical equipment and the same underlying mistake can reach much further.

The tool layer is growing

This is not only a theoretical architecture question.

The UK’s AI Security Institute examined 177,436 publicly observable tools built for AI agents using the Model Context Protocol between November 2024 and February 2026. It divided them into tools that perceive information, reason about it, or take actions that modify an external environment.

In the sampled usage, the share attributed to action tools rose from 27% to 65% over the period studied. Most were not weapons or critical infrastructure controls. Many did ordinary computer work such as editing files. But the dataset also contained higher-stakes capabilities, including financial transactions.

The important change is simple: AI systems are increasingly being built not merely to answer but to do.

Once that happens, measuring the quality of the answer is not enough. The safety question becomes what the system lets the answer cause.

Bad instructions can cross into real actions

The AI Security Institute has also tested what happens when tool-using agents are attacked directly.

In a large public red-team competition, researchers tested 22 frontier AI agents across 44 realistic deployment scenarios. Participants submitted 1.8 million prompt-injection attacks. More than 60,000 produced policy violations in the test environments, including unauthorised data access, illicit financial actions and regulatory non-compliance.

Those numbers should be read carefully. They are attacks submitted in a security competition, not 60,000 real people losing money or data. The study does not show that every agent is unsafe in normal use.

It does show why model-only thinking is incomplete.

A prompt injection is a model and interface problem. Whether it ends as strange text, a leaked file, a sent message or a transaction depends on the machinery connected to the model and the authority that machinery has been given.

AI does add some new difficulty

There is another trap here. If permissions, injection and access control already existed in ordinary software, it would be easy to conclude that this is simply old cybersecurity with an AI label attached.

That would also go too far.

NIST’s AI Risk Management Framework identifies several risks that are new or increased compared with traditional software. AI systems can depend heavily on training data and deployment context. Training changes can alter behaviour. Data, models and concepts can drift. Large pretrained systems can have harder-to-predict failure modes. They can also be more opaque and harder to test exhaustively than ordinary engineered software.

None of that means conventional software is predictable or safe. It means the behaviour being connected to the tool layer can itself be less directly specified and less stable than a handwritten rule engine.

That combination is the important one:

a system that can interpret open-ended inputs + a model that can fail in less predictable ways + tools that can change the outside world + permissions that determine how far those changes can reach.

The danger is produced by the combination.

A safer model can still be put in an unsafe system

This changes what we should mean when we say an AI system has been tested.

A model evaluation can tell us useful things about hallucination, harmful outputs, manipulation resistance and other behaviours. It cannot, by itself, tell us the consequence of connecting that model to a production database with broad credentials.

Nor can a good system design make model safety irrelevant. A tightly permissioned agent can still give false information. A human approval step can become routine and careless. Read-only access can still expose private material. Controls can fail, as the first pgAdmin safeguard did.

The point is not to replace model safety with system safety.

It is to stop pretending they are the same thing.

A serious audit of a tool-using AI needs to ask at least two different questions.

First: what can the model do wrong?

Then: what has the surrounding system made it possible for that wrongness to do?

That second question means looking at the tools, the credentials behind them, what data can enter the model context, whether consequential actions need approval, what gets logged, whether changes can be reversed, and how far a compromised action can travel.

The pgAdmin vulnerability provides a useful test because the model failure can be held roughly constant while the surrounding authority changes. With stronger privileges, the same unsafe output could reach further. With stricter validation, the same output could be stopped before execution.

That is the system doing safety work.

AI is often discussed as though the dangerous object lives inside the model: a set of weights that becomes too biased, too deceptive, too capable or too difficult to control.

Sometimes it will.

But an AI that cannot act and an AI that can act with powerful credentials are not the same safety problem, even when the model underneath them is identical.

Don’t only audit the AI. Audit the system it has been allowed to become part of.