The model's output is an untrusted input

We are wary of what the user sends to the model. We often forget to be wary of what the model sends back. Yet its output may have been influenced by an injection, by a booby-trapped document, or may simply be wrong. As soon as it is displayed, executed or passed on to another system, it must be treated as input from a stranger.

The four classic paths

• Displayed as HTML. A response inserted as-is into a page may contain code that runs in the user's browser: this is an XSS flaw, known for twenty years, coming back through this door.
• Inserted into a database query, a system command, a file path: SQL or command injection, AI version.
• Rendered as Markdown with images. A well-documented attack: an injection pushes the model to write an image whose address points to the attacker's server, with data from the conversation in the address. The interface loads the image, and the data leaves, without a single click.
• Used as an action parameter: an amount, a recipient, an identifier that the application executes without checking.

The defences

• Systematically escape all text from the model before display, and go through a Markdown rendering engine configured restrictively.
• Block external resources in responses (images, automatically loaded content), or allow only a closed list of domains, ideally reinforced by a content security policy on the browser side.
• Validated structured outputs: when the model produces data for an action, require a strict schema and validate it in your code (types, bounds, allowed values).
• Parameterised queries and action allowlists, never model text concatenated into a command.
• A sandbox for any code the model writes and that you execute: with no network access or secrets by default.

Key points

• Model output is treated as input from a stranger.
• Four paths: HTML, queries and commands, Markdown images, action parameters.
• Escape, block external resources, validate by schema, parameterise, isolate.