Essay · Nº 015

Assume the Injection Will Work

At Guidewire Connections in Las Vegas this week, Viyat Bhalodia — who leads security work for Codex and the agentic products built on it — told insurers moving agents from pilot to production to assume prompt injection attacks will sometimes succeed, and build enforced controls around the agent anyway. "You do not want to rely on just one single layer of software to protect you," he said, "because if that fails, then you are completely vulnerable."

The framing case was a claims agent reading a police report, medical records, or a repair estimate — any of which can carry a hidden instruction. OpenAI trains its models to resist that, Bhalodia said, but resistance is not containment. He argued for deterministic controls on top of the model: a sandbox limiting which files the agent can open and which destinations it can reach, least-privilege access with expiring credentials, and constraints on actions that cannot be undone.

He called the combination the "lethal trifecta": sensitive data, untrusted content, and external network access together. Guidewire's Ben Chang, sharing the session, noted ordinary claims handling assembles all three at once. Claims is not an edge case for agent security; it is the case where the attack surface is the workflow itself.

What it means for carriers

First, the human-approval safety net has a hole in it, and both speakers named it. Chang described answering "yes, yes, yes" to his own coding agent's approval prompts, and asked how to avoid "cognitively surrendering our responsibilities." Bhalodia warned too many prompts push users to switch protections off entirely: "you are basically babysitting the agent, not working with it." A human trained by the loop to click yes is not a control. If your governance case rests on "a person approves each step," you have described the failure mode, not finished the design.

Second, logging everything is not the answer either. Logs accumulate sensitive information, Bhalodia cautioned, and persistent agent memory can carry data across tasks. Raw telemetry and an activity record, he said, would not on their own make an agent's behaviour explainable to a regulator. What would: readable, consistently structured output, with references a reviewer can check against the policy or the source consulted. That is observability versus accountability: a tape nobody can read where the decision is defended is storage.

Third, not all agent actions are equal, and the governance has to say so in advance. Bhalodia drew the line between reversible and irreversible actions: restoring deleted data from backup is not the same problem as recovering a disclosed Social Security number. Payment functions, he said, should carry constraints preventing an agent from issuing payments it should not make. "And we have checks and balances on human employees," he said. "The same things apply to an agent." Its checks therefore have to be enforced by the system, not requested of the model.

From the operator's chair

One: take Chang's closing assignment: find your CISO and map which of your agents combine sensitive data, untrusted content, and external connectivity. Every agent on that map gets containment before scale — no exceptions for the pilot that "already works," because nobody was trying to break it.

Two: build the cage before you widen the mandate. Sandbox the agent to the files and destinations its task needs. Give it expiring credentials instead of the long-lived access common in existing infrastructure. Put hard constraints on payments and disclosures — the irreversible actions — so the agent cannot do them alone no matter what a document tells it to do. These hold even when the model is fooled.

Three: fix the approval design, don't just add approvals. A second agent reviewing proposed actions against the user's request and task context — the "auto review" approach Bhalodia described in Codex and ChatGPT — is worth more than a tenth human click-through. Route flagged actions to a human with a clean summary and source references attached, and count how often that human stops something. If the stop rate is zero for a month, your approval step is theatre.

Four: write the output for the examiner, not the engineer. Every claim decision should produce a readable record: what the agent saw, which sources it relied on, what it recommended, and what the human confirmed. After this week, the question will come phrased as "show me" rather than "tell me" — the carrier that can produce that record answers in minutes, and the carrier with a telemetry dump answers in months, if at all.

The industry spent this year learning to give agents work. This session was about the other half: boundaries that survive contact with an adversary. A model that resists injection most of the time is a good product; a system that stays safe the time it doesn't is a governed operation. Carriers get judged on the second.

— Krishna