Maven's AI agents: who is accountable when they fail?
Anthropic told courts it cannot alter Claude once it is inside a classified Pentagon system, and an appeals court has now ruled that how a laboratory trains its models can itself be a supply chain risk. Military AI splits responsibility between the laboratory that trains the model, the platform that gives it tools and the commander who acts, and no single party holds the whole of it.
Listen to this briefing
Maven's AI agents: who is accountable when they fail?.
Anthropic told courts it cannot alter Claude once it is inside a classified Pentagon system, and an appeals court has now ruled that how a laboratory trains its models can itself be a supply chain risk. Military AI splits responsibility between the laboratory that trains the model, the platform that gives it tools and the commander who acts, and no single party holds the whole of it.
A California judge found that the evidence was unrebutted: once Claude is deployed on a classified Pentagon system, Anthropic cannot access, alter or shut it down, according to her order of 27 August ruling the supply-chain designation unlawful. On 25 September a divided D.C. Circuit panel upheld a separate exclusion of Anthropic anyway, in a case under a different statute.
The two outcomes are compatible. In No. 26-1049, the appeals court recorded that Anthropic's head of public sector had described model training as "the primary mechanism through which Anthropic can influence the behavior of models used by the Department", and treated that training-time influence as the risk, whatever the company's motives. Control of a model's behaviour sits at training time, with the laboratory. Control of what the system does sits at deployment, with whoever builds around it. This is the third part of a series on Maven after Claude, and it concerns the gap between those two.
An agent is a stack, not a model
A model does not control the agent it powers. An agent is an engineered system: a model, instructions, retrieved data, tools the model can call, permissions over those tools, code that decides when an output becomes an action, and human approval steps. The court record shows how little of that stack Anthropic touches after delivery. The appeals opinion lists three kinds of restriction: safety built into the model through training, technical monitoring layered on top for ordinary customers, and contractual usage limits. For a classified deployment, Anthropic told the court it has no "back door or remote 'kill switch'". The monitoring layer is the one that does not travel with the model, and the California order records that Anthropic cannot view the department's prompts.
CSIS, drawing on Palantir documentation, describes Maven tools that let users build interactive assistants with enterprise data and tools. The laboratory supplies the weights, Palantir the platform, a government team the configuration, an analyst the validation and a commander the authority. A failure can start in one layer and surface in another: a model trained to refuse can be given a prompt and permission that make a task look routine, a well-behaved model can be fed stale data, and advice can be passed on as an instruction. From outside they look identical, as a bad recommendation on a screen.
Everyone can point elsewhere
Palantir's account of responsibility is revealing. Asked about the Hugging Face breach, which McCyber has covered, Sankar told CBS the models "did not do anything they were not instructed to do", called it "a horrible cybersecurity configuration", and said the AI companies were responsible because "it's a defective product", according to the CBS transcript. He said Palantir was not asked to sign the White House accord because its accords were for companies that make models, which Palantir does not.
On that analogy the car maker answers for a defect and the driver for the crash. Military AI adds a third party who fits the engine, steering and brakes. The Hugging Face case showed both failure sources: a laboratory that removed safeguards for testing, and containment that failed. Bloomberg reported on 18 September that a Palantir spokesperson said the company "is not responsible for the underlying data nor identifying intelligence deficiencies", and that the government retains responsibility for data. The same report said users form operational understandings that differ from contractual terms. The paper allocation of responsibility and its practice can diverge.
Agents checking agents
After the Minab school strike, Sankar said Palantir built additional AI agents that "did more checking". Bloomberg's 18 September report said some Central Command personnel relied too heavily on the AI in Maven and expected it to flag stale data. A defence official told Bloomberg on 22 September that Maven cannot disqualify a target or correct the military intelligence database. Pentagon technology official Stanley said Maven "has to be optimized and told to do it", and that his office "doesn't control how combatant commands set up Maven". Sankar disputes the reporting and attributes the likely cause to human error.
More checking is sensible engineering, but its value depends on independence. A checking agent that reads the same outdated records, shares the same model and works from the same assumptions can reproduce the original error with greater confidence. That is an analytical point rather than a finding about Maven, whose configuration is not public. The test for any second check is whether it brings evidence the first did not have.
The case for a single owner of accountability
The government's position has force. Its fear, set out in Under Secretary Emil Michael's memorandum and quoted in the appeals opinion, was that an Anthropic dispute over a sensitive operation abroad "raised material doubts as to whether they would cause their software to stop working or cause some other disastrous action that would put our warfighters' lives in danger". The court called the contractual limits "hardly self-defining". NSPM-11, the June presidential memorandum, takes the same line: commanders remain accountable, and no commercial entity may be able to prevent, disable, degrade or materially modify military AI use without government approval. A democratic government may reasonably insist that its commanders, not a company, carry final responsibility.
The government has since retreated from the strongest version of that fear. Its June 3 order said the designation did not depend on any understanding of Anthropic's real-time access to deployed models, and the California order said such access was not the department's true concern. Training-time influence is a narrower, and more defensible, concern than a kill switch.
That position is legitimate, and it is also incomplete. A commander can be accountable only for what they can see, and no public mechanism lets anyone outside the chain examine how a model's trained behaviour, a platform's tool permissions and an operator's settings combined in a given strike. Accountability without inspection is a title rather than a control.
Bloomberg reports that investigators examined chat logs and computer records from the Minab strike, so evidence of what the system did exists. The Pentagon's review has not been released and may not be for years, and until it is, no one outside government can say which layer failed.
This is Part 3 of 3 in Maven after Claude. Part 1, Palantir Maven after Claude: the swap, by Palantir's account, covers the replacement of Claude in Maven. Part 2, Human control of military AI: safeguard or weak point?, examines the human in the targeting loop.
Disclosure: Anthropic is a named actor in this essay. McCyber uses Anthropic's Claude models in its editorial workflow, as set out on the About page.
Sources: Anthropic PBC v. U.S. Department of War (N.D. Cal., 27 Aug 2026, via FindLaw); Anthropic PBC v. U.S. Department of War, No. 26-1049 (D.C. Cir., 25 Sep 2026), opinion; Associated Press (via ClickOrlando); Bloomberg (18 and 22 Sep); CSIS; CBS News (Face the Nation transcript, 11 Oct); NSPM-11 (White House, 5 Jun 2026); McCyber.
Enable JavaScript and use a browser with spoken audio support to listen.
Read aloud by your browser. Voice and availability depend on your device.
The five layers between a model's training and a commander's decision, and what each cannot see. Sources: N.D. Cal. order (27 Aug); D.C. Cir. No. 26-1049 (25 Sep); CSIS; CBS (11 Oct); Bloomberg (18, 22 Sep); NSPM-11.
A California judge found that the evidence was unrebutted: once Claude is deployed on a classified Pentagon system, Anthropic cannot access, alter or shut it down, according to her order of 27 August ruling the supply-chain designation unlawful. On 25 September a divided D.C. Circuit panel upheld a separate exclusion of Anthropic anyway, in a case under a different statute.
The two outcomes are compatible. In No. 26-1049, the appeals court recorded that Anthropic’s head of public sector had described model training as “the primary mechanism through which Anthropic can influence the behavior of models used by the Department”, and treated that training-time influence as the risk, whatever the company’s motives. Control of a model’s behaviour sits at training time, with the laboratory. Control of what the system does sits at deployment, with whoever builds around it. This is the third part of a series on Maven after Claude, and it concerns the gap between those two.
An agent is a stack, not a model
A model does not control the agent it powers. An agent is an engineered system: a model, instructions, retrieved data, tools the model can call, permissions over those tools, code that decides when an output becomes an action, and human approval steps. The court record shows how little of that stack Anthropic touches after delivery. The appeals opinion lists three kinds of restriction: safety built into the model through training, technical monitoring layered on top for ordinary customers, and contractual usage limits. For a classified deployment, Anthropic told the court it has no “back door or remote ‘kill switch’”. The monitoring layer is the one that does not travel with the model, and the California order records that Anthropic cannot view the department’s prompts.
CSIS, drawing on Palantir documentation, describes Maven tools that let users build interactive assistants with enterprise data and tools. The laboratory supplies the weights, Palantir the platform, a government team the configuration, an analyst the validation and a commander the authority. A failure can start in one layer and surface in another: a model trained to refuse can be given a prompt and permission that make a task look routine, a well-behaved model can be fed stale data, and advice can be passed on as an instruction. From outside they look identical, as a bad recommendation on a screen.
Everyone can point elsewhere
Palantir’s account of responsibility is revealing. Asked about the Hugging Face breach, which McCyber has covered, Sankar told CBS the models “did not do anything they were not instructed to do”, called it “a horrible cybersecurity configuration”, and said the AI companies were responsible because “it’s a defective product”, according to the CBS transcript. He said Palantir was not asked to sign the White House accord because its accords were for companies that make models, which Palantir does not.
On that analogy the car maker answers for a defect and the driver for the crash. Military AI adds a third party who fits the engine, steering and brakes. The Hugging Face case showed both failure sources: a laboratory that removed safeguards for testing, and containment that failed. Bloomberg reported on 18 September that a Palantir spokesperson said the company “is not responsible for the underlying data nor identifying intelligence deficiencies”, and that the government retains responsibility for data. The same report said users form operational understandings that differ from contractual terms. The paper allocation of responsibility and its practice can diverge.
Subscribe
New essays and snapshots, by email, roughly weekly.
Agents checking agents
After the Minab school strike, Sankar said Palantir built additional AI agents that “did more checking”. Bloomberg’s 18 September report said some Central Command personnel relied too heavily on the AI in Maven and expected it to flag stale data. A defence official told Bloomberg on 22 September that Maven cannot disqualify a target or correct the military intelligence database. Pentagon technology official Stanley said Maven “has to be optimized and told to do it”, and that his office “doesn’t control how combatant commands set up Maven”. Sankar disputes the reporting and attributes the likely cause to human error.
More checking is sensible engineering, but its value depends on independence. A checking agent that reads the same outdated records, shares the same model and works from the same assumptions can reproduce the original error with greater confidence. That is an analytical point rather than a finding about Maven, whose configuration is not public. The test for any second check is whether it brings evidence the first did not have.
The case for a single owner of accountability
The government’s position has force. Its fear, set out in Under Secretary Emil Michael’s memorandum and quoted in the appeals opinion, was that an Anthropic dispute over a sensitive operation abroad “raised material doubts as to whether they would cause their software to stop working or cause some other disastrous action that would put our warfighters’ lives in danger”. The court called the contractual limits “hardly self-defining”. NSPM-11, the June presidential memorandum, takes the same line: commanders remain accountable, and no commercial entity may be able to prevent, disable, degrade or materially modify military AI use without government approval. A democratic government may reasonably insist that its commanders, not a company, carry final responsibility.
The government has since retreated from the strongest version of that fear. Its June 3 order said the designation did not depend on any understanding of Anthropic’s real-time access to deployed models, and the California order said such access was not the department’s true concern. Training-time influence is a narrower, and more defensible, concern than a kill switch.
That position is legitimate, and it is also incomplete. A commander can be accountable only for what they can see, and no public mechanism lets anyone outside the chain examine how a model’s trained behaviour, a platform’s tool permissions and an operator’s settings combined in a given strike. Accountability without inspection is a title rather than a control.
Bloomberg reports that investigators examined chat logs and computer records from the Minab strike, so evidence of what the system did exists. The Pentagon’s review has not been released and may not be for years, and until it is, no one outside government can say which layer failed.
Disclosure: Anthropic is a named actor in this essay. McCyber uses Anthropic’s Claude models in its editorial workflow, as set out on the About page.
Sources: Anthropic PBC v. U.S. Department of War (N.D. Cal., 27 Aug 2026, via FindLaw); Anthropic PBC v. U.S. Department of War, No. 26-1049 (D.C. Cir., 25 Sep 2026), opinion; Associated Press (via ClickOrlando); Bloomberg (18 and 22 Sep); CSIS; CBS News (Face the Nation transcript, 11 Oct); NSPM-11 (White House, 5 Jun 2026); McCyber.
