An AI hallucination nearly triggered a US military strike
Military aircraft were already airborne this spring when officials realized the intelligence driving an armed operation against a Chinese vessel had been fabricated by an AI chatbot. The strike was aborted at the last minute.
Published: 2026-09-21 Category: Quick Take
Sources:
The near-miss
This isn't a hypothetical about the future of AI in warfare. It happened this spring. Military aircraft were already in the air when U.S. officials realized the intelligence driving an armed operation against a Chinese vessel had been hallucinated by an AI chatbot, CNN reported Friday. The operation was aborted at the last minute, narrowly averting a potential conflict with China.
The false intelligence — which circulated during the war with Iran — claimed the vessel was carrying components for a nuclear weapons program. It originated with a Special Operations Command analyst who queried an AI chatbot to synthesize open source data with classified signals intelligence. The chatbot misidentified the ship's cargo manifest. The analyst then used the tool a second time to format the erroneous findings into an official-looking summary, which was circulated across command channels.
The failure is procedural, not just technical
The most troubling part isn't the hallucination itself — LLMs hallucinate, that's a known failure mode. It's that the error traveled up the chain of command without anyone catching it in time. The analyst treated the chatbot's output as authoritative input, formatted it into something that looked official, and that document propagated through the kill chain. The model made the mistake; the process amplified it.
The Pentagon has framed AI as a strategic advantage, letting commanders speed up the kill chain and respond faster than adversaries. But the same speed that makes AI attractive is precisely what allows hallucinations to slip through with insufficient human oversight. When decision-making accelerates, the checkpoints that used to catch errors can get compressed out of existence.
What experts say
Jake Steckler, a research scholar at GovAI and a veteran U.S. Army officer, told TechCrunch it's critical for service members to understand the uncertainty inherent to LLMs — "especially for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences."
But he cautioned against treating this as a reason to abandon AI. "These tools can be useful in the right contexts and with the right safeguards in place," he said. "But prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems."
The pattern
This incident mirrors a broader concern: AI errors are increasingly hard to pin down because they travel in the form of polished, confident output. An AI that formats a hallucinated finding into an official-looking summary is doubly dangerous — it doesn't just get the facts wrong, it makes the wrong facts look authoritative. For an analyst under pressure, the formatting is indistinguishable from competence.
The lesson isn't "don't use AI in the military." It's that the safeguards must keep pace with the adoption speed — and that an AI which can format its own hallucinations into official documents needs a human confirmation step that was clearly missing here. That we got within minutes of a conflict with China before catching it is the alarm, and it should be ringing loudly.
This Quick Take is based on reporting by Aditya Mehta at TechCrunch.