Key Takeaways
- The incident: An intelligence report generated by an AI chatbot mistakenly identified a Chinese vessel as carrying nuclear components, triggering a near-military operation with American special forces already poised to board.
- The technology involved: An analyst at Special Operations Command Pacific (SOCOM) used an AI chatbot both to analyze data and to draft the formal report, amplifying the error as it moved up the chain of command.
- The impact: The episode exposes the contradictions of the Pentagon's "AI-first" strategy, reigniting debate over human oversight, decision-making speed, and the risk of unintended escalation with China.
The operation stopped at the last moment
This past spring, as American military aircraft were already airborne and armed special forces personnel prepared to board a Chinese vessel, officials in Washington discovered that the intelligence behind the operation was false. CNN broke the story, citing four sources familiar with the matter. The report, circulated during the war with Iran, claimed the ship was carrying components for a nuclear program in the Middle East.
Just before the interception, as officials dug deeper into the document, they discovered it had been drafted by a SOCOM analyst with help from an AI chatbot. The system had incorrectly merged open-source intelligence with classified signals about the ship's manifest, misidentifying the cargo. The analyst then used the AI again to format the flawed findings into a formal report, which was pushed through official command channels. One source called it "entirely false"; another said it "nearly started a war."
An "AI-first" department
The incident comes as the Department of Defense brands itself an "AI-first war department," with the stated goal of keeping the United States "AI dominant," according to posts from the Pentagon's CTO office. Defense Secretary Pete Hegseth has said he intends to "unleash experimentation, cut through bureaucratic red tape, and focus investment" in military AI. A former senior official, however, noted that "the internal tools are mostly copies of commercial ones with lipstick on them."
The core risk isn't just the flawed content, but the speed at which it spreads. A military source told CNN that "AI lets you get to a bad idea faster," adding that "the use of AI in targeting is increasing, and there's no real guidance on how the human in the loop will prevent civilian casualties or friendly fire." The phenomenon echoes the "automation bias" observed back in 1988, when the USS Vincennes' Aegis system misclassified an Iranian passenger jet as an enemy fighter, killing 290 civilians.
Speed versus oversight
Jake Steckler, an Army veteran and research scholar at GovAI, stresses that the military must understand "the inherent uncertainty of LLMs, especially for decisions that could lead to the use of force," since these are choices with "life-and-death consequences." He also warns that prioritizing adoption speed above all else will generate incidents that erode operators' trust in the systems themselves, ultimately slowing down implementation.
That same week, scholars from Brookings and Tsinghua University publicly called for AI safeguards comparable to those governing nuclear weapons, along with a direct military hotline between the United States and China. The Trump administration dismissed the alarmist scenarios, insisting that slowing AI development risks ceding ground to Beijing.
Who checks the next report
Emma Ashford of the Stimson Center distilled the danger into a line that went viral: "This is how AI kills us all — not intentionally, not consciously, but by making a dumb mistake that humans don't bother to double-check."
No autonomous robots or sentient systems are required — just an analyst trusting the tool, a report that looks credible, and a chain of command under pressure. In this case, someone caught it before it was too late. The question of who catches the next one remains open.
