Oops! Misleading AI-Generated Report 'Almost Started a War' With China

If you've used AI before, you know that it's not immune from mistakes, and hallucinations can easily crop up if you don't check the results. But though a large language model telling you to put glue on pizza might mean nothing more than a few laughs—not all uses of AI are so low stakes. The US military recently planned to take military action against a Chinese ship which it falsely believed to be transporting materials for a nuclear program to Iran, as a result of false information produced by an AI model, CNN reports. The report "almost started a war" according to one source. The US military had planned to intercept the vessel, according to four sources who spoke to the news outlet, but had realized the error shortly before the planned operation. Two other sources claimed that military officers were preparing to board the ship, while another said that the military were "in the air." The misleading report in question had reportedly been put together by a special operations command analyst using AI. The news comes as the Trump administration is making moves to incorporate AI into more parts of the military. In January, The Department of War released the Artificial Intelligence Acceleration Strategy, which outlined plans for the military to "become an 'AI-first' warfighting force across all domains," by unleashing experimentation, and eliminating "bureaucratic barriers." Many analysts within the military are concerned pitfalls of expanding AI use for use cases like targeting, with one source telling CNN "there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide." Another source told CNN that: "AI allows you to get to a bad idea faster." The Department of War has yet to on the reports. The wisdom of fully incorporating AI into warfare has already been questioned by some research. In February 2024, a study from Georgia Institute of Technology, Stanford University, Northeastern University, and the Hoover Wargaming and Crisis Simulation Initiative played simulated wargames with five of the most common large language models. The researchers found that overall the LLMs acted more aggressively than humans, and in what they saw they developed greater "arms-race dynamics, leading to greater conflict, and in rare cases, even to the deployment of nuclear weapons."

The US military recently planned to take military action against a Chinese ship which it falsely believed to be transporting materials for a nuclear program to Iran, as a result of false information produced by an AI model, CNN reports. The report "almost started a war" according to one source. The US military had planned to intercept the vessel, according to four sources who spoke to the news outlet, but had realized the error shortly before the planned operation. Two other sources claimed that military officers were preparing to board the ship, while another said that the military were "in the air." The misleading report in question had reportedly been put together by a special operations command analyst using AI. The news comes as the Trump administration is making moves to incorporate AI into more parts of the military. In January, The Department of War released the Artificial Intelligence Acceleration Strategy, which outlined plans for the military to "become an 'AI-first' warfighting force across all domains," by unleashing experimentation, and eliminating "bureaucratic barriers." Many analysts within the military are concerned pitfalls of expanding AI use for use cases like targeting, with one source telling CNN "there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide." Another source told CNN that: "AI allows you to get to a bad idea faster." The Department of War has yet to on the reports. The wisdom of fully incorporating AI into warfare has already been questioned by some research. In February 2024, a study from Georgia Institute of Technology, Stanford University, Northeastern University, and the Hoover Wargaming and Crisis Simulation Initiative played simulated wargames with five of the most common large language models. The researchers found that overall the LLMs acted more aggressively than humans, and in what they saw they developed greater "arms-race dynamics, leading to greater conflict, and in rare cases, even to the deployment of nuclear weapons."

The US military had planned to intercept the vessel, according to four sources who spoke to the news outlet, but had realized the error shortly before the planned operation. Two other sources claimed that military officers were preparing to board the ship, while another said that the military were "in the air." The misleading report in question had reportedly been put together by a special operations command analyst using AI. The news comes as the Trump administration is making moves to incorporate AI into more parts of the military. In January, The Department of War released the Artificial Intelligence Acceleration Strategy, which outlined plans for the military to "become an 'AI-first' warfighting force across all domains," by unleashing experimentation, and eliminating "bureaucratic barriers." Many analysts within the military are concerned pitfalls of expanding AI use for use cases like targeting, with one source telling CNN "there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide." Another source told CNN that: "AI allows you to get to a bad idea faster." The Department of War has yet to on the reports. The wisdom of fully incorporating AI into warfare has already been questioned by some research. In February 2024, a study from Georgia Institute of Technology, Stanford University, Northeastern University, and the Hoover Wargaming and Crisis Simulation Initiative played simulated wargames with five of the most common large language models. The researchers found that overall the LLMs acted more aggressively than humans, and in what they saw they developed greater "arms-race dynamics, leading to greater conflict, and in rare cases, even to the deployment of nuclear weapons."

The misleading report in question had reportedly been put together by a special operations command analyst using AI. The news comes as the Trump administration is making moves to incorporate AI into more parts of the military. In January, The Department of War released the Artificial Intelligence Acceleration Strategy, which outlined plans for the military to "become an 'AI-first' warfighting force across all domains," by unleashing experimentation, and eliminating "bureaucratic barriers." Many analysts within the military are concerned pitfalls of expanding AI use for use cases like targeting, with one source telling CNN "there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide." Another source told CNN that: "AI allows you to get to a bad idea faster." The Department of War has yet to on the reports. The wisdom of fully incorporating AI into warfare has already been questioned by some research. In February 2024, a study from Georgia Institute of Technology, Stanford University, Northeastern University, and the Hoover Wargaming and Crisis Simulation Initiative played simulated wargames with five of the most common large language models. The researchers found that overall the LLMs acted more aggressively than humans, and in what they saw they developed greater "arms-race dynamics, leading to greater conflict, and in rare cases, even to the deployment of nuclear weapons."

The news comes as the Trump administration is making moves to incorporate AI into more parts of the military. In January, The Department of War released the Artificial Intelligence Acceleration Strategy, which outlined plans for the military to "become an 'AI-first' warfighting force across all domains," by unleashing experimentation, and eliminating "bureaucratic barriers." Many analysts within the military are concerned pitfalls of expanding AI use for use cases like targeting, with one source telling CNN "there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide." Another source told CNN that: "AI allows you to get to a bad idea faster." The Department of War has yet to on the reports. The wisdom of fully incorporating AI into warfare has already been questioned by some research. In February 2024, a study from Georgia Institute of Technology, Stanford University, Northeastern University, and the Hoover Wargaming and Crisis Simulation Initiative played simulated wargames with five of the most common large language models. The researchers found that overall the LLMs acted more aggressively than humans, and in what they saw they developed greater "arms-race dynamics, leading to greater conflict, and in rare cases, even to the deployment of nuclear weapons."

Many analysts within the military are concerned pitfalls of expanding AI use for use cases like targeting, with one source telling CNN "there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide." Another source told CNN that: "AI allows you to get to a bad idea faster." The Department of War has yet to on the reports. The wisdom of fully incorporating AI into warfare has already been questioned by some research. In February 2024, a study from Georgia Institute of Technology, Stanford University, Northeastern University, and the Hoover Wargaming and Crisis Simulation Initiative played simulated wargames with five of the most common large language models. The researchers found that overall the LLMs acted more aggressively than humans, and in what they saw they developed greater "arms-race dynamics, leading to greater conflict, and in rare cases, even to the deployment of nuclear weapons."

The Department of War has yet to on the reports. The wisdom of fully incorporating AI into warfare has already been questioned by some research. In February 2024, a study from Georgia Institute of Technology, Stanford University, Northeastern University, and the Hoover Wargaming and Crisis Simulation Initiative played simulated wargames with five of the most common large language models. The researchers found that overall the LLMs acted more aggressively than humans, and in what they saw they developed greater "arms-race dynamics, leading to greater conflict, and in rare cases, even to the deployment of nuclear weapons."

The wisdom of fully incorporating AI into warfare has already been questioned by some research. In February 2024, a study from Georgia Institute of Technology, Stanford University, Northeastern University, and the Hoover Wargaming and Crisis Simulation Initiative played simulated wargames with five of the most common large language models. The researchers found that overall the LLMs acted more aggressively than humans, and in what they saw they developed greater "arms-race dynamics, leading to greater conflict, and in rare cases, even to the deployment of nuclear weapons."

The researchers found that overall the LLMs acted more aggressively than humans, and in what they saw they developed greater "arms-race dynamics, leading to greater conflict, and in rare cases, even to the deployment of nuclear weapons."