
Secretaries of the Military Departments discuss Project Agincourt at the 2026 Association of the United States Army in Washington, D.C., Oct.14, 2026. Defense leaders outlined plans for a new four-star Autonomous Warfare Command designed to unify unmanned systems, artificial intelligence, and drone capabilities across the Joint Force. The initiative reflects a broader effort to build a faster, more agile military by accelerating innovation, streamlining acquisition, and strengthening partnerships with industry. Leaders emphasized rapidly delivering capabilities to Soldiers, enhancing deterrence, and preparing the force for future conflicts.
Can Commanders Tell When Artificial Intelligence Is Wrong?
Published Nov. 17, 2026 • Read time: 8m
In January, the Department of War directed the military to become an “AI-first” warfighting force. Its strategy included an initiative called Agent Network, intended to advance AI-enabled battle management and decision support from campaign planning to kill-chain execution.[1] The attraction is clear. A system that processes large amounts of information quickly could help commanders assess a changing situation and consider options while there is still time to act.
But an AI-generated recommendation can be wrong even when the software operates exactly as designed. A system may misinterpret a sensor report, rely on outdated information, or produce a plausible assessment from an incorrect assumption. The resulting recommendation may arrive faster than a staff could develop it and appear more certain than the evidence warrants.
The Pentagon needs to establish whether commanders and their staffs can identify those errors before acting on them. Testing a model’s accuracy is necessary. It does not establish that the military personnel using it will recognize an incorrect recommendation under operational pressure.
The Limits of an Accurate Model
An October 2026 Government Accountability Office report examined efforts to implement responsible artificial intelligence across the department. GAO found that most of the ten components it surveyed had taken at least initial steps to implement oversight in selected AI uses. It also identified coordination and staffing problems. As of February 2026, only three of ten authorized positions in the Chief Digital and AI Officer’s Responsible AI Division were filled.[2]
Those findings do not establish that fielded military AI systems are unsafe or that commanders are routinely accepting bad recommendations. They do show that the department is accelerating adoption while parts of the organization responsible for guiding responsible use remain underdeveloped.
The operational question goes beyond whether an algorithm performs well on a benchmark. Consider an AI tool that helps a staff correlate tracks, intelligence reports, and other information to recommend a course of action. It may perform accurately in testing against representative data. During operations, however, one source could be delayed, a contact could be misidentified, or an adversary could feed deceptive information into the reporting chain. The system might then produce an internally consistent answer from a faulty account of the situation.
A second AI tool would not necessarily resolve the problem if it relied on the same underlying data. Agreement between two outputs can create the appearance of independent confirmation when both share the same error. Commanders need to know what evidence supports a recommendation, how current that evidence is, and which assumptions could change the result.
There is also a distinction between a calibrated confidence estimate and confident language. Some classification systems can report a score that has been evaluated against observed outcomes. A generative AI system may instead present a persuasive explanation without a valid numerical measure of its likelihood of being correct.[3] Treating the two as equivalent gives an operator more assurance than the system has earned.
The Limits of Human Oversight
Human oversight is often presented as the answer to these problems. Keeping a person in the decision process is important, particularly where an error could endanger friendly forces or civilians. Yet a human approval step offers limited protection if the person has no practical means of recognizing a mistake.
The problem is familiar in human-factors research. In a 2010 review in Human Factors, Raja Parasuraman and Dietrich Manzey examined automation bias, the tendency to follow an automated recommendation despite contrary evidence or to miss a problem because an automated aid did not flag it. They found that these errors can affect experienced operators as well as novices and that simple training or instructions do not reliably eliminate them.[4]
Military command environments can intensify the conditions that make independent judgment difficult. Staff members work with incomplete information, competing demands, and deadlines imposed by an adversary. An AI tool that consistently produces useful results may become the first source consulted and eventually the answer against which other information is judged. None of this requires an operator to be careless.
There is an opposite risk. A staff that rejects useful AI recommendations after encountering a few errors may give up the speed or analytical capacity the system was intended to provide. Effective oversight therefore requires appropriate reliance: accepting assistance when the evidence supports it and challenging the system when it does not.
Test the Team, Not Just the Technology
The Department of War should evaluate AI decision aids as human-machine systems, with mission performance as the principal measure. Software tests should establish technical accuracy, reliability, and security within defined operating conditions. Operational evaluations should then determine what happens when personnel use the capability during realistic missions.
First, evaluators should introduce incorrect or misleading recommendations into exercises without telling operators which outputs are suspect. Scenarios should include stale data, contradictory reporting, adversarially manipulated inputs, and recommendations that exceed the system’s validated purpose. The tests should measure whether personnel identify the problem, how long detection takes, and whether the resulting decision is better or worse than decisions made without AI assistance.
The objective should not be to reward operators for disagreeing with a machine. A team that rejects every recommendation avoids automation bias but fails to use the capability. Evaluators should measure appropriate acceptance as well as correct rejection, and assess the operational consequences of both kinds of error.
Second, acquisition requirements should specify what information the operator must receive to evaluate a recommendation. That may include source attribution, data age, known limitations, competing explanations, and indications that a system is operating outside the conditions for which it was validated. A long technical explanation is not necessarily useful in combat. The information must be understandable at the speed and level of detail required for the decision.
The National Institute of Standards and Technology’s AI Risk Management Framework calls for validity and reliability testing, documentation of limitations, and measurements suited to the way a system will be used.[5] The military should translate those principles into mission-specific acceptance criteria. For a time-sensitive track-identification task, for example, the relevant measures might include identification errors, operator correction rates, and the time required to confirm a track using independent evidence.
Third, commanders should establish in advance which decisions require independent corroboration, what happens when AI outputs conflict with other reporting, and when the system must be bypassed. Those rules will differ between an administrative planning aid and a tool supporting the use of force. Operational authority remains with the designated human decisionmaker; the level of verification must reflect the consequences of being wrong.
A Readiness Standard for AI-Supported Decisions
The department has already recognized the need to test new capabilities more effectively. In December 2025, GAO found that defense test and evaluation policies did not fully reflect leading practices such as early tester involvement and iterative user feedback.[6] Those weaknesses are particularly relevant to AI, where performance depends on the interaction between the technology, the data it receives, and the people using it.
A June 2026 national security memorandum directed the development of standardized AI test, evaluation, verification, and validation methods and an AI curriculum addressing capabilities and limitations.[7] Those requirements create an opportunity to make operator error recognition an explicit part of readiness rather than an assumed benefit of human oversight.
An AI capability should not receive full operational credit simply because it meets an accuracy target in controlled testing. The using command should demonstrate that its personnel can recognize material errors, obtain independent evidence when required, and complete the assigned mission when the automated recommendation cannot be trusted.
Artificial intelligence may help commanders make better decisions faster. Whether it does so in combat depends on more than the quality of its answers. The Pentagon must also establish what happens when an answer is wrong and whether the people responsible for acting on it can tell the difference.
CAPT Jonathan K. Corrado, Ph.D. (U.S. Navy Reserve) works in the nuclear industry.
The views in this article are the author’s own and do not represent the Department of War, the Department of the Navy, or any other organization.
Notes:
[1] U.S. Department of War, “Artificial Intelligence Strategy for the Department of War,” January 9, 2026,
https://media.defense.gov/2026/Jan/12/2003855671/-1/-1/0/ARTIFICIAL-INTELLIGENCE-STRATEGY-FOR-THE-DEPARTMENT-OF-WAR.PDF.
[2] U.S. Government Accountability Office, “Artificial Intelligence: Actions and Opportunities to Implement Responsible Use in DOD,” GAO-27-107837, October 14, 2026,
https://www.gao.gov/products/gao-27-107837.
[3] Jonathan K. Corrado, Michael Thompson, and Peter Line, “Thinking, Cognition, and the Limits of Artificial Intelligence,” Journal of Creation 41, no. 1 (2027), forthcoming.
[4] Raja Parasuraman and Dietrich H. Manzey, “Complacency and Bias in Human Use of Automation: An Attentional Integration,” Human Factors 52, no. 3 (2010): 381–410,
https://doi.org/10.1177/0018720810376055.
[5] National Institute of Standards and Technology, “AI Risk Management Framework Playbook: Measure,”
https://airc.nist.gov/airmf-resources/playbook/measure/.
[6] U.S. Government Accountability Office, “Weapon Systems Testing: DOD Needs to Update Policies to Better Support Modernization Efforts,” GAO-26-107009, December 11, 2025,
https://www.gao.gov/products/gao-26-107009.
[7] The White House, “National Security Presidential Memorandum/NSPM-11,” June 5, 2026, sections 5(c) and 5(f),
https://www.whitehouse.gov/presidential-actions/2026/06/national-security-presidential-memorandum-nspm-11/.

Why Iran Cannot Simply Surrender
By Eric Garland • RealClearDefense
The Formation Left Us Behind
By Mark W. Castillon • RealClearDefense
Can Commanders Tell When Artificial Intelligence Is Wrong?
By Jonathan K. Corrado • RealClearDefense
The Summit as Mirror
By Stephen Blank • RealClearDefense
The False Choice That Could Get Us Killed
By Adam Strickland • RealClearDefense

Why Iran Cannot Simply Surrender
By Eric Garland • RealClearDefense
U.S. Navy Cuts Submarine Maintenance Delays by 40%
By Ethan Gossrow, Naval News
Air Force Bets on Hybrid-Electric Aircraft for Transport
By L.C. Williams, Defense One
Roosevelt Joins 2 Other Carriers in Mideast As Pressure on Iran Rises
By Stars & Stripes
Fight Now? Time to beef up III MEF
By Marine Corps Compass Points
