Tesari

•

October 7 2026

Remaining Human in the Loop

The illusion of expertise endangers operations

October 29, 2026

Authors: Elisar Nurmagambet and Amy McFadden

​

The AI safety debate just exploded into the headlines. On September 8, Anthropic researcher Jacob Coxon resigned, accusing his employer and OpenAI of "gambling with our lives." Anthropic's alignment lead responded, putting the odds of catastrophe above ten percent. By the following weekend, Anthropic chief Dario Amodei had published an essay calling for the frontier labs to slow down, and OpenAI chief Sam Altman publicly agreed.

Meanwhile, an Iran-linked actor used Claude to assemble targeting handbooks on US naval movements. A cell in northern Yemen used Claude to write guidance software for missiles. An actor linked to the PLA's Academy of Military Sciences simulated strikes on twelve targets in Taiwan.

Every one of these stories is framed around the machine, its capabilities, and whether it can be contained. Anthropic has drawn a pair of red lines: no mass domestic surveillance and no weapon that selects and engages targets without a human making the final call.

However, militaries, corporations, and intelligence professionals must make consequential decisions, including targeting, under deep uncertainty and tight time constraints. This often sends practitioners reaching for the LLM to act quickly, which means walking through a minefield of cognitive biases. In essence, if practitioners want to be the human in the loop, they must add what the machine promises to relieve: intellectual rigor.

Prosthetic expertise

An artificial limb does not replace an arm or leg. It can perform many of the same tasks and in some cases enhance abilities, but the human being must learn to use the device, starting with understanding its shortcomings and its advantages. AI must be used the same way.

In Thinking, Fast and Slow, Daniel Kahneman outlines two systems of thinking for decision making. System 1 is fast, fluent, and unconscious. System 2 is slow, deliberate, and rigorous. Ideally, both systems check one another for clear and dynamic decision making. Consider chess masters who carry some 50,000 board patterns in memory, built up over years of study. This depth of understanding, assembled through rigorous System 2 work, allows them to deploy it rapidly in a blitz match.

Magnus Carlsen, the world’s top-ranked chess player, has described his own process this way: "I usually know what I am going to do after 10 seconds; the rest is double-checking." System 1 provides the choice right away. System 2 makes sure that it is correct.

Now, apply this process to high-level practitioners. Imagine an intelligence officer approving a targeting assessment in Iran. She has an IRGC naval facility packaged in front of her with reams of LLM-produced intelligence supporting a rapid strike. The clock is ticking. That same officer has not spent months watching the facility. She did not see indicators that the building had become a grammar school: the cessation of truck traffic coming at night, the removal of the antenna array in the spring, the posting of a school sign over the gate. The imagery and the reporting that the model was trained on stopped before the change. When the officer asked whether the target was still valid, the LLM produced three fluent paragraphs explaining why it was. Its output was optimized for a fast and easy “yes” rather than a hard and consequential “let's find out more.”

The human in the loop had not spent years learning 50,000 positions, but she felt as if she had. This is the problem. LLMs give the feeling of using System 2 thinking while circumventing it. Being a good human in the loop means using artificial intelligence to question and clarify, rather than to answer and act.

The right tool for the right task

Machines are exceptional at many tasks. Some industries, like intelligence, defense, and medicine, can apply AI rapidly to good effect due to their repeatable problem sets and depth of training data. But knowing the tool means understanding the benefits and costs of using it as well as the tool’s strengths and limitations. Consider the colonoscopy and the mammogram. A 2024 meta-analysis of 44 randomized trials found that AI assistance raised adenoma detection by about a fifth. In mammography, the first randomized trial of AI-supported screening, covering more than 100,000 women in Sweden, found radiologists detected 29% more cancers with no rise in false positives, and a 44% cut in reading workload.

Any executive who can improve detection of critical information while reducing cognitive load understands that as an easy win. But there is another side to this story. In 2025, Budzyń and colleagues looked at four centers that had adopted the tool and asked what the endoscopists did when the AI was switched off. Their unassisted detection rate fell from 28.4% to 22.4%. The endoscopists had come to lean on the extraordinary abilities afforded by AI, while their organic experience had begun to atrophy.

Now consider your due-diligence team, your country-risk desk, your red cell. If they have been drafting with a model for 18 months, what does their unassisted judgment look like today? What would it look like in a constrained environment on a diminishing timeline? When AI is used to supply answers and drive actions, human judgment erodes in its absence. When it is used instead to sharpen questions and clarify thinking, the same expertise can be brought to bear faster and more effectively.

In a 2025 collaboration between Microsoft Research and Carnegie Mellon that surveyed 319 knowledge workers across 936 real tasks, the researchers found that the more a worker trusted the AI, the less critical thinking they engaged in. This might seem obvious, but the reverse also held. Workers who trusted their own judgment thought harder and treated the output as something to verify. A separate study of 666 participants found AI use negatively associated with critical-thinking performance because users were offloading more of their thinking to the technology. This practice was most pronounced among the youngest users, who need the most at-bats to train their System 2 thinking. Machines keep them in the dugout.

Intelligence professionals will recognize the problem and the fix. The intelligence community learned how much forecasting skill it was leaving on the table only when IARPA's Aggregative Contingent Estimation (ACE) program, run by Jason Matheny, spent four years keeping score: more than 20,000 forecasters made two million judgments, each graded against what actually happened. Calibration became apparent, and once it was visible it could be trained. But this was only possible because a baseline had been established. Most organizations adopting language models have no unassisted baseline, so whatever is happening to their people's judgment, they have no instrument that can detect it. These costs and limitations are interrelated.

Many AI companies approach problems from the capabilities of their technology rather than from deep knowledge of the domains in which it will operate. However, in complex fields such as intelligence, defense, and medicine, it is more important to know what should be automated rather than what can be automated, and most of all where human judgment must remain decisive. The best engineering teams cannot substitute for the accumulated expertise required to understand the problem, recognize when the machine is wrong, and know when not to use it. AI therefore makes deep domain expertise more important, not less.

 

The right tool for the right task

Machines are exceptional at many tasks. Some industries, like intelligence, defense, and medicine, can apply AI rapidly to good effect due to their repeatable problem sets and depth of training data. But knowing the tool means understanding the benefits and costs of using it as well as the tool’s strengths and limitations. Consider the colonoscopy and the mammogram. A 2024 meta-analysis of 44 randomized trials found that AI assistance raised adenoma detection by about a fifth. In mammography, the first randomized trial of AI-supported screening, covering more than 100,000 women in Sweden, found radiologists detected 29% more cancers with no rise in false positives, and a 44% cut in reading workload.

Any executive who can improve detection of critical information while reducing cognitive load understands that as an easy win. But there is another side to this story. In 2025, Budzyń and colleagues looked at four centers that had adopted the tool and asked what the endoscopists did when the AI was switched off. Their unassisted detection rate fell from 28.4% to 22.4%. The endoscopists had come to lean on the extraordinary abilities afforded by AI, while their organic experience had begun to atrophy.

Now consider your due-diligence team, your country-risk desk, your red cell. If they have been drafting with a model for 18 months, what does their unassisted judgment look like today? What would it look like in a constrained environment on a diminishing timeline? When AI is used to supply answers and drive actions, human judgment erodes in its absence. When it is used instead to sharpen questions and clarify thinking, the same expertise can be brought to bear faster and more effectively.

In a 2025 collaboration between Microsoft Research and Carnegie Mellon that surveyed 319 knowledge workers across 936 real tasks, the researchers found that the more a worker trusted the AI, the less critical thinking they engaged in. This might seem obvious, but the reverse also held. Workers who trusted their own judgment thought harder and treated the output as something to verify. A separate study of 666 participants found AI use negatively associated with critical-thinking performance because users were offloading more of their thinking to the technology. This practice was most pronounced among the youngest users, who need the most at-bats to train their System 2 thinking. Machines keep them in the dugout.

Intelligence professionals will recognize the problem and the fix. The intelligence community learned how much forecasting skill it was leaving on the table only when IARPA's Aggregative Contingent Estimation (ACE) program, run by Jason Matheny, spent four years keeping score: more than 20,000 forecasters made two million judgments, each graded against what actually happened. Calibration became apparent, and once it was visible it could be trained. But this was only possible because a baseline had been established. Most organizations adopting language models have no unassisted baseline, so whatever is happening to their people's judgment, they have no instrument that can detect it. These costs and limitations are interrelated.

Many AI companies approach problems from the capabilities of their technology rather than from deep knowledge of the domains in which it will operate. However, in complex fields such as intelligence, defense, and medicine, it is more important to know what should be automated rather than what can be automated, and most of all where human judgment must remain decisive. The best engineering teams cannot substitute for the accumulated expertise required to understand the problem, recognize when the machine is wrong, and know when not to use it. AI therefore makes deep domain expertise more important, not less.

 

Confidence you did not earn

False confidence hits the bottom line and could cost lives. In 2025, a team published a study of 991 forecasters in ACM Transactions on Interactive Intelligent Systems. Participants were fitted with one of two different prosthetics. The first was an LLM assistant prompted to reason like a superforecaster: weigh base rates, express calibrated uncertainty, resist the confident guess. The second was prompted to do the opposite, to be overconfident and to ignore base rates entirely. Both groups improved their forecasting accuracy over a control group, and by a comparable margin. The bad prosthetic worked almost as well as the good one, because the forecasters were not evaluating the evidence it offered. The most dangerous part is that they felt as if they were arriving at the answer organically.

The mechanism is now well documented. In 2025, cognitive-science researchers found that human confidence in a model's answer tracks the length and fluency of its explanation rather than its accuracy. Essentially, the more the machine wrote, the more convincing it was. Other researchers found the same thing in 2021, before the current generation of models: adding explanations to AI output increased human reliance. A more recent study, in 2024, described this phenomenon as “an illusion of understanding,” in which people feel they know more than they actually do.

One plausible reason is that a lengthy explanation overwhelms the reader, and they accept spilled digital ink over their earned intuition. This reflects a leadership liability; when decision makers are surrounded by convincing yes men, they tend to make poor choices. When those yes men live in their cellphones and offer the simulacrum of understanding at a moment's notice, even questionable judgments can look well-founded. As Richard Heuer indicated in Psychology of Intelligence Analysis, the mental model you bring to the evidence determines what you see in it. LLMs are optimized for interaction and not incision. That is why they will not only confirm a decision maker's mental model, but eventually shape and potentially displace it, becoming less an aid to judgment than a cognitive prosthesis. Stopping this effect is the responsibility of the human in the loop.

Confidence you did not earn

False confidence hits the bottom line and could cost lives. In 2025, a team published a study of 991 forecasters in ACM Transactions on Interactive Intelligent Systems. Participants were fitted with one of two different prosthetics. The first was an LLM assistant prompted to reason like a superforecaster: weigh base rates, express calibrated uncertainty, resist the confident guess. The second was prompted to do the opposite, to be overconfident and to ignore base rates entirely. Both groups improved their forecasting accuracy over a control group, and by a comparable margin. The bad prosthetic worked almost as well as the good one, because the forecasters were not evaluating the evidence it offered. The most dangerous part is that they felt as if they were arriving at the answer organically.

The mechanism is now well documented. In 2025, cognitive-science researchers found that human confidence in a model's answer tracks the length and fluency of its explanation rather than its accuracy. Essentially, the more the machine wrote, the more convincing it was. Other researchers found the same thing in 2021, before the current generation of models: adding explanations to AI output increased human reliance. A more recent study, in 2024, described this phenomenon as “an illusion of understanding,” in which people feel they know more than they actually do.

One plausible reason is that a lengthy explanation overwhelms the reader, and they accept spilled digital ink over their earned intuition. This reflects a leadership liability; when decision makers are surrounded by convincing yes men, they tend to make poor choices. When those yes men live in their cellphones and offer the simulacrum of understanding at a moment's notice, even questionable judgments can look well-founded. As Richard Heuer indicated in Psychology of Intelligence Analysis, the mental model you bring to the evidence determines what you see in it. LLMs are optimized for interaction and not incision. That is why they will not only confirm a decision maker's mental model, but eventually shape and potentially displace it, becoming less an aid to judgment than a cognitive prosthesis. Stopping this effect is the responsibility of the human in the loop.

How to be the “human in the loop”

Learning how to use LLMs to enhance decision makers' speed and accuracy means practicing with the prosthetic rather than relying on it. Three practices can ensure that the human in the loop remains more than a rubber stamp and preserves cognitive skills once the device is switched off.

 

1.     Score the unassisted baseline. Most organizations have rushed to leverage AI as quickly as possible with the promise of lowering costs and increasing output. Many of them never understood what their organization looked like beforehand. Brier scores and tool-free exercises can identify the true talents of working groups. If you cannot measure judgment without the machine, you cannot see it decay. Regularly turn off the bots and require teams to work from first principles.

2.     Build in friction. Thinking should remain difficult. That is how you know that System 2 is functioning. A 2021 study found that requiring people to commit to a view before seeing AI output substantially reduced overreliance. Decision makers at all levels should know what unassisted cognition feels like. The team that prepares for a future where digital consideration is turned off or corrupted will retain decision advantage.

3.     Focus on questions and clarifications, not answers and actions. Use the model to sharpen the question, generate the alternative hypothesis, and surface what you have not considered. Then turn it against your own conclusion. This is the machine version of Heuer's Analysis of Competing Hypotheses, done at speed, which preserves the analytic discipline that keeps judgment sharp while exposing disagreement.

How to be the “human in the loop”

Learning how to use LLMs to enhance decision makers' speed and accuracy means practicing with the prosthetic rather than relying on it. Three practices can ensure that the human in the loop remains more than a rubber stamp and preserves cognitive skills once the device is switched off.

 

1.     Score the unassisted baseline. Most organizations have rushed to leverage AI as quickly as possible with the promise of lowering costs and increasing output. Many of them never understood what their organization looked like beforehand. Brier scores and tool-free exercises can identify the true talents of working groups. If you cannot measure judgment without the machine, you cannot see it decay. Regularly turn off the bots and require teams to work from first principles.

2.     Build in friction. Thinking should remain difficult. That is how you know that System 2 is functioning. A 2021 study found that requiring people to commit to a view before seeing AI output substantially reduced overreliance. Decision makers at all levels should know what unassisted cognition feels like. The team that prepares for a future where digital consideration is turned off or corrupted will retain decision advantage.

3.     Focus on questions and clarifications, not answers and actions. Use the model to sharpen the question, generate the alternative hypothesis, and surface what you have not considered. Then turn it against your own conclusion. This is the machine version of Heuer's Analysis of Competing Hypotheses, done at speed, which preserves the analytic discipline that keeps judgment sharp while exposing disagreement.

The implementation of AI by Industry: Use Case of Arcanum and Tesari AI

The intelligence market is responding to AI development in three major ways. First, legacy compliance firms are introducing AI internally to perform faster routine checks. Top-layer data surfaces obvious red flags and sorts entities into two buckets: safe and flagged. This approach is as old as the compliance industry itself. Yet the shift is underway, moving from box-checking toward deep vetting of every related entity, including vendors, clients, partners, owners, and investors.

Secondly, legacy consulting and professional services firms like Booz Allen and the Big Four are partnering with major AI labs such as OpenAI and Anthropic. This direction is driven not by an interest in expanding what organizations can know but by the goal of automating workflows and operational systems.

The third response is the most ambitious and arguably the most in demand: using AI to create an entirely new category of intelligence and investigative capability. Private intelligence firms, national security agencies, global capital, and other organizations operating in high-stakes environments are seeking solutions that scale their capabilities to enter new markets, gain visibility into more regions, and reduce the delay and blind spots that have defined this work until now. This approach is the only one positioned to withstand the global shifts already reshaping economics, geopolitics, capital movements, trade, logistics, and finance.

One timely example of the third response is the September 2026 partnership between Arcanum Global, a government-grade intelligence firm founded by veterans of America's elite military and intelligence units, and Tesari AI, a transnational risk intelligence platform built for high-stakes environments. Arcanum holds that the value of intelligence ultimately depends on deep human expertise: knowing which questions to ask, how to interpret what is found, and what it means for the decision at hand. Tesari's role in this partnership is to multiply the reach behind that expertise: analysts, on-the-ground agents, translators, and regional specialists, extended across far more ground than any team could cover manually.

The result is a process reliable and fast enough to keep pace with current national security requirements and rising global tension, while the judgment itself stays where it belongs, with the people who know how to read what the intelligence means. Partnerships like this one keep the human at the helm, where the decisions and the judgment still belong.

References:

Axios. "Anthropic, OpenAI CEOs call for slowdown in AI development." 12 September 2026. https://www.axios.com/2026/09/12/anthropic-ai-amodei-pacing.

Axios. "Anthropic report: 5 ways Claude was exploited for war, spying and repression." 12 September 2026. https://www.axios.com/2026/09/12/anthropic-ai-threat-report-russia-iran-china.

Axios. "Exclusive: Pentagon threatens to cut off Anthropic in AI safeguards dispute." 15 February 2026. https://www.axios.com/2026/02/15/claude-pentagon-anthropic-contract-maduro.

Bansal, Gagan, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. "Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance." In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2021. https://doi.org/10.1145/3411764.3445717.

Buçinca, Zana, Maja Barbara Malaya, and Krzysztof Z. Gajos. "To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making." Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021): article 188. https://doi.org/10.1145/3449287.

Budzyń, Krzysztof, Marcin Romańczyk, Diana Kitala, et al. "Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study." The Lancet Gastroenterology & Hepatology 10, no. 10 (2025): 896–903. https://doi.org/10.1016/S2468-1253(25)00133-5.

CNBC. "'Extinction' warnings ramp up as more OpenAI, Anthropic researchers join calls for an AI slowdown." 10 September 2026. https://www.cnbc.com/2026/09/10/openai-anthropic-ai-safety-slowdown-extinction.html.

Euronews. "AI on the battlefield: How is the US integrating AI into its military?" 6 March 2026. https://www.euronews.com/next/2026/03/06/ai-on-the-battlefield-how-is-the-us-integrating-ai-into-its-military.

Gerlich, Michael. "AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking." Societies 15, no. 1 (2025): 6. https://doi.org/10.3390/soc15010006.

Hernström, Veronica, Viktoria Josefsson, Hanna Sartor, et al. "Screening performance and characteristics of breast cancer detected in the Mammography Screening with Artificial Intelligence trial (MASAI): a randomised, controlled, parallel-group, non-inferiority, single-blinded, screening accuracy study." The Lancet Digital Health 7, no. 3 (2025): e175–e183. https://doi.org/10.1016/S2589-7500(24)00267-X.

Heuer, Richards J., Jr. Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency, 1999. IARPA. "Aggregative Contingent Estimation (ACE)." https://www.iarpa.gov/research- programs/ace. IEEE Spectrum. "IARPA's New Director Wants You to Surprise Him." 2015. https://spectrum.ieee.org/iarpas-new-director-wants-you-to-surprise-him.

Kahneman, Daniel. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011.

Lång, Kristina, Viktoria Josefsson, Anna-Maria Larsson, et al. "Artificial intelligence-supported screen reading versus standard double reading in the Mammography Screening with Artificial Intelligence trial (MASAI): a clinical safety analysis of a randomised, controlled, non-inferiority, single-blinded, screening accuracy study." The Lancet Oncology 24, no. 8 (2023): 936–944. https://doi.org/10.1016/S1470-2045(23)00298-X.

Lee, Hao-Ping (Hank), Advait Sarkar, Lev Tankelevitch, Ian Drosos, Sean Rintel, Richard Banks, and Nicholas Wilson. "The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers." In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2025. https://doi.org/10.1145/3706598.3713778.

Messeri, Lisa, and M. J. Crockett. "Artificial intelligence and illusions of understanding in scientific research." Nature 627 (2024): 49–58. https://doi.org/10.1038/s41586-024-07146-0. Reuters. "Factbox: How Anthropic says Claude was used for weapons, spying and cyber operations." 11 September 2026. https://www.usnews.com/news/world/articles/2026-09-11/factbox-how-anthropic-says-claude-was-used-for-weapons-spying-and-cyber-operations.

Schoenegger, Philipp, Peter S. Park, Ezra Karger, Sean Trott, and Philip E. Tetlock. "AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy." ACM Transactions on Interactive Intelligent Systems 15, no. 1 (2025): article 4. https://doi.org/10.1145/3707649.

Simon, Herbert A., and William G. Chase. "Skill in Chess." American Scientist 61, no. 4 (1973): 394–403. Soleymanjahi, Saeed, Jack Huebner, Lina Elmansy, et al. "Artificial Intelligence–Assisted Colonoscopy for Polyp Detection: A Systematic Review and Meta-analysis." Annals of Internal Medicine 177, no. 12 (2024): 1652–1663. https://doi.org/10.7326/ANNALS-24-00981. Steyvers, Mark, Heliodoro Tejeda, Aakriti Kumar, et al. "What large language models know and what people think they know." Nature Machine Intelligence 7, no. 2 (2025): 221–231. https://doi.org/10.1038/s42256-024-00976-7.

Tetlock, Philip E., and Dan Gardner. Superforecasting: The Art and Science of Prediction. New York: Crown, 2015.

​

Product

Linkedin

Substack

Tesari Council

Partners

support@tesari.ai

© 2026 All rights reserved

Privacy Statement

Terms of Use

Security Policy

Apply for a Demo

Community Impact

News

Testimonials

TESARI INC

Texas, United States

ic_menu

Tesari

•

October 7 2026

Remaining Human in the Loop

The illusion of expertise endangers operations

October 29, 2026

Authors: Elisar Nurmagambet and Amy McFadden

​

The AI safety debate just exploded into the headlines. On September 8, Anthropic researcher Jacob Coxon resigned, accusing his employer and OpenAI of "gambling with our lives." Anthropic's alignment lead responded, putting the odds of catastrophe above ten percent. By the following weekend, Anthropic chief Dario Amodei had published an essay calling for the frontier labs to slow down, and OpenAI chief Sam Altman publicly agreed.

Meanwhile, an Iran-linked actor used Claude to assemble targeting handbooks on US naval movements. A cell in northern Yemen used Claude to write guidance software for missiles. An actor linked to the PLA's Academy of Military Sciences simulated strikes on twelve targets in Taiwan.

Every one of these stories is framed around the machine, its capabilities, and whether it can be contained. Anthropic has drawn a pair of red lines: no mass domestic surveillance and no weapon that selects and engages targets without a human making the final call.

However, militaries, corporations, and intelligence professionals must make consequential decisions, including targeting, under deep uncertainty and tight time constraints. This often sends practitioners reaching for the LLM to act quickly, which means walking through a minefield of cognitive biases. In essence, if practitioners want to be the human in the loop, they must add what the machine promises to relieve: intellectual rigor.

Prosthetic expertise

An artificial limb does not replace an arm or leg. It can perform many of the same tasks and in some cases enhance abilities, but the human being must learn to use the device, starting with understanding its shortcomings and its advantages. AI must be used the same way.

In Thinking, Fast and Slow, Daniel Kahneman outlines two systems of thinking for decision making. System 1 is fast, fluent, and unconscious. System 2 is slow, deliberate, and rigorous. Ideally, both systems check one another for clear and dynamic decision making. Consider chess masters who carry some 50,000 board patterns in memory, built up over years of study. This depth of understanding, assembled through rigorous System 2 work, allows them to deploy it rapidly in a blitz match.

Magnus Carlsen, the world’s top-ranked chess player, has described his own process this way: "I usually know what I am going to do after 10 seconds; the rest is double-checking." System 1 provides the choice right away. System 2 makes sure that it is correct.

Now, apply this process to high-level practitioners. Imagine an intelligence officer approving a targeting assessment in Iran. She has an IRGC naval facility packaged in front of her with reams of LLM-produced intelligence supporting a rapid strike. The clock is ticking. That same officer has not spent months watching the facility. She did not see indicators that the building had become a grammar school: the cessation of truck traffic coming at night, the removal of the antenna array in the spring, the posting of a school sign over the gate. The imagery and the reporting that the model was trained on stopped before the change. When the officer asked whether the target was still valid, the LLM produced three fluent paragraphs explaining why it was. Its output was optimized for a fast and easy “yes” rather than a hard and consequential “let's find out more.”

The human in the loop had not spent years learning 50,000 positions, but she felt as if she had. This is the problem. LLMs give the feeling of using System 2 thinking while circumventing it. Being a good human in the loop means using artificial intelligence to question and clarify, rather than to answer and act.

_________________________________________________________________________

​

About Tesari

Tesari AI, a transnational risk intelligence platform, is an autonomous AI platform that combines proprietary data access across the world's most challenging regions, encoded exclusive AI skills, workflows contributed by domain experts, and all-source analysis through hundreds of integrated tools. It is managed end-to-end by an agentic AI system to deliver expert-grade outcomes in minutes. Tesari is advised by industry experts and senior executives from the Department of Justice, the SEC, LexisNexis, Meta, Deutsche Bank, and the CIA, who bring decades of experience to address national security challenges and transnational policy. For more information visit www.tesari.ai.

The right tool for the right task

Machines are exceptional at many tasks. Some industries, like intelligence, defense, and medicine, can apply AI rapidly to good effect due to their repeatable problem sets and depth of training data. But knowing the tool means understanding the benefits and costs of using it as well as the tool’s strengths and limitations. Consider the colonoscopy and the mammogram. A 2024 meta-analysis of 44 randomized trials found that AI assistance raised adenoma detection by about a fifth. In mammography, the first randomized trial of AI-supported screening, covering more than 100,000 women in Sweden, found radiologists detected 29% more cancers with no rise in false positives, and a 44% cut in reading workload.

Any executive who can improve detection of critical information while reducing cognitive load understands that as an easy win. But there is another side to this story. In 2025, Budzyń and colleagues looked at four centers that had adopted the tool and asked what the endoscopists did when the AI was switched off. Their unassisted detection rate fell from 28.4% to 22.4%. The endoscopists had come to lean on the extraordinary abilities afforded by AI, while their organic experience had begun to atrophy.

Now consider your due-diligence team, your country-risk desk, your red cell. If they have been drafting with a model for 18 months, what does their unassisted judgment look like today? What would it look like in a constrained environment on a diminishing timeline? When AI is used to supply answers and drive actions, human judgment erodes in its absence. When it is used instead to sharpen questions and clarify thinking, the same expertise can be brought to bear faster and more effectively.

In a 2025 collaboration between Microsoft Research and Carnegie Mellon that surveyed 319 knowledge workers across 936 real tasks, the researchers found that the more a worker trusted the AI, the less critical thinking they engaged in. This might seem obvious, but the reverse also held. Workers who trusted their own judgment thought harder and treated the output as something to verify. A separate study of 666 participants found AI use negatively associated with critical-thinking performance because users were offloading more of their thinking to the technology. This practice was most pronounced among the youngest users, who need the most at-bats to train their System 2 thinking. Machines keep them in the dugout.

Intelligence professionals will recognize the problem and the fix. The intelligence community learned how much forecasting skill it was leaving on the table only when IARPA's Aggregative Contingent Estimation (ACE) program, run by Jason Matheny, spent four years keeping score: more than 20,000 forecasters made two million judgments, each graded against what actually happened. Calibration became apparent, and once it was visible it could be trained. But this was only possible because a baseline had been established. Most organizations adopting language models have no unassisted baseline, so whatever is happening to their people's judgment, they have no instrument that can detect it. These costs and limitations are interrelated.

Many AI companies approach problems from the capabilities of their technology rather than from deep knowledge of the domains in which it will operate. However, in complex fields such as intelligence, defense, and medicine, it is more important to know what should be automated rather than what can be automated, and most of all where human judgment must remain decisive. The best engineering teams cannot substitute for the accumulated expertise required to understand the problem, recognize when the machine is wrong, and know when not to use it. AI therefore makes deep domain expertise more important, not less.

 

The right tool for the right task

Machines are exceptional at many tasks. Some industries, like intelligence, defense, and medicine, can apply AI rapidly to good effect due to their repeatable problem sets and depth of training data. But knowing the tool means understanding the benefits and costs of using it as well as the tool’s strengths and limitations. Consider the colonoscopy and the mammogram. A 2024 meta-analysis of 44 randomized trials found that AI assistance raised adenoma detection by about a fifth. In mammography, the first randomized trial of AI-supported screening, covering more than 100,000 women in Sweden, found radiologists detected 29% more cancers with no rise in false positives, and a 44% cut in reading workload.

Any executive who can improve detection of critical information while reducing cognitive load understands that as an easy win. But there is another side to this story. In 2025, Budzyń and colleagues looked at four centers that had adopted the tool and asked what the endoscopists did when the AI was switched off. Their unassisted detection rate fell from 28.4% to 22.4%. The endoscopists had come to lean on the extraordinary abilities afforded by AI, while their organic experience had begun to atrophy.

Now consider your due-diligence team, your country-risk desk, your red cell. If they have been drafting with a model for 18 months, what does their unassisted judgment look like today? What would it look like in a constrained environment on a diminishing timeline? When AI is used to supply answers and drive actions, human judgment erodes in its absence. When it is used instead to sharpen questions and clarify thinking, the same expertise can be brought to bear faster and more effectively.

In a 2025 collaboration between Microsoft Research and Carnegie Mellon that surveyed 319 knowledge workers across 936 real tasks, the researchers found that the more a worker trusted the AI, the less critical thinking they engaged in. This might seem obvious, but the reverse also held. Workers who trusted their own judgment thought harder and treated the output as something to verify. A separate study of 666 participants found AI use negatively associated with critical-thinking performance because users were offloading more of their thinking to the technology. This practice was most pronounced among the youngest users, who need the most at-bats to train their System 2 thinking. Machines keep them in the dugout.

Intelligence professionals will recognize the problem and the fix. The intelligence community learned how much forecasting skill it was leaving on the table only when IARPA's Aggregative Contingent Estimation (ACE) program, run by Jason Matheny, spent four years keeping score: more than 20,000 forecasters made two million judgments, each graded against what actually happened. Calibration became apparent, and once it was visible it could be trained. But this was only possible because a baseline had been established. Most organizations adopting language models have no unassisted baseline, so whatever is happening to their people's judgment, they have no instrument that can detect it. These costs and limitations are interrelated.

Many AI companies approach problems from the capabilities of their technology rather than from deep knowledge of the domains in which it will operate. However, in complex fields such as intelligence, defense, and medicine, it is more important to know what should be automated rather than what can be automated, and most of all where human judgment must remain decisive. The best engineering teams cannot substitute for the accumulated expertise required to understand the problem, recognize when the machine is wrong, and know when not to use it. AI therefore makes deep domain expertise more important, not less.

 

Confidence you did not earn

False confidence hits the bottom line and could cost lives. In 2025, a team published a study of 991 forecasters in ACM Transactions on Interactive Intelligent Systems. Participants were fitted with one of two different prosthetics. The first was an LLM assistant prompted to reason like a superforecaster: weigh base rates, express calibrated uncertainty, resist the confident guess. The second was prompted to do the opposite, to be overconfident and to ignore base rates entirely. Both groups improved their forecasting accuracy over a control group, and by a comparable margin. The bad prosthetic worked almost as well as the good one, because the forecasters were not evaluating the evidence it offered. The most dangerous part is that they felt as if they were arriving at the answer organically.

The mechanism is now well documented. In 2025, cognitive-science researchers found that human confidence in a model's answer tracks the length and fluency of its explanation rather than its accuracy. Essentially, the more the machine wrote, the more convincing it was. Other researchers found the same thing in 2021, before the current generation of models: adding explanations to AI output increased human reliance. A more recent study, in 2024, described this phenomenon as “an illusion of understanding,” in which people feel they know more than they actually do.

One plausible reason is that a lengthy explanation overwhelms the reader, and they accept spilled digital ink over their earned intuition. This reflects a leadership liability; when decision makers are surrounded by convincing yes men, they tend to make poor choices. When those yes men live in their cellphones and offer the simulacrum of understanding at a moment's notice, even questionable judgments can look well-founded. As Richard Heuer indicated in Psychology of Intelligence Analysis, the mental model you bring to the evidence determines what you see in it. LLMs are optimized for interaction and not incision. That is why they will not only confirm a decision maker's mental model, but eventually shape and potentially displace it, becoming less an aid to judgment than a cognitive prosthesis. Stopping this effect is the responsibility of the human in the loop.

Confidence you did not earn

False confidence hits the bottom line and could cost lives. In 2025, a team published a study of 991 forecasters in ACM Transactions on Interactive Intelligent Systems. Participants were fitted with one of two different prosthetics. The first was an LLM assistant prompted to reason like a superforecaster: weigh base rates, express calibrated uncertainty, resist the confident guess. The second was prompted to do the opposite, to be overconfident and to ignore base rates entirely. Both groups improved their forecasting accuracy over a control group, and by a comparable margin. The bad prosthetic worked almost as well as the good one, because the forecasters were not evaluating the evidence it offered. The most dangerous part is that they felt as if they were arriving at the answer organically.

The mechanism is now well documented. In 2025, cognitive-science researchers found that human confidence in a model's answer tracks the length and fluency of its explanation rather than its accuracy. Essentially, the more the machine wrote, the more convincing it was. Other researchers found the same thing in 2021, before the current generation of models: adding explanations to AI output increased human reliance. A more recent study, in 2024, described this phenomenon as “an illusion of understanding,” in which people feel they know more than they actually do.

One plausible reason is that a lengthy explanation overwhelms the reader, and they accept spilled digital ink over their earned intuition. This reflects a leadership liability; when decision makers are surrounded by convincing yes men, they tend to make poor choices. When those yes men live in their cellphones and offer the simulacrum of understanding at a moment's notice, even questionable judgments can look well-founded. As Richard Heuer indicated in Psychology of Intelligence Analysis, the mental model you bring to the evidence determines what you see in it. LLMs are optimized for interaction and not incision. That is why they will not only confirm a decision maker's mental model, but eventually shape and potentially displace it, becoming less an aid to judgment than a cognitive prosthesis. Stopping this effect is the responsibility of the human in the loop.

How to be the “human in the loop”

Learning how to use LLMs to enhance decision makers' speed and accuracy means practicing with the prosthetic rather than relying on it. Three practices can ensure that the human in the loop remains more than a rubber stamp and preserves cognitive skills once the device is switched off.

 

1.     Score the unassisted baseline. Most organizations have rushed to leverage AI as quickly as possible with the promise of lowering costs and increasing output. Many of them never understood what their organization looked like beforehand. Brier scores and tool-free exercises can identify the true talents of working groups. If you cannot measure judgment without the machine, you cannot see it decay. Regularly turn off the bots and require teams to work from first principles.

2.     Build in friction. Thinking should remain difficult. That is how you know that System 2 is functioning. A 2021 study found that requiring people to commit to a view before seeing AI output substantially reduced overreliance. Decision makers at all levels should know what unassisted cognition feels like. The team that prepares for a future where digital consideration is turned off or corrupted will retain decision advantage.

3.     Focus on questions and clarifications, not answers and actions. Use the model to sharpen the question, generate the alternative hypothesis, and surface what you have not considered. Then turn it against your own conclusion. This is the machine version of Heuer's Analysis of Competing Hypotheses, done at speed, which preserves the analytic discipline that keeps judgment sharp while exposing disagreement.

How to be the “human in the loop”

Learning how to use LLMs to enhance decision makers' speed and accuracy means practicing with the prosthetic rather than relying on it. Three practices can ensure that the human in the loop remains more than a rubber stamp and preserves cognitive skills once the device is switched off.

 

1.     Score the unassisted baseline. Most organizations have rushed to leverage AI as quickly as possible with the promise of lowering costs and increasing output. Many of them never understood what their organization looked like beforehand. Brier scores and tool-free exercises can identify the true talents of working groups. If you cannot measure judgment without the machine, you cannot see it decay. Regularly turn off the bots and require teams to work from first principles.

2.     Build in friction. Thinking should remain difficult. That is how you know that System 2 is functioning. A 2021 study found that requiring people to commit to a view before seeing AI output substantially reduced overreliance. Decision makers at all levels should know what unassisted cognition feels like. The team that prepares for a future where digital consideration is turned off or corrupted will retain decision advantage.

3.     Focus on questions and clarifications, not answers and actions. Use the model to sharpen the question, generate the alternative hypothesis, and surface what you have not considered. Then turn it against your own conclusion. This is the machine version of Heuer's Analysis of Competing Hypotheses, done at speed, which preserves the analytic discipline that keeps judgment sharp while exposing disagreement.

The implementation of AI by Industry: Use Case of Arcanum and Tesari AI

The intelligence market is responding to AI development in three major ways. First, legacy compliance firms are introducing AI internally to perform faster routine checks. Top-layer data surfaces obvious red flags and sorts entities into two buckets: safe and flagged. This approach is as old as the compliance industry itself. Yet the shift is underway, moving from box-checking toward deep vetting of every related entity, including vendors, clients, partners, owners, and investors.

Secondly, legacy consulting and professional services firms like Booz Allen and the Big Four are partnering with major AI labs such as OpenAI and Anthropic. This direction is driven not by an interest in expanding what organizations can know but by the goal of automating workflows and operational systems.

The third response is the most ambitious and arguably the most in demand: using AI to create an entirely new category of intelligence and investigative capability. Private intelligence firms, national security agencies, global capital, and other organizations operating in high-stakes environments are seeking solutions that scale their capabilities to enter new markets, gain visibility into more regions, and reduce the delay and blind spots that have defined this work until now. This approach is the only one positioned to withstand the global shifts already reshaping economics, geopolitics, capital movements, trade, logistics, and finance.

One timely example of the third response is the September 2026 partnership between Arcanum Global, a government-grade intelligence firm founded by veterans of America's elite military and intelligence units, and Tesari AI, a transnational risk intelligence platform built for high-stakes environments. Arcanum holds that the value of intelligence ultimately depends on deep human expertise: knowing which questions to ask, how to interpret what is found, and what it means for the decision at hand. Tesari's role in this partnership is to multiply the reach behind that expertise: analysts, on-the-ground agents, translators, and regional specialists, extended across far more ground than any team could cover manually.

The result is a process reliable and fast enough to keep pace with current national security requirements and rising global tension, while the judgment itself stays where it belongs, with the people who know how to read what the intelligence means. Partnerships like this one keep the human at the helm, where the decisions and the judgment still belong.

References:

Axios. "Anthropic, OpenAI CEOs call for slowdown in AI development." 12 September 2026. https://www.axios.com/2026/09/12/anthropic-ai-amodei-pacing.

Axios. "Anthropic report: 5 ways Claude was exploited for war, spying and repression." 12 September 2026. https://www.axios.com/2026/09/12/anthropic-ai-threat-report-russia-iran-china.

Axios. "Exclusive: Pentagon threatens to cut off Anthropic in AI safeguards dispute." 15 February 2026. https://www.axios.com/2026/02/15/claude-pentagon-anthropic-contract-maduro.

Bansal, Gagan, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. "Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance." In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2021. https://doi.org/10.1145/3411764.3445717.

Buçinca, Zana, Maja Barbara Malaya, and Krzysztof Z. Gajos. "To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making." Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021): article 188. https://doi.org/10.1145/3449287.

Budzyń, Krzysztof, Marcin Romańczyk, Diana Kitala, et al. "Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study." The Lancet Gastroenterology & Hepatology 10, no. 10 (2025): 896–903. https://doi.org/10.1016/S2468-1253(25)00133-5.

CNBC. "'Extinction' warnings ramp up as more OpenAI, Anthropic researchers join calls for an AI slowdown." 10 September 2026. https://www.cnbc.com/2026/09/10/openai-anthropic-ai-safety-slowdown-extinction.html.

Euronews. "AI on the battlefield: How is the US integrating AI into its military?" 6 March 2026. https://www.euronews.com/next/2026/03/06/ai-on-the-battlefield-how-is-the-us-integrating-ai-into-its-military.

Gerlich, Michael. "AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking." Societies 15, no. 1 (2025): 6. https://doi.org/10.3390/soc15010006.

Hernström, Veronica, Viktoria Josefsson, Hanna Sartor, et al. "Screening performance and characteristics of breast cancer detected in the Mammography Screening with Artificial Intelligence trial (MASAI): a randomised, controlled, parallel-group, non-inferiority, single-blinded, screening accuracy study." The Lancet Digital Health 7, no. 3 (2025): e175–e183. https://doi.org/10.1016/S2589-7500(24)00267-X.

Heuer, Richards J., Jr. Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency, 1999. IARPA. "Aggregative Contingent Estimation (ACE)." https://www.iarpa.gov/research- programs/ace. IEEE Spectrum. "IARPA's New Director Wants You to Surprise Him." 2015. https://spectrum.ieee.org/iarpas-new-director-wants-you-to-surprise-him.

Kahneman, Daniel. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011.

Lång, Kristina, Viktoria Josefsson, Anna-Maria Larsson, et al. "Artificial intelligence-supported screen reading versus standard double reading in the Mammography Screening with Artificial Intelligence trial (MASAI): a clinical safety analysis of a randomised, controlled, non-inferiority, single-blinded, screening accuracy study." The Lancet Oncology 24, no. 8 (2023): 936–944. https://doi.org/10.1016/S1470-2045(23)00298-X.

Lee, Hao-Ping (Hank), Advait Sarkar, Lev Tankelevitch, Ian Drosos, Sean Rintel, Richard Banks, and Nicholas Wilson. "The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers." In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2025. https://doi.org/10.1145/3706598.3713778.

Messeri, Lisa, and M. J. Crockett. "Artificial intelligence and illusions of understanding in scientific research." Nature 627 (2024): 49–58. https://doi.org/10.1038/s41586-024-07146-0. Reuters. "Factbox: How Anthropic says Claude was used for weapons, spying and cyber operations." 11 September 2026. https://www.usnews.com/news/world/articles/2026-09-11/factbox-how-anthropic-says-claude-was-used-for-weapons-spying-and-cyber-operations.

Schoenegger, Philipp, Peter S. Park, Ezra Karger, Sean Trott, and Philip E. Tetlock. "AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy." ACM Transactions on Interactive Intelligent Systems 15, no. 1 (2025): article 4. https://doi.org/10.1145/3707649.

Simon, Herbert A., and William G. Chase. "Skill in Chess." American Scientist 61, no. 4 (1973): 394–403. Soleymanjahi, Saeed, Jack Huebner, Lina Elmansy, et al. "Artificial Intelligence–Assisted Colonoscopy for Polyp Detection: A Systematic Review and Meta-analysis." Annals of Internal Medicine 177, no. 12 (2024): 1652–1663. https://doi.org/10.7326/ANNALS-24-00981. Steyvers, Mark, Heliodoro Tejeda, Aakriti Kumar, et al. "What large language models know and what people think they know." Nature Machine Intelligence 7, no. 2 (2025): 221–231. https://doi.org/10.1038/s42256-024-00976-7.

Tetlock, Philip E., and Dan Gardner. Superforecasting: The Art and Science of Prediction. New York: Crown, 2015.

​

Product

Linkedin

Substack

Tesari Council

Partners

support@tesari.ai

© 2026 All rights reserved

Privacy Statement

Terms of Use

Security Policy

Apply for a Demo

Community Impact

News

Testimonials

TESARI INC

Texas, United States

ic_menu

Tesari

•

October 7 2026

Remaining Human in the Loop

The illusion of expertise endangers operations

October 29, 2026

Authors: Elisar Nurmagambet and Amy McFadden

​

The AI safety debate just exploded into the headlines. On September 8, Anthropic researcher Jacob Coxon resigned, accusing his employer and OpenAI of "gambling with our lives." Anthropic's alignment lead responded, putting the odds of catastrophe above ten percent. By the following weekend, Anthropic chief Dario Amodei had published an essay calling for the frontier labs to slow down, and OpenAI chief Sam Altman publicly agreed.

Meanwhile, an Iran-linked actor used Claude to assemble targeting handbooks on US naval movements. A cell in northern Yemen used Claude to write guidance software for missiles. An actor linked to the PLA's Academy of Military Sciences simulated strikes on twelve targets in Taiwan.

Every one of these stories is framed around the machine, its capabilities, and whether it can be contained. Anthropic has drawn a pair of red lines: no mass domestic surveillance and no weapon that selects and engages targets without a human making the final call.

However, militaries, corporations, and intelligence professionals must make consequential decisions, including targeting, under deep uncertainty and tight time constraints. This often sends practitioners reaching for the LLM to act quickly, which means walking through a minefield of cognitive biases. In essence, if practitioners want to be the human in the loop, they must add what the machine promises to relieve: intellectual rigor.

Prosthetic expertise

An artificial limb does not replace an arm or leg. It can perform many of the same tasks and in some cases enhance abilities, but the human being must learn to use the device, starting with understanding its shortcomings and its advantages. AI must be used the same way.

In Thinking, Fast and Slow, Daniel Kahneman outlines two systems of thinking for decision making. System 1 is fast, fluent, and unconscious. System 2 is slow, deliberate, and rigorous. Ideally, both systems check one another for clear and dynamic decision making. Consider chess masters who carry some 50,000 board patterns in memory, built up over years of study. This depth of understanding, assembled through rigorous System 2 work, allows them to deploy it rapidly in a blitz match.

Magnus Carlsen, the world’s top-ranked chess player, has described his own process this way: "I usually know what I am going to do after 10 seconds; the rest is double-checking." System 1 provides the choice right away. System 2 makes sure that it is correct.

Now, apply this process to high-level practitioners. Imagine an intelligence officer approving a targeting assessment in Iran. She has an IRGC naval facility packaged in front of her with reams of LLM-produced intelligence supporting a rapid strike. The clock is ticking. That same officer has not spent months watching the facility. She did not see indicators that the building had become a grammar school: the cessation of truck traffic coming at night, the removal of the antenna array in the spring, the posting of a school sign over the gate. The imagery and the reporting that the model was trained on stopped before the change. When the officer asked whether the target was still valid, the LLM produced three fluent paragraphs explaining why it was. Its output was optimized for a fast and easy “yes” rather than a hard and consequential “let's find out more.”

The human in the loop had not spent years learning 50,000 positions, but she felt as if she had. This is the problem. LLMs give the feeling of using System 2 thinking while circumventing it. Being a good human in the loop means using artificial intelligence to question and clarify, rather than to answer and act.

The right tool for the right task

Machines are exceptional at many tasks. Some industries, like intelligence, defense, and medicine, can apply AI rapidly to good effect due to their repeatable problem sets and depth of training data. But knowing the tool means understanding the benefits and costs of using it as well as the tool’s strengths and limitations. Consider the colonoscopy and the mammogram. A 2024 meta-analysis of 44 randomized trials found that AI assistance raised adenoma detection by about a fifth. In mammography, the first randomized trial of AI-supported screening, covering more than 100,000 women in Sweden, found radiologists detected 29% more cancers with no rise in false positives, and a 44% cut in reading workload.

Any executive who can improve detection of critical information while reducing cognitive load understands that as an easy win. But there is another side to this story. In 2025, Budzyń and colleagues looked at four centers that had adopted the tool and asked what the endoscopists did when the AI was switched off. Their unassisted detection rate fell from 28.4% to 22.4%. The endoscopists had come to lean on the extraordinary abilities afforded by AI, while their organic experience had begun to atrophy.

Now consider your due-diligence team, your country-risk desk, your red cell. If they have been drafting with a model for 18 months, what does their unassisted judgment look like today? What would it look like in a constrained environment on a diminishing timeline? When AI is used to supply answers and drive actions, human judgment erodes in its absence. When it is used instead to sharpen questions and clarify thinking, the same expertise can be brought to bear faster and more effectively.

In a 2025 collaboration between Microsoft Research and Carnegie Mellon that surveyed 319 knowledge workers across 936 real tasks, the researchers found that the more a worker trusted the AI, the less critical thinking they engaged in. This might seem obvious, but the reverse also held. Workers who trusted their own judgment thought harder and treated the output as something to verify. A separate study of 666 participants found AI use negatively associated with critical-thinking performance because users were offloading more of their thinking to the technology. This practice was most pronounced among the youngest users, who need the most at-bats to train their System 2 thinking. Machines keep them in the dugout.

Intelligence professionals will recognize the problem and the fix. The intelligence community learned how much forecasting skill it was leaving on the table only when IARPA's Aggregative Contingent Estimation (ACE) program, run by Jason Matheny, spent four years keeping score: more than 20,000 forecasters made two million judgments, each graded against what actually happened. Calibration became apparent, and once it was visible it could be trained. But this was only possible because a baseline had been established. Most organizations adopting language models have no unassisted baseline, so whatever is happening to their people's judgment, they have no instrument that can detect it. These costs and limitations are interrelated.

Many AI companies approach problems from the capabilities of their technology rather than from deep knowledge of the domains in which it will operate. However, in complex fields such as intelligence, defense, and medicine, it is more important to know what should be automated rather than what can be automated, and most of all where human judgment must remain decisive. The best engineering teams cannot substitute for the accumulated expertise required to understand the problem, recognize when the machine is wrong, and know when not to use it. AI therefore makes deep domain expertise more important, not less.

 

The right tool for the right task

Machines are exceptional at many tasks. Some industries, like intelligence, defense, and medicine, can apply AI rapidly to good effect due to their repeatable problem sets and depth of training data. But knowing the tool means understanding the benefits and costs of using it as well as the tool’s strengths and limitations. Consider the colonoscopy and the mammogram. A 2024 meta-analysis of 44 randomized trials found that AI assistance raised adenoma detection by about a fifth. In mammography, the first randomized trial of AI-supported screening, covering more than 100,000 women in Sweden, found radiologists detected 29% more cancers with no rise in false positives, and a 44% cut in reading workload.

Any executive who can improve detection of critical information while reducing cognitive load understands that as an easy win. But there is another side to this story. In 2025, Budzyń and colleagues looked at four centers that had adopted the tool and asked what the endoscopists did when the AI was switched off. Their unassisted detection rate fell from 28.4% to 22.4%. The endoscopists had come to lean on the extraordinary abilities afforded by AI, while their organic experience had begun to atrophy.

Now consider your due-diligence team, your country-risk desk, your red cell. If they have been drafting with a model for 18 months, what does their unassisted judgment look like today? What would it look like in a constrained environment on a diminishing timeline? When AI is used to supply answers and drive actions, human judgment erodes in its absence. When it is used instead to sharpen questions and clarify thinking, the same expertise can be brought to bear faster and more effectively.

In a 2025 collaboration between Microsoft Research and Carnegie Mellon that surveyed 319 knowledge workers across 936 real tasks, the researchers found that the more a worker trusted the AI, the less critical thinking they engaged in. This might seem obvious, but the reverse also held. Workers who trusted their own judgment thought harder and treated the output as something to verify. A separate study of 666 participants found AI use negatively associated with critical-thinking performance because users were offloading more of their thinking to the technology. This practice was most pronounced among the youngest users, who need the most at-bats to train their System 2 thinking. Machines keep them in the dugout.

Intelligence professionals will recognize the problem and the fix. The intelligence community learned how much forecasting skill it was leaving on the table only when IARPA's Aggregative Contingent Estimation (ACE) program, run by Jason Matheny, spent four years keeping score: more than 20,000 forecasters made two million judgments, each graded against what actually happened. Calibration became apparent, and once it was visible it could be trained. But this was only possible because a baseline had been established. Most organizations adopting language models have no unassisted baseline, so whatever is happening to their people's judgment, they have no instrument that can detect it. These costs and limitations are interrelated.

Many AI companies approach problems from the capabilities of their technology rather than from deep knowledge of the domains in which it will operate. However, in complex fields such as intelligence, defense, and medicine, it is more important to know what should be automated rather than what can be automated, and most of all where human judgment must remain decisive. The best engineering teams cannot substitute for the accumulated expertise required to understand the problem, recognize when the machine is wrong, and know when not to use it. AI therefore makes deep domain expertise more important, not less.

 

Confidence you did not earn

False confidence hits the bottom line and could cost lives. In 2025, a team published a study of 991 forecasters in ACM Transactions on Interactive Intelligent Systems. Participants were fitted with one of two different prosthetics. The first was an LLM assistant prompted to reason like a superforecaster: weigh base rates, express calibrated uncertainty, resist the confident guess. The second was prompted to do the opposite, to be overconfident and to ignore base rates entirely. Both groups improved their forecasting accuracy over a control group, and by a comparable margin. The bad prosthetic worked almost as well as the good one, because the forecasters were not evaluating the evidence it offered. The most dangerous part is that they felt as if they were arriving at the answer organically.

The mechanism is now well documented. In 2025, cognitive-science researchers found that human confidence in a model's answer tracks the length and fluency of its explanation rather than its accuracy. Essentially, the more the machine wrote, the more convincing it was. Other researchers found the same thing in 2021, before the current generation of models: adding explanations to AI output increased human reliance. A more recent study, in 2024, described this phenomenon as “an illusion of understanding,” in which people feel they know more than they actually do.

One plausible reason is that a lengthy explanation overwhelms the reader, and they accept spilled digital ink over their earned intuition. This reflects a leadership liability; when decision makers are surrounded by convincing yes men, they tend to make poor choices. When those yes men live in their cellphones and offer the simulacrum of understanding at a moment's notice, even questionable judgments can look well-founded. As Richard Heuer indicated in Psychology of Intelligence Analysis, the mental model you bring to the evidence determines what you see in it. LLMs are optimized for interaction and not incision. That is why they will not only confirm a decision maker's mental model, but eventually shape and potentially displace it, becoming less an aid to judgment than a cognitive prosthesis. Stopping this effect is the responsibility of the human in the loop.

Confidence you did not earn

False confidence hits the bottom line and could cost lives. In 2025, a team published a study of 991 forecasters in ACM Transactions on Interactive Intelligent Systems. Participants were fitted with one of two different prosthetics. The first was an LLM assistant prompted to reason like a superforecaster: weigh base rates, express calibrated uncertainty, resist the confident guess. The second was prompted to do the opposite, to be overconfident and to ignore base rates entirely. Both groups improved their forecasting accuracy over a control group, and by a comparable margin. The bad prosthetic worked almost as well as the good one, because the forecasters were not evaluating the evidence it offered. The most dangerous part is that they felt as if they were arriving at the answer organically.

The mechanism is now well documented. In 2025, cognitive-science researchers found that human confidence in a model's answer tracks the length and fluency of its explanation rather than its accuracy. Essentially, the more the machine wrote, the more convincing it was. Other researchers found the same thing in 2021, before the current generation of models: adding explanations to AI output increased human reliance. A more recent study, in 2024, described this phenomenon as “an illusion of understanding,” in which people feel they know more than they actually do.

One plausible reason is that a lengthy explanation overwhelms the reader, and they accept spilled digital ink over their earned intuition. This reflects a leadership liability; when decision makers are surrounded by convincing yes men, they tend to make poor choices. When those yes men live in their cellphones and offer the simulacrum of understanding at a moment's notice, even questionable judgments can look well-founded. As Richard Heuer indicated in Psychology of Intelligence Analysis, the mental model you bring to the evidence determines what you see in it. LLMs are optimized for interaction and not incision. That is why they will not only confirm a decision maker's mental model, but eventually shape and potentially displace it, becoming less an aid to judgment than a cognitive prosthesis. Stopping this effect is the responsibility of the human in the loop.

How to be the “human in the loop”

Learning how to use LLMs to enhance decision makers' speed and accuracy means practicing with the prosthetic rather than relying on it. Three practices can ensure that the human in the loop remains more than a rubber stamp and preserves cognitive skills once the device is switched off.

 

1.     Score the unassisted baseline. Most organizations have rushed to leverage AI as quickly as possible with the promise of lowering costs and increasing output. Many of them never understood what their organization looked like beforehand. Brier scores and tool-free exercises can identify the true talents of working groups. If you cannot measure judgment without the machine, you cannot see it decay. Regularly turn off the bots and require teams to work from first principles.

2.     Build in friction. Thinking should remain difficult. That is how you know that System 2 is functioning. A 2021 study found that requiring people to commit to a view before seeing AI output substantially reduced overreliance. Decision makers at all levels should know what unassisted cognition feels like. The team that prepares for a future where digital consideration is turned off or corrupted will retain decision advantage.

3.     Focus on questions and clarifications, not answers and actions. Use the model to sharpen the question, generate the alternative hypothesis, and surface what you have not considered. Then turn it against your own conclusion. This is the machine version of Heuer's Analysis of Competing Hypotheses, done at speed, which preserves the analytic discipline that keeps judgment sharp while exposing disagreement.

How to be the “human in the loop”

Learning how to use LLMs to enhance decision makers' speed and accuracy means practicing with the prosthetic rather than relying on it. Three practices can ensure that the human in the loop remains more than a rubber stamp and preserves cognitive skills once the device is switched off.

 

1.     Score the unassisted baseline. Most organizations have rushed to leverage AI as quickly as possible with the promise of lowering costs and increasing output. Many of them never understood what their organization looked like beforehand. Brier scores and tool-free exercises can identify the true talents of working groups. If you cannot measure judgment without the machine, you cannot see it decay. Regularly turn off the bots and require teams to work from first principles.

2.     Build in friction. Thinking should remain difficult. That is how you know that System 2 is functioning. A 2021 study found that requiring people to commit to a view before seeing AI output substantially reduced overreliance. Decision makers at all levels should know what unassisted cognition feels like. The team that prepares for a future where digital consideration is turned off or corrupted will retain decision advantage.

3.     Focus on questions and clarifications, not answers and actions. Use the model to sharpen the question, generate the alternative hypothesis, and surface what you have not considered. Then turn it against your own conclusion. This is the machine version of Heuer's Analysis of Competing Hypotheses, done at speed, which preserves the analytic discipline that keeps judgment sharp while exposing disagreement.

The implementation of AI by Industry: Use Case of Arcanum and Tesari AI

The intelligence market is responding to AI development in three major ways. First, legacy compliance firms are introducing AI internally to perform faster routine checks. Top-layer data surfaces obvious red flags and sorts entities into two buckets: safe and flagged. This approach is as old as the compliance industry itself. Yet the shift is underway, moving from box-checking toward deep vetting of every related entity, including vendors, clients, partners, owners, and investors.

Secondly, legacy consulting and professional services firms like Booz Allen and the Big Four are partnering with major AI labs such as OpenAI and Anthropic. This direction is driven not by an interest in expanding what organizations can know but by the goal of automating workflows and operational systems.

The third response is the most ambitious and arguably the most in demand: using AI to create an entirely new category of intelligence and investigative capability. Private intelligence firms, national security agencies, global capital, and other organizations operating in high-stakes environments are seeking solutions that scale their capabilities to enter new markets, gain visibility into more regions, and reduce the delay and blind spots that have defined this work until now. This approach is the only one positioned to withstand the global shifts already reshaping economics, geopolitics, capital movements, trade, logistics, and finance.

One timely example of the third response is the September 2026 partnership between Arcanum Global, a government-grade intelligence firm founded by veterans of America's elite military and intelligence units, and Tesari AI, a transnational risk intelligence platform built for high-stakes environments. Arcanum holds that the value of intelligence ultimately depends on deep human expertise: knowing which questions to ask, how to interpret what is found, and what it means for the decision at hand. Tesari's role in this partnership is to multiply the reach behind that expertise: analysts, on-the-ground agents, translators, and regional specialists, extended across far more ground than any team could cover manually.

The result is a process reliable and fast enough to keep pace with current national security requirements and rising global tension, while the judgment itself stays where it belongs, with the people who know how to read what the intelligence means. Partnerships like this one keep the human at the helm, where the decisions and the judgment still belong.

References:

Axios. "Anthropic, OpenAI CEOs call for slowdown in AI development." 12 September 2026. https://www.axios.com/2026/09/12/anthropic-ai-amodei-pacing.

Axios. "Anthropic report: 5 ways Claude was exploited for war, spying and repression." 12 September 2026. https://www.axios.com/2026/09/12/anthropic-ai-threat-report-russia-iran-china.

Axios. "Exclusive: Pentagon threatens to cut off Anthropic in AI safeguards dispute." 15 February 2026. https://www.axios.com/2026/02/15/claude-pentagon-anthropic-contract-maduro.

Bansal, Gagan, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. "Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance." In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2021. https://doi.org/10.1145/3411764.3445717.

Buçinca, Zana, Maja Barbara Malaya, and Krzysztof Z. Gajos. "To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making." Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021): article 188. https://doi.org/10.1145/3449287.

Budzyń, Krzysztof, Marcin Romańczyk, Diana Kitala, et al. "Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study." The Lancet Gastroenterology & Hepatology 10, no. 10 (2025): 896–903. https://doi.org/10.1016/S2468-1253(25)00133-5.

CNBC. "'Extinction' warnings ramp up as more OpenAI, Anthropic researchers join calls for an AI slowdown." 10 September 2026. https://www.cnbc.com/2026/09/10/openai-anthropic-ai-safety-slowdown-extinction.html.

Euronews. "AI on the battlefield: How is the US integrating AI into its military?" 6 March 2026. https://www.euronews.com/next/2026/03/06/ai-on-the-battlefield-how-is-the-us-integrating-ai-into-its-military.

Gerlich, Michael. "AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking." Societies 15, no. 1 (2025): 6. https://doi.org/10.3390/soc15010006.

Hernström, Veronica, Viktoria Josefsson, Hanna Sartor, et al. "Screening performance and characteristics of breast cancer detected in the Mammography Screening with Artificial Intelligence trial (MASAI): a randomised, controlled, parallel-group, non-inferiority, single-blinded, screening accuracy study." The Lancet Digital Health 7, no. 3 (2025): e175–e183. https://doi.org/10.1016/S2589-7500(24)00267-X.

Heuer, Richards J., Jr. Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency, 1999. IARPA. "Aggregative Contingent Estimation (ACE)." https://www.iarpa.gov/research- programs/ace. IEEE Spectrum. "IARPA's New Director Wants You to Surprise Him." 2015. https://spectrum.ieee.org/iarpas-new-director-wants-you-to-surprise-him.

Kahneman, Daniel. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011.

Lång, Kristina, Viktoria Josefsson, Anna-Maria Larsson, et al. "Artificial intelligence-supported screen reading versus standard double reading in the Mammography Screening with Artificial Intelligence trial (MASAI): a clinical safety analysis of a randomised, controlled, non-inferiority, single-blinded, screening accuracy study." The Lancet Oncology 24, no. 8 (2023): 936–944. https://doi.org/10.1016/S1470-2045(23)00298-X.

Lee, Hao-Ping (Hank), Advait Sarkar, Lev Tankelevitch, Ian Drosos, Sean Rintel, Richard Banks, and Nicholas Wilson. "The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers." In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2025. https://doi.org/10.1145/3706598.3713778.

Messeri, Lisa, and M. J. Crockett. "Artificial intelligence and illusions of understanding in scientific research." Nature 627 (2024): 49–58. https://doi.org/10.1038/s41586-024-07146-0. Reuters. "Factbox: How Anthropic says Claude was used for weapons, spying and cyber operations." 11 September 2026. https://www.usnews.com/news/world/articles/2026-09-11/factbox-how-anthropic-says-claude-was-used-for-weapons-spying-and-cyber-operations.

Schoenegger, Philipp, Peter S. Park, Ezra Karger, Sean Trott, and Philip E. Tetlock. "AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy." ACM Transactions on Interactive Intelligent Systems 15, no. 1 (2025): article 4. https://doi.org/10.1145/3707649.

Simon, Herbert A., and William G. Chase. "Skill in Chess." American Scientist 61, no. 4 (1973): 394–403. Soleymanjahi, Saeed, Jack Huebner, Lina Elmansy, et al. "Artificial Intelligence–Assisted Colonoscopy for Polyp Detection: A Systematic Review and Meta-analysis." Annals of Internal Medicine 177, no. 12 (2024): 1652–1663. https://doi.org/10.7326/ANNALS-24-00981. Steyvers, Mark, Heliodoro Tejeda, Aakriti Kumar, et al. "What large language models know and what people think they know." Nature Machine Intelligence 7, no. 2 (2025): 221–231. https://doi.org/10.1038/s42256-024-00976-7.

Tetlock, Philip E., and Dan Gardner. Superforecasting: The Art and Science of Prediction. New York: Crown, 2015.

​

Product

Privacy Statement

Terms of Use

Security Policy

Linkedin

Substack

Tesari Council

Newsroom

Partners

Community Impact

Community Impact

Apply for a Demo

© 2026 All rights reserved

support@tesari.ai

Testimonials

TESARI INC

Texas, United States

​

Log in

Apply for a Demo