PME means professional military education. The Classroom version is also available for high school.
The owner of a print-and-internet shop in Nairobi asks an AI assistant two things: how to keep printing through power blackouts, and how to win back customers from a new shop nearby. They share two supplier quotes and the new shop's flyer, but not their own sales records. They check the first answer, find it right, and act on both. What can happen next is uncalibrated reliance: confident output is accepted without knowing whether the task fits the tool. Later a reviewer, say a lender or business adviser, sees only the plan and asks about each part. Illustration based on Otis et al. (2026) and its working paper. The owners, their documents and every number in their plans are constructed.
An animated figure. A red dot stands for the owner of a print shop, with an example task in two parts: keep printing in blackouts, and win back customers. The owner has four documents: two battery quotes, a rival's flyer and their own sales records. They give the AI copies of the quotes and the flyer, not the sales records. The AI answers both in the same form, in grey, each marked ready: "Buy battery A: runs 6 hours, costs less" and "Win them back: loyalty discount". The owner checks Part 1 against both battery quotes, finds it right, and puts both answers into their plan, marking Part 1 checked and Part 2 used as is. A second owner, with the same documents and the same answers, checks each part at its source. Part 1 checks out. For Part 2, the flyer, headed their offer, says cheaper prints, and the sales records, headed our sales and not given to the AI, say the shop lost rush jobs. The second owner changes the plan to "Win them back: same-day service". Part 2 depends on why customers left: the records' lost rush jobs is underlined, and the AI's "loyalty discount" is marked their flyer. Then an uneven line is drawn between the AI's answers, with Part 1 inside and Part 2 outside: a jagged frontier, which marks where an AI stops being reliable. A panel headed a real study, not these owners, shows a field experiment that gave some Kenyan entrepreneurs an AI assistant. On revenue and profit, weaker businesses did nearly 10% worse, and stronger ones may have gained over 15%, a less certain result. The study suggests what mattered was which AI advice owners acted on. The owners' working marks come off the plans, the AI's column fades back, and a reviewer sees only the plans. The reviewer points to Part 1 in both: each owner says, "The battery quotes." Then Part 2: the first owner says, "Part 1 checked out, so I trusted the rest." The second says, "Our records. Rush customers left in blackouts." The figure ends on the reviewer's check, with a line under the picture: illustration based on Otis et al. (2026) and its working paper. Each beat is listed below with what is on screen.
0Uncalibrated reliance · Failure mode 4 of 7
On screen: the name, "Uncalibrated reliance", appears large over the stage with "Failure mode 4 of 7" above it, then moves into the top-left corner, where it stays.
1A shop owner asks an AI two questions, sharing everything but the sales records.
On screen: a task card appears at the top left: "Example task. Part 1: Keep printing in blackouts. Part 2: Win back customers." A red dot labelled Owner appears below. Four documents appear across the top: Battery quote, Battery quote, Rival's flyer and Sales records. The dot pulses, and grey copies of the two quotes and the flyer slide into a small stack labelled "Given to the AI", where they stay. The sales records stay with the owner.
2It answers both with equal confidence.
On screen: two answer cards labelled "AI's answers" write in together, in grey on a beige band. Part 1: "Buy battery A: runs 6 hours, costs less." Part 2: "Win them back: loyalty discount." Each gets the same grey "Ready" tag at the same moment.
3The owner checks Part 1 against the two battery quotes. It's right.
On screen: the dot opens both battery quotes. A sheet labelled "First owner's plan" appears, Part 1 slides into it, and the dot marks it "Checked" in red.
4Part 2 looks just as sure, so they use it.
On screen: the "Ready" tag on the AI's Part 2 swells, and the dot moves down to the Part 2 row. Part 2 slides into the first plan, and the dot marks it "Used as is". The quotes close.
5Another owner, with the same answers, checks each at its source.
On screen: a second red dot labelled Second owner and an empty sheet labelled "Second owner's plan" appear. The dot pulses and both "Ready" tags blink. The dot opens both battery quotes, and Part 1 slides into its plan, marked "Checked". The label Second owner stays in place to the end.
6That owner checks Part 2 against the flyer and records, and changes it.
On screen: the dot opens the rival's flyer, headed "Their offer": "Cheaper prints". Then the sales records, headed "Our sales": "Lost: rush jobs", and set apart, "not given to AI". After a pause, Part 2 slides into the second plan. The dot strikes out "loyalty discount" and writes "same-day service", so it reads "Win them back: same-day service", and marks it "Checked". The first plan still offers a loyalty discount.
7Part 2 depends on why customers left. A jagged frontier marks where an AI stops being reliable.
On screen: the sales records' "Lost: rush jobs" is underlined. The AI's "loyalty discount" is underlined and marked "Their flyer". Then an uneven line is drawn between the AI's two answer cards, labelled "Inside the frontier" above and "Outside the frontier" below, and the two cards, alike in form, pulse together.
8A real study suggests what mattered was which AI advice owners acted on.
On screen: the picture dims almost away and a panel rises, headed "A real study, not these owners" and titled "Revenue and profit". Weaker businesses: nearly 10% worse, a solid bar to the left of zero. Stronger businesses: may have gained over 15%, a lighter dashed bar to the right, marked "Less certain". Source: Otis et al. (2026), Kenyan entrepreneurs, Management Science.
9A reviewer who sees only the plans points to each part and asks what justified relying on it.
On screen: the panel goes. The red marks lift off both plans, the documents and the AI's column fade back, and "The reviewer sees only the plans" appears. Part 1 is outlined in both plans, and each owner says, "The battery quotes." Then Part 2: the first owner says, "Part 1 checked out, so I trusted the rest." The second says, "Our records. Rush customers left in blackouts."
10Reviewer’s check: What evidence justified relying on the AI here? Ask part by part: was the evidence about that part?
On screen: both lines appear under the picture at the same size. The plans, both owners' answers, the AI's answers, the frontier and the sales records stay in view. Under the picture: "Illustration based on Otis et al. (2026) and its working paper."
Reviewer’s check
What evidence justified relying on the AI here?
Ask part by part: was the evidence about that part?
Both owners give the same reason for Part 1. For Part 2, the first owner's evidence is about Part 1, and the second owner's is about Part 2. The AI's two answers looked equally sure, and Part 2 turned on the sales records the AI never saw: the customers who left were rush jobs lost on blackout days, so a discount chases the wrong customers. The second owner found that by checking each part against its own evidence.
The study behind this example
From the published paper's abstract (Otis, Clarke, Delecourt, Holtz and Koning, Management Science, 2026): a field experiment with Kenyan entrepreneurs, "randomizing access to a GPT-4-powered AI business assistant." On average the authors were "unable to reject the null hypothesis of no average treatment effect on firm revenues and profits." But "low performers did nearly 10% worse because of the AI assistant, whereas high performers may have benefited by over 15%," and the effect for low performers was "over 0.20-standard-deviations lower than for initial high performers."
Also from the abstract, where the gap came from: "This differential impact does not appear to result from differences in the questions posed to the AI or the advice that it provided but rather, from the advice that entrepreneurs chose to implement." The study didn't measure checking, and the two owners in the figure don't stand for its groups. The figure's lesson, to check each part against evidence for that part, is what the result suggests, not something the study tested.
The limits. The percentages above are the published abstract's own figures. The revised working paper on eScholarship (PDF created July 2024) describes the outcome as "an index that combines standardized measures of weekly and monthly revenue and profits" (p. 8) and converts its own estimates into percent changes, which differ from the published ones (p. 11). In that version, "low" and "high" performers are businesses below and above the median before the study (p. 9). Only the stronger group's gain is uncertain: the published abstract says it "may have" happened, and the revised working paper calls its estimate "noisier" (p. 9). The assistant was built on GPT-4.
The same revised working paper adds detail the published abstract doesn't give. It describes 640 entrepreneurs randomly assigned to the assistant on WhatsApp (p. 3) or to business training guides (p. 7). The median owner was 26, had run the business for a year and held a college degree, and the businesses ranged "from fast-food joints to poultry farms to cybershops," which offer internet access and printing (p. 8). In an analysis the authors call exploratory, "treated low performers were especially likely to implement 'generic' advice focused on lowering prices and investing in advertising," while high performers found "tailored and specific changes," from "finding alternative ways to get power during blackouts" to a new car wash detergent (pp. 3–4). The figure's two parts echo that reading. Treat it as a lead, not a measured result. The published abstract doesn't mention it, and the February 2024 version of the working paper gave a different explanation in its abstract: weaker owners "sought help on more challenging business tasks."
Everything in the figure is constructed: the owners, the battery, the discount and the sales records. The jagged frontier is the essay's term, taken from Dell'Acqua et al. (2023); the appropriate reliance figure shows that study's numbers.
From the original PME essay
Uncalibrated reliance begins with a reasonable impulse: AI is useful, the output is confident, and parts of the task feel tedious. The problem is that AI performance is uneven in ways that are not always visible from the outside. Tasks that look similar may differ significantly in whether AI helps or hurts.
The distinction between trust and reliance matters here. Trust is a subjective disposition, a feeling of confidence that a system will perform well. Reliance is the observable act of accepting its output and acting on it. Appropriate reliance is harder and more discriminating: accepting support when it is warranted, refusing or verifying when it is not, and being able to give an account of the difference (Raees & Papangelis, 2026).