PME means professional military education. The Classroom version is also available for high school.
In a business class, the instructor gives every student the same case packet on a print-and-internet shop in Nairobi: two battery quotes, a new rival's flyer and, as a separate exhibit at the back, the shop's sales records. The case asks two things: how to keep printing through power blackouts, and how to win back customers lost to the new shop. The second question seems to be about the rival's lower prices, so a student asks an AI and shares the quotes and the flyer. They check the first answer, find it right, and use both. What can happen next is uncalibrated reliance: confident output is accepted without knowing whether the task fits the tool. The class allows AI, and the students say they used it. The instructor sees only the plans and asks about each part. Illustration based on Otis et al. (2026) and its working paper, a study of entrepreneurs in Kenya. No students took part in it; the figure's students, the shop's documents and every number in their plans are constructed.
An animated figure. A red dot stands for a student working a print shop's case, with an example task in two parts: keep printing in blackouts, and win back customers. Every student gets the same case packet: two battery quotes, a rival's flyer and the shop's sales records. The student gives the AI copies of the quotes and the flyer and leaves the sales records out. The AI answers both in the same form, in grey, each marked ready: "Buy battery A: runs 6 hours, costs less" and "Win them back: loyalty discount". The student checks Part 1 against both battery quotes, finds it right, and puts both answers into their plan, marking Part 1 checked and Part 2 used as is. A second student asks the same AI the same questions, gets the same answers, and checks each part at its source. Part 1 checks out. For Part 2, the flyer, headed their offer, says cheaper prints, and the shop's sales records, headed shop's sales and not given to the AI, say it lost rush jobs. The second student changes the plan to "Win them back: same-day service". Part 2 turns on facts the AI wasn't given: the records' lost rush jobs is underlined, and the AI's "loyalty discount" is marked flyer only. An uneven line is drawn between the AI's answers, with Part 1 covered by its input and Part 2 needing facts it wasn't given. Even with them, its answer would still need checking. A panel headed a real study of entrepreneurs shows a field experiment in Kenya that gave some of them an AI assistant. On revenue and profit, weaker businesses did nearly 10% worse, and stronger ones may have gained over 15%, a less certain result. The study suggests what mattered was which AI advice owners acted on. The working marks come off the plans, the AI's column fades back, a key reads AI-drafted (disclosed) and A student's checks and edits, and the instructor sees only the plans. The instructor asks what each part rests on: each student says, "The battery quotes." Then Part 2, and why customers left: the first student says, "The flyer. They left because the rival is cheaper." The second says, "The shop's records. Rush customers left." As they answer, each part is tagged: checked against the quotes for both Part 1s, used as is for the first student's Part 2, and checked against the records for the second's. The figure ends on the instructor’s check, with a line under the picture: illustration based on Otis et al. (2026) and its working paper. Each beat is listed below with what is on screen.
0Uncalibrated reliance · Failure mode 4 of 7
On screen: the name, "Uncalibrated reliance", appears large over the stage with "Failure mode 4 of 7" above it, then moves into the top-left corner, where it stays.
1A student asks an AI two questions about the shop's case, sharing everything but the sales records.
On screen: a task card appears at the top left: "Example task. Part 1: Keep printing in blackouts. Part 2: Win back customers." A red dot labelled Student appears below. The case packet's four documents appear across the top: Battery quote, Battery quote, Rival's flyer and Sales records. The dot pulses, and grey copies of the two quotes and the flyer slide into a small stack labelled "Given to the AI", where they stay. The sales records stay in the packet.
2It answers both with equal confidence.
On screen: two answer cards labelled "AI's answers" write in together, in grey on a beige band. Part 1: "Buy battery A: runs 6 hours, costs less." Part 2: "Win them back: loyalty discount." Each gets the same grey "Ready" tag at the same moment.
3The student checks Part 1 against the two battery quotes. It's right.
On screen: the dot opens both battery quotes. A sheet labelled "First student's plan" appears, Part 1 slides into it, and the dot marks it "Checked" in red.
4Part 2 looks just as sure, so they use it.
On screen: the "Ready" tag on the AI's Part 2 swells, and the dot moves down to the Part 2 row. Part 2 slides into the first plan, and the dot marks it "Used as is". The quotes close.
5Another student asks the same AI the same questions, then checks each answer at its source.
On screen: a second red dot labelled Second student and an empty sheet labelled "Second student's plan" appear. The dot pulses and both "Ready" tags blink. The dot opens both battery quotes, and Part 1 slides into its plan, marked "Checked". The label Second student stays in place to the end.
6That student checks Part 2 against the flyer and records, and changes it.
On screen: the dot opens the rival's flyer, headed "Their offer": "Cheaper prints". Then the shop's sales records, headed "Shop's sales": "Lost: rush jobs", and set apart, "not given to AI". After a pause, Part 2 slides into the second plan. The dot strikes out "loyalty discount" and writes "same-day service", so it reads "Win them back: same-day service", and marks it "Checked". The first plan still offers a loyalty discount.
7Part 2 turns on facts the AI wasn't given. Even with them, its answer would still need checking.
On screen: the sales records' "Lost: rush jobs" is underlined. The AI's "loyalty discount" is underlined and marked "Flyer only". Then an uneven line is drawn between the AI's two answer cards, labelled "Covered by its input" above and "Needs facts it wasn't given" below, and the two cards, alike in form, pulse together.
8A real study suggests what mattered was which AI advice owners acted on.
On screen: the picture dims almost away and a panel rises, headed "A real study of entrepreneurs" and titled "Revenue and profit". Weaker businesses: nearly 10% worse, a solid bar to the left of zero. Stronger businesses: may have gained over 15%, a lighter dashed bar to the right, marked "Less certain". Source: Otis et al. (2026), Kenyan entrepreneurs, Management Science.
9The instructor, who sees only the plans, asks what each part rests on and why customers left.
On screen: the panel goes. The red marks lift off both plans, the documents and the AI's column fade back, and "The instructor sees only the plans" appears, with a key: "AI-drafted (disclosed)" and "A student's checks and edits". Part 1 is outlined in both plans, and each student says, "The battery quotes." As each answers, that part gets a red tag, "Checked: quotes". Then Part 2, and why customers left: the first student says, "The flyer. They left because the rival is cheaper.", and their Part 2 is tagged "Used as is". The second says, "The shop's records. Rush customers left.", and theirs is tagged "Checked: records".
10Instructor’s check: What evidence justified relying on the AI here? Ask part by part. Part 2: why did customers leave?
On screen: both lines appear under the picture at the same size. The plans, both students' answers, the AI's answers, the frontier and the sales records stay in view. Under the picture: "Illustration based on Otis et al. (2026) and its working paper."
Instructor’s check
What evidence justified relying on the AI here?
Ask part by part. Part 2: why did customers leave?
Both students give the same reason for Part 1. For Part 2, both point to evidence about Part 2, so ask what that part turns on: why did the customers leave? The first student's flyer shows only that the rival is cheaper. The second student's records show the shop lost rush jobs, so a discount chases the wrong customers. The instructor knows the case, so the first plan's discount may already look off; the question shows how it got in. Part 2 turned on facts the AI wasn't given, and even with the records, its answer would still need checking against them. The move is to check each part against the evidence for that part, including what the AI didn't see.
The study behind this example
From the published paper's abstract (Otis, Clarke, Delecourt, Holtz and Koning, Management Science, 2026): a field experiment with Kenyan entrepreneurs, "randomizing access to a GPT-4-powered AI business assistant." On average the authors were "unable to reject the null hypothesis of no average treatment effect on firm revenues and profits." But "low performers did nearly 10% worse because of the AI assistant, whereas high performers may have benefited by over 15%," and the effect for low performers was "over 0.20-standard-deviations lower than for initial high performers."
Also from the abstract, where the gap came from: "This differential impact does not appear to result from differences in the questions posed to the AI or the advice that it provided but rather, from the advice that entrepreneurs chose to implement." The entrepreneurs used the assistant for their own small businesses. No students took part, and the study didn't measure checking, so the two students in the figure don't stand for its groups. The figure's lesson, to check each part against evidence for that part, is what the result suggests. The study didn't test it.
The limits. The percentages above are the published abstract's own figures. The revised working paper on eScholarship (PDF created July 2024) describes the outcome as "an index that combines standardized measures of weekly and monthly revenue and profits" (p. 8) and converts its own estimates into percent changes, which differ from the published ones (p. 11). In that version, "low" and "high" performers are businesses below and above the median before the study (p. 9). Only the stronger group's gain is uncertain: the published abstract says it "may have" happened, and the revised working paper calls its estimate "noisier" (p. 9). The assistant was built on GPT-4.
The same revised working paper adds detail the published abstract doesn't give. It describes 640 entrepreneurs randomly assigned to the assistant on WhatsApp (p. 3) or to business training guides (p. 7). The median entrepreneur was 26, had run the business for a year and held a college degree, and the businesses ranged "from fast-food joints to poultry farms to cybershops," which offer internet access and printing (p. 8). In an analysis the authors call exploratory, "treated low performers were especially likely to implement 'generic' advice focused on lowering prices and investing in advertising," while high performers found "tailored and specific changes," from "finding alternative ways to get power during blackouts" to a new car wash detergent (pp. 3–4). The figure's two parts echo that reading. Treat it as a lead. It isn't a measured result. The published abstract doesn't mention it, and the February 2024 version of the working paper gave a different explanation in its abstract: weaker owners "sought help on more challenging business tasks."
Everything in the figure is constructed: the students, the battery, the discount and the sales records. The jagged frontier is the essay's term, taken from Dell'Acqua et al. (2023); the appropriate reliance figure shows that study's numbers.
From the original PME essay
Uncalibrated reliance begins with a reasonable impulse: AI is useful, the output is confident, and parts of the task feel tedious. The problem is that AI performance is uneven in ways that are not always visible from the outside. Tasks that look similar may differ significantly in whether AI helps or hurts.
The distinction between trust and reliance matters here. Trust is a subjective disposition, a feeling of confidence that a system will perform well. Reliance is the observable act of accepting its output and acting on it. Appropriate reliance is harder and more discriminating: accepting support when it is warranted, refusing or verifying when it is not, and being able to give an account of the difference (Raees & Papangelis, 2026).