A case study is an advertisement. Five questions dispose of most of the numbers in one: who was counted, what the starting point was, which instrument measured it, over what window, and compared with what. Whatever survives that screen earns a reference call, and the call has one question almost nobody asks.
One clarification, because the phrase points two ways. Search “AI training reference checks” and you mostly get software for calling a job candidate’s referees. Different article. This is for the person at a US company of 50 to 500 people holding an AI enablement proposal with a PDF attached that has a 40% in it.
The case study is an advertisement, and there is a standard for that
A customer story on a vendor’s site is advertising, and US law already says what has to sit behind it. The Federal Trade Commission’s Guides Concerning the Use of Endorsements and Testimonials in Advertising, at 16 CFR Part 255, were finalized on June 29, 2023. Under § 255.2(b), an advertiser using a customer’s results must have substantiation that those results represent what customers generally achieve, or disclose the generally expected performance clearly enough to change the impression the ad leaves. Under § 255.5, an unexpected connection between endorser and seller must be disclosed.
Then enforcement. On September 25, 2024 the FTC announced Operation AI Comply, five actions over claims about artificial intelligence. The DoNotPay settlement is the one to read before a procurement call. The company paid $193,000, and the allegation was not that the product was bad: the complaint states that “the company did not conduct testing to determine whether its AI chatbot’s output was equal to the level of a human lawyer.” Failure to test was the violation.
You are not going to sue your training vendor, and that is not the point. The standard puts the substantiation on the advertiser, which makes asking for it a reasonable request rather than an awkward one. A vendor who measured something sends the instrument and the dates within a day. A vendor who did not sends a longer PDF.
Five questions that dispose of most numbers
Every claim in a training case study is one sentence with five things missing. Put them back and most stop making the claim they appeared to make.
| Question | What it exposes | A surviving answer |
|---|---|---|
| Who was counted? | The denominator. “Teams report faster drafting” can mean four self-selected people out of sixty. | “Of the 42 people in the two sessions, 38 submitted the follow-up task.” |
| Measured from what? | The baseline. A percentage with no starting value is a shape, not a quantity. | “Median first-draft time on this document class was 50 minutes before, 31 after.” |
| Which instrument, scored by whom? | A vendor-written assessment scored by the vendor measures the assessment. | “Their operations lead scored the output against the firm’s existing review rubric.” |
| Over what window? | Same-day numbers measure a session. Nine-week numbers measure a change. | “Measured at the session, then again at eight weeks.” |
| Compared with what? | The counterfactual. Capability was moving anyway, because the tools improved. | “An untrained team in the same function was measured on the same task that month.” |
Run those against a typical headline. “Participants showed a 40% improvement in AI skills,” with no window and no comparison group, describes a pre-test and a post-test taken close together by the party that taught to them. It may be true. It is not evidence that anyone works differently in November.
A claim that survives reads unglamorously: of 42 people in the session, 31 completed a graded task on their own documents two weeks later, and the firm’s reviewer scored 24 of those at or above the standard already used for junior work. No percentage in sight. Vendors who build a program around a measurable change can produce that paragraph on request, which is why asking sorts the field fast. The design decisions that make it producible are in our guide to proving capability actually moved.
Reading a real one, line by line
Take a published, named example rather than a straw man. Coursera’s enterprise case study on the Larsen & Toubro GenAI Academy reports a 40% improvement in AI-related skills between pre- and post-assessment, across 86,654 assessments.
That page is better than most. The scale is genuine, pre/post is a real design, and both the client and the instrument type are named — which already clears a hurdle anonymous “a Fortune 500 manufacturer” stories cannot.
Now the screen. The page does not state the interval between the two assessments, who authored or graded them, or whether any comparison group existed. It does not say whether the 40% is a change in mean score or in the share of people crossing a threshold, which are different claims. And it reports assessments completed rather than people assessed, so the denominator stays open.
None of that makes the number false. It makes it unusable as a forecast for your 60-person operations team. If one of the better-documented case studies in the market leaves four of five questions open, a four-paragraph story from a smaller vendor will not either. Move to the call.
Why the evidence is thin on both sides of the table
The missing data is not only a vendor problem. It is largely a client-side artifact: a vendor cannot report what their client never collected.
ATD’s 2016 study Evaluating Learning: Getting to Measurements That Matter (n=199) found that only 35% of organizations evaluated business results. Nine years on, the LinkedIn 2025 Workplace Learning Report (937 L&D and HR professionals, 679 learners) shows a recognizable pattern: among organizations most committed to career development, 72% track engagement, 64% retention, and 55% promotions. Participation and sentiment measures, all three. None says whether anybody got better at the work.
The money, meanwhile, moved. Training magazine’s 2025 Training Industry Report put US training expenditure at $102.8 billion, up 4.9%, with spending on outside products and services up 29% to $16 billion. More is bought from outside, by buyers whose dashboards measure attendance.
So reframe both halves. Ask the vendor what was measured and who held the instrument, not for proof of an outcome the client never instrumented — “the client measured completion, and here is what we would propose measuring with you” beats a confident percentage. Ask the reference what they measured. A client who measured nothing but can name what changed in which workflow is more useful than one who forwards the vendor’s slide.
The reference call, and who to call
Ask for three references matched on headcount band, function mix, and scope of the enablement work. A 400-person rollout says little about your 60-person team, and a sales engagement says little about finance and operations.
Then ask for a fourth that generic vendor-vetting advice never mentions: a client who did not continue. Every vendor has them, and one who hands you a name and a reason is telling the truth about their failure modes. Find one more person yourself — the nominated reference is usually the signer, while someone who sat in the room is on LinkedIn with a different view of the day.
Eight questions, in this order:
- What did the sessions cover, hour by hour, as far as you remember?
- Who facilitated, and was that the person the proposal named?
- What did your people bring into the room — their own documents, or the vendor’s examples?
- Nine weeks later, what was anyone doing differently? Name a task.
- How do you know that? What did you look at?
- What did you have to supply that you had not budgeted for — admin setup, licenses, prep, follow-up?
- What did they get wrong, and what happened when you raised it?
- Would you buy the same scope again, or a different one?
Question four is the whole call. Satisfaction scores come from the vendor; nine-week behavior only from the client. Hesitation there is data, and it usually means the session was enjoyable and nothing downstream changed. Question six catches the costs that skip the quote and reach the invoice; which side pays is settled by the contract terms that decide whether AI training lands.
The question nobody asks: who stood at the front of the room
Generic diligence asks whether the proposed team was the delivering team. For facilitated training that is not one question among several. It is the main one.
In a software purchase the product ships and the people rotate. In a workshop the person is the product. Same deck, two facilitators, two different rooms — one where a skeptical controller tries the tool on a real reconciliation, one where forty people watch a demo. Case studies name the client and the firm, almost never the facilitator, which is the most predictive fact on the page.
Ask three things and put the answers in writing: who delivered this engagement, by name; whether they still work there; and whether they will deliver yours. The failure pattern is mundane. A principal sells the work, demand grows, delivery passes to a contractor hired for availability, and the website still describes the principal’s session. Named-facilitator terms with a substitution clause cost nothing before signature and are unobtainable after.
Platform purchases invert the exposure: the curriculum is fixed and you can inspect it in an afternoon, but no case study tells you about completion among people who were not volunteers. Which of the two fits is the argument in the buy-versus-build decision for enablement capability, and the field-level view is in our comparison of AI workshop providers for mid-market companies.
Six claims that should end the conversation
| Claim | Why it fails |
|---|---|
| Every client logo, no named contacts | References are a cost of doing business at this size. Reluctance signals a thin client base. |
| “Guaranteed ROI” on a training engagement | Return depends on what the client does in the eight weeks after. Nobody guarantees that from outside. |
| Percentages with no denominator on the page | The most common defect, and the easiest to fix if the data exists. |
| Satisfaction scores presented as outcomes | A 4.8 out of 5 measures the room, not the work. |
| Model version numbers in a curriculum | The material is dated before delivery. Durable training teaches judgment about a class of tool. |
| “Our AI-powered assessment proves the lift” | The vendor wrote, taught to, and graded the test. Ask for the client’s own rubric. |
One answer that is not a red flag, despite being treated as one: a vendor who says they measured attendance and task completion rather than business results, and names what they would instrument with you. That posture is what the case for building capability rather than buying licenses argues buyers should expect. If you are still assembling a shortlist, start from the companies worth considering for corporate team training.
What verification costs
| Activity | Time | Notes |
|---|---|---|
| Run the five questions against every claim | 45 minutes | In writing; send the gaps back |
| Request the substantiation and the instrument | 10 minutes | Response time is itself the signal |
| Three matched reference calls | 25 minutes each | One week’s lead time |
| One non-renewal reference | 20 minutes | Highest-yield call in the set |
| One back-channel attendee | 15 minutes | Found on LinkedIn, not nominated |
Three hours, against an AI enablement workshop that runs roughly $2,000 to $15,000 in the open market or a follow-on automation project of $25,000 to $150,000. The cheapest line in the decision, and the only one that lowers the odds of buying a pleasant afternoon.
A two-week sequence
Days 1 to 2. Mark the proposal in two colors: facts you can verify yourself (names, titles, dates, client identity, session length) and claims you cannot (percentages, improvements, cultural change). Verify the first, send the second back as questions.
Days 3 to 5. Request references — three matched, one non-renewal — plus the facilitator’s name and the measurement instrument. Note how long each takes to arrive.
Days 6 to 9. Make the calls, leading with question four. Find your back-channel attendee in parallel.
Days 10 to 14. Decide, and convert what you learned into terms: named facilitator with a substitution clause, your documents in the room, one measurement you will both look at eight weeks out. Then run it on one team before nine.
Frequently asked questions
What should I verify in an AI training vendor’s case study?
Five things: the denominator, the baseline, the instrument and who scored it, the window between measurements, and whether a comparison group existed. Verify the checkable parts too — client name, dates, titles, session length.
What questions should I ask an AI training vendor’s references?
What the sessions covered hour by hour, who facilitated and whether that matched the proposal, whether participants worked on their own material, what anyone was doing differently nine weeks later and how they know, and whether they would buy the same scope again.
Should I ask for a reference from a client who did not renew?
Yes, and it is the most informative call you will make. Every vendor has non-renewals, so the ask is not hostile. One provided with a candid reason shows how the vendor handles a bad outcome; a claim that none exist means new or selective.
How do I evaluate an AI training vendor’s claimed results?
Treat it as an advertising claim and request the substantiation the FTC’s endorsement guides expect the advertiser to hold. Then judge the response, not the number: an instrument, a date range and a denominator inside a day is strong; a longer PDF is weak.
Is a 40% improvement in AI skills a meaningful number?
Only with a window, an instrument and a comparison. A pre/post assessment written and graded by the vendor who taught the material, with no stated interval, measures the assessment. The same 40% at eight weeks against the client’s own rubric would mean something.
Why do so few training vendors have outcome data?
Because few buyers collect it. ATD found 35% of organizations evaluated business results, and L&D measurement still centers on engagement and retention rather than capability. Vendors cannot report what clients never instrumented, so ask what they propose to measure with you.
Does it matter who facilitates the workshop?
For live training it is the most predictive variable on the page. The same material delivered by two facilitators produces two different rooms, and case studies name the firm rather than the person. Ask who delivered that engagement and who delivers yours.
How long should vendor verification take?
About three hours over two weeks: under an hour to screen the claims, three matched reference calls plus one non-renewal, and one attendee you find yourself. Against a $2,000 to $15,000 engagement, the cheapest diligence in the deal.
Can I just run a paid pilot instead of checking references?
A pilot is better evidence than any case study, and it does not replace the calls. Pilots get the vendor’s best facilitator and your most willing team, so they show the ceiling rather than the median. Run one, and still ask about week nine.
Key takeaways
- A case study is advertising. The FTC’s endorsement guides put substantiation on the advertiser, so ask for it and judge the response time as much as the data.
- Five questions dispose of most claims: who was counted, measured from what, which instrument and scored by whom, over what window, compared with what.
- The FTC’s 2024 action against DoNotPay charged failure to test as the violation. Untested claims are the exposure, not unsuccessful products.
- ATD found 35% of organizations evaluated business results, so vendors often cannot report what clients never measured. Ask what they propose to measure with you.
- The nine-week question is the only one a reference can answer and a vendor cannot: what is anyone doing differently, and how do you know?
- Ask for a non-renewal reference, and ask who stood at the front of the room — by name, still employed, contracted to deliver yours.
If you want a second read on a proposal already on your desk, or a plain answer on what a session for your team would cover and what you could measure afterwards, book a free AI-readiness call. We will go through the claims you were sent and name what is still unanswered.
Dirk Jan van Veen, PhD
