On retention, live-versus-recorded is close to a red herring. The 2010 federal meta-analysis of online learning scored instructor-directed conditions at +0.39 and independent, self-directed conditions at +0.05, which was not statistically significant. What predicts whether a skill survives to March is not screen against room. It is whether anyone directs the learning, makes people produce, and corrects what they produce.
That reframes the purchase. A platform can be bought with direction attached or without it, and most are bought without. A live session can be run as practice or as a demo, and plenty are run as demos. Format is what appears on the invoice. Direction is what retains.
This is for the person holding the training budget at a US company of 50 to 500 people, who has paid for licenses and is being asked what capability the money bought.
Retention and completion are two different questions
Completion asks whether a person reached the end of something. Retention asks whether they can still do the task three months later without help. The two come apart in both directions, and the gap is where training budgets die.
A team can finish a course library and retain nothing, because watching is not doing. A team can skip half a session and retain the one habit that mattered, because the half they attended made them produce something on their own file.
Vendors on both sides have a reason not to draw the distinction. Platforms report completion cheaply and at scale. Live providers report attendance and a good room. Neither can report retention without going back to the team in February and testing them, and almost nobody sells that. It is the gap behind the case that buying licenses is not the same as building capability.
What a platform dashboard is counting
Read the vendor documentation rather than the label on the chart.
On LinkedIn Learning, a video counts as complete once a learner has watched 70% of it without fast-forwarding. Hours viewed are credited against the content’s real length, so an hour consumed at double speed logs a full hour, and a rewatch logs those hours again. The headline metrics in the admin engagement view are the share of activated seats that logged in, average video views per viewer, and average hours per viewer.
Coursera for Business reports skill mastery in tiers, with median hours to the next tier and a peer benchmark. Those tiers are derived from activity on the platform. No post-course assessment set by an outside party is documented.
None of that is dishonest. A platform is built to report platform usage and reports it well. The error is on the buyer’s side. A dashboard green on logins and hours says people used what you paid for, and nothing about whether they can now do the task.
The finding in the federal meta-analysis that settles this
The most quoted evidence here is the US Department of Education’s 2010 review, Evaluation of Evidence-Based Practices in Online Learning (Means, Toyama, Murphy, Bakia and Jones). Screening produced 50 independent effects: 27 purely online against face to face, 23 blended against face to face. The headline is a mean effect size of +0.20 favoring online conditions, at p below .001. The report’s own footnote sets that scale against Cohen: 0.20 is small, 0.50 medium, 0.80 or more large.
Almost everyone stops at the headline. The moderator analysis is the part that answers a buying question.
| Condition | Mean effect size | Significant? |
|---|---|---|
| Instructor-directed online learning | +0.39 | Yes |
| Collaborative online learning | +0.25 | Yes |
| Independent, self-directed learning | +0.05 | No |
So the condition most license purchases buy by default, a seat and a catalog with nobody directing it, is the one that did not clear significance. The report also cautions that online and classroom conditions differ on many dimensions at once, including time on task. Part of any online advantage may be extra hours rather than the medium. And when one study compares blended against purely online, outcomes are usually comparable. The medium keeps failing to be the variable.
This is pre-generative-AI research about learning in general. Nobody has run a controlled retention trial of AI workshops against AI course libraries, and anyone claiming otherwise should be asked for the study.
Why the most confident learner is the one who forgot
Roediger and Karpicke, in Psychological Science in 2006, had students read prose passages, then either restudy them or take free-recall tests with no feedback the same number of times, and measured recall later.
| Final test | Repeated study | Repeated testing |
|---|---|---|
| After 5 minutes | 81% | 75% |
| After 2 days | 54% | 68% |
| After 1 week | 42% | 56% |
Restudying wins immediately and loses badly a week out. Their second experiment pushed it further: at five minutes recall tracked study repetitions (83%, 78%, 71%), and at one week it tracked test repetitions and inverted (40%, 56%, 61%).
The result that should change how you read a vendor report is the confidence finding. Students in the repeated-study condition were the most sure they would remember the passage a week later, and they remembered least. Passive review feels like learning while it happens. That is the feeling a well-produced course library sells, and the feeling a smiling post-session survey captures.
The question to put to either option is not how good it felt, but what the learner had to produce unaided.
Three numbers you will be quoted that do not hold up
“70% of training is forgotten within 24 hours.” Credited to Hermann Ebbinghaus, 1885. Ebbinghaus measured relearning savings, not recall: about 58% at 20 minutes, 34% at one day, 21% at 31 days. A savings score is how much faster material is relearned, and turning it into a share forgotten is not a valid step. Murre and Dros reproduced the curve’s shape in 2015. The shape is real; the percentage is folklore.
“85% of training is never applied.” The traceable benchmark is a 2014 CEB white paper, now Gartner, putting unapplied learning near 45% in the average organization. The 85% version traces to Brinkerhoff and Mooney in 2008 and is reported inconsistently; Brinkerhoff has put the 90-day range across studies at 45% to 80%. Most pages carrying these figures sell scrap-learning measurement.
“93% of instructor-led learners score better.” One small cohort of medical students in a single journal, pasted into corporate comparison pages ever since. Not a finding about office workers learning a chat assistant.
Thin evidence is not a reason to buy nothing. It is a reason to stop outsourcing the question.
The four mechanisms that move retention
Strip the format away and four things predict whether a skill survives. Score any proposal, from either side, against them.
| Mechanism | What it means | Platform by default | Live session by default |
|---|---|---|---|
| Retrieval | The learner produces output from memory, not alongside a video | Rare; quizzes test recognition, not production | Common when hands-on, absent when it is a demo |
| Spacing | The skill is revisited at intervals over weeks | Strong, and the one place platforms genuinely win | Absent unless follow-ups are contracted |
| Feedback | Someone qualified corrects the learner’s own output | Almost never | The core of the format |
| Own-work context | Practice runs on the learner’s real documents and tasks | Never; content is generic by design | Available if the provider asks for your material |
The scoreboard: a live session ships retrieval, feedback and context when designed well, and almost never ships spacing. A platform ships spacing cheaply and the other three almost never. That beats “blended is best” because it names the job each purchase is doing.
When a platform is the right purchase
Buy the platform when the goal is baseline literacy across a few hundred people and the marginal cost of the next learner has to be near zero. Buy it when the content is stable and declarative: terminology, data-handling rules, or what your acceptable-use policy permits. Recall of fixed facts is the outcome there, and a quiz fairly tests it. Buy it when you need an auditable record that named people finished named modules on named dates, which is a compliance artifact rather than a capability claim. Buy it when the team is spread across time zones and a shared slot is not realistic, a constraint also at work in education-benefit platforms against direct enablement spend.
And buy it as the spacing layer. If a live session gave people the habit, a library is a cheap place to send them back in week three and week seven.
When a facilitated session is the right purchase
Buy facilitation when the outcome is judgment rather than knowledge. Deciding whether a draft is good enough to send, spotting a confident wrong answer, knowing which task to hand to a tool: these need someone to read what you produced and say where it went wrong. No library does that.
Buy it when the material has to be your own. A session that works runs on the team’s real documents, because transfer is weakest when the practice context sits furthest from the work context. Generic examples teach a pattern the learner then has to port alone, and the porting is where it fails.
Buy it when the group is mixed, when there is open resistance, or when the sticking point is political rather than technical. A room surfaces the objection nobody types into a course player. Who stands in that room matters more than the logo on the deck, the subject of the choice between a strategy consultancy and a practitioner.
Market rates for a private session run roughly $2,000 to $15,000 per engagement, and the wide band is mostly design and prep time.
A 90-day retention test you can run yourself
You do not need a vendor’s dashboard. You need one task and three dates.
- Pick one real task that AI should improve: a first-draft client email, a weekly summary from a transcript, a cleanup pass on a messy spreadsheet. One task, not a skill matrix.
- Baseline it before any training. Have eight to twelve people do the task as they do it now. Record minutes taken and keep the output. Skipping this step makes everything after it unusable.
- Re-run the identical task at 30, 60 and 90 days. No warning, no notes, no refresher the night before. A retest people revise for measures revision, not retention.
- Have the output scored by someone who is not selling you training, using a three-criteria rubric written before the baseline.
- Track the number that matters at 90 days: how many of the twelve still choose to use the tool on that task without being told to. Voluntary use is the closest honest proxy for retention you can collect.
What to ask either vendor
Price questions are easy to answer and tell you little. These are harder, and the hesitation is informative.
- What share of session or module time has the learner producing output from memory rather than following along?
- Who reviews that output, what qualifies them, and how long do they spend per person?
- Will practice run on our documents? When do you need them, and under what data terms?
- What happens in weeks two through eight, and is it in scope?
- How would you measure whether this retained at 90 days, and will you commit to measuring it?
A platform vendor answers the spacing question well and struggles on feedback and context. A live provider is the reverse. A vendor strong on all five is selling, the gap set out in the ledger on what a training marketplace does not sell you.
The decision, in one table
| Your situation | Buy | Why |
|---|---|---|
| Baseline literacy for 200 or more people, flat budget | Platform | Marginal cost near zero; fixed content matches recall |
| Policy, terminology and acceptable use on record | Platform | The outcome is an auditable completion, not a changed habit |
| One function must work differently within a quarter | Live session | Needs feedback on their own output; nothing else delivers it |
| Mixed skill levels, or visible resistance | Live session | Objections surface in a room, never in a course player |
| You already ran a session and want it to hold | Platform, as the spacing layer | The one retention mechanism platforms do well and cheaply |
| Budget covers one, and the goal is changed work | Live session plus your own 90-day test | Fewer people trained properly beats everyone enrolled |
| Budget covers one, and the goal is coverage | Platform plus a named internal owner | Direction is what the meta-analysis says you are missing |
Those last two lines are the real decision for most companies this size. Whether capability of this kind should be rented or built at all is the larger question behind it, laid out asset by asset in the buy-versus-build playbook.
Frequently asked questions
Which retains better, an AI training platform or a facilitated workshop?
A well-designed live session retains better, because it ships retrieval practice, feedback on the learner’s own output, and real work context. But format is not what decides it. A directed platform program beats an undirected one by more than live beats recorded.
Is the “70% forgotten in 24 hours” statistic true?
No, not as stated. Ebbinghaus measured relearning savings in 1885, about 34% at one day, and savings cannot be converted into a share forgotten. The curve’s shape is well supported and was reproduced in 2015. The percentage is not a finding.
Does a high completion rate mean the training worked?
It means people consumed content. On LinkedIn Learning a video completes at 70% watched without fast-forwarding, an hour at double speed logs a full hour, and rewatches log again. Those are usage measures, and none tests whether a person can do the task unaided.
How do I measure whether AI training retained?
Baseline one real task before training, re-run the identical task unannounced at 30, 60 and 90 days, and have the outputs scored by someone who is not selling you training. Then count how many people still use the tool on that task voluntarily.
Can we get platform scale and live retention at the same time?
Yes, and sequencing matters more than the mix. Run the live session first so people build the habit on their own work, then use the library as the place they return to in weeks three and seven. A library alone teaches a habit nobody formed.
How soon after a session should reinforcement happen?
Inside two weeks, and again inside two months. Short, spaced, production-based touches beat one long refresher: a 30-minute block where people redo the task beats a 90-minute recap.
Does it matter whether the session is in person or on video?
Less than most buyers expect. Blended and purely online conditions compared inside one study produced broadly comparable outcomes, and the moderators that mattered were direction and collaboration. Live video with cameras on, small groups and real files retains well. A large in-person audience watching a demo does not.
How much should we expect to pay?
Per-seat platform licenses are published and sit in the low hundreds of dollars per user per year at team tiers. Private sessions are quoted rather than published, with market rates roughly $2,000 to $15,000 per engagement.
Key takeaways
- Retention and completion are different measures, and no platform dashboard reports the first. Read the vendor’s own definition of a completion before citing it.
- The evidence that settles the format debate is a moderator finding, not a headline: instructor-directed learning at +0.39, independent self-directed learning at +0.05 and not significant.
- Repeated study beats testing at five minutes and loses by 14 points at a week, and the restudy group is the most confident of all.
- Three widely circulated statistics here do not survive a provenance check. Ask for the source before one reaches your board.
- Score any proposal on four mechanisms: retrieval, spacing, feedback and own-work context. Live sessions ship three, platforms ship one.
- Measure it yourself: one task, a baseline, unannounced retests at 30, 60 and 90 days, and a count of voluntary use.
Weighing these two and want a second read on which gap is yours? Book a free AI-readiness call and we will look at the work your team does before anyone recommends a format.
Arthur Wandzel
