Why Do the Best Numbers Always Come From the Smallest Studies?

Because small studies are where big numbers come from, and it is not a conspiracy. It is arithmetic, selection, and human nature working together. One single-system study of an AI documentation tool reported clinician burnout down 30.3%. It had 24 physicians in it. A study of the same class of technology across five health systems, with 8,581 clinicians, found 13.4 fewer minutes of electronic record time, a result its own authors described as "modest." Same technology, same claim, and only one measurement survived contact with places the vendor did not choose. The AI Alignment Filter™ cannot weigh evidence you have not examined, so this is the step before it: effect sizes shrink as sample sizes grow, and they shrink in one direction. Ask how many, and where.

Introduction

There is a pattern in technology claims that, once you see it, you cannot stop seeing.

The most impressive results come from the smallest studies. The most modest results come from the largest ones. And the impressive ones are the ones that reach you, because impressive results are what get put into slide decks.

This is not fraud. Almost none of it is fraud. It is the predictable output of an honest system with ordinary human incentives running through it, which makes it far more common than fraud and considerably harder to spot.

The good news is that recognizing it requires no statistical training whatsoever. It requires two questions and the willingness to ask them out loud in a room where everyone else is nodding.

This week is the most transferable in the month. What follows applies to AI, and it applies just as well to lasers, scanners, membership plans, consultants, and practice-management systems. The technology changes. The pattern does not.

Key Takeaways

•     Effect sizes shrink as sample sizes grow, and the shrinkage runs in one direction. That is the single most useful pattern in evaluating any claim.

•     A study of 24 physicians reported burnout down 30.3%. A study of 8,581 clinicians across five health systems found 13.4 fewer minutes, described by its authors as "modest."

•     Five ordinary mechanisms produce this, none of which require anyone to lie.

•     A percentage without a sample size and a setting is not a finding. It is a decoration.

•     The question that does the most work is: has this been measured anywhere the seller did not choose?

•     Real technology is incremental, unevenly validated, and frequently disappointing under rigorous measurement. That is what makes it credible.

Two Studies of the Same Thing

Here is the comparison that holds this week together, and both halves of it are real.

A single health system studied an AI documentation assistant and reported clinician burnout down 30.3%. That is a striking figure, and it is the kind that travels. The study had 24 physicians in it.

Separately, researchers examined the same class of technology across five academic health systems, with 8,581 clinicians. They found 13.4 fewer minutes of electronic record time per eight hours of scheduled care, 16 fewer minutes of documentation time, and no significant change in after-hours record time.

Their own word for the effect was "modest."

Notice what did not happen. Nobody was caught out. Nobody fabricated anything. Both groups measured carefully and reported what they found. The difference between a headline figure and a modest one was produced entirely by how many people were measured, and by whether the measuring happened somewhere the technology had been invited.

Thirteen minutes a day is not nothing. Across a career it is a great deal. But thirteen minutes is a different purchasing decision from a third of your burnout, and only one of those numbers was ever going to be on a slide.

Five Reasons This Happens, and None of Them Are Lying

This is worth understanding mechanically, because once you know why it happens, you can predict it rather than being surprised by it.

Selection. The site that agrees to be studied is usually the site where things are going well. Nobody volunteers their disaster for publication. So the first measurement of anything is taken in unusually favorable conditions, by definition.

Enthusiasm. Early adopters are motivated people who chose the thing. They work around its rough edges, they persist through the awkward weeks, and their results include their own commitment. The tenth site to adopt has none of that and gets a different number.

Noise. In a small group, chance produces extreme results. Twenty-four of anything will occasionally produce a spectacular average for reasons that have nothing to do with the intervention, and the spectacular ones are the ones that get written up.

Selective visibility. The site where it did nothing does not hold a press conference. You are seeing a filtered sample of all the measurements that were taken, filtered by how interesting the result was.

Regression to the mean. An unusually good first result tends to be followed by a more ordinary second one, purely because the first was partly luck. This is why the sequel study is almost always duller than the original, in every field, forever.

Put those five together, and you do not need anyone to behave badly to get a market full of impressive small numbers and disappointing large ones.

The Two Questions

The AI Alignment Filter™ runs four steps on any tool: name the bottleneck, identify the cognitive load, count the true integration cost, and decide whether it replaces judgment or enhances it. None of those can be answered honestly from a claim you have not examined, which is why this week sits underneath the whole instrument rather than inside it.

Everything above collapses into two questions, and both of them are polite.

How many, and where?

That is it. A number with a sample size and a setting attached is a finding you can weigh. A number without them is a decoration, and you should treat it exactly as you would treat a nice photograph on the same slide.

Has this been measured anywhere you did not choose?

This is the sharper one, and it is the one that separates a company with a product from a company with a story. Independent measurement, on a site the vendor did not select, is expensive and risky, and the companies that have done it are proud of it.

Neither question has an edge on it. Neither implies the person in front of you is dishonest. And both have correct answers that a well-prepared vendor can give immediately.

What you are listening for is not the number. It is whether the question is familiar to them.

The Percentage With No Denominator

One specific habit worth building, because it catches a great deal.

When you see a percentage, find the denominator. Percentage of what?

An example from outside dentistry makes the point cleanly. A health system's autonomous coding system was described as cutting coding-related denials by 70%. What the source actually reported was that denials on autonomously coded cases ran about 70% lower than on manually coded cases, measured at one point in time. Roughly a third of one percent against roughly one percent.

That is a comparison between two groups, not a reduction over time. Nothing fell by 70%. And the absolute difference is under one percentage point, which is a perfectly respectable result and a completely different fact from the one the headline conveys.

Relative changes sound large and absolute changes tell you what actually happens. When only the relative figure is offered, the absolute one is usually the reason.

What a Careful Claim Looks Like

It is worth knowing what good looks like, so that you are not simply suspicious of everything.

The best-sourced figure I have found in dental AI is a measurement accuracy claim: software that measures alveolar bone level to within roughly three tenths of a millimeter of what a panel of dental radiologists agreed it was. It comes from a regulatory clearance summary rather than a brochure. It reports to three decimal places across two different image types. And it claims nothing beyond itself.

It does not say patients do better. It does not say your practice will produce more. It says this tool measures this thing to this precision under these conditions.

That is a modest, bounded, checkable claim, and it is far more persuasive than a figure three times as impressive would be.

When a vendor gives you a claim that small and that precise, they are telling you something about how they work.

Why the Disappointment Is the Point

There is a conclusion available here that sounds cynical and is the opposite.

Real technology is incremental. It is heavily scrutinized in some areas and barely examined in others. It is unevenly validated, and it frequently disappoints when somebody measures it rigorously somewhere it did not expect to be measured.

That is not a failure of the field. That is the field working.

What distinguishes a real technology from a story is not that the real one always succeeds. It is that the real one is built so that failure is detectable and gets published. The thirteen-minute result exists because somebody went looking for it in five places at once, knowing it might deflate a number their own institution had an interest in.

A claim that has no mechanism for being wrong is not a stronger claim. It is a weaker one wearing better clothes.

So when a technology's literature contains disappointments, external validations that went badly, and authors describing their own results as "modest," you are not looking at a weak field. You are looking at one where the checking works.

Frequently Asked Questions

Does this mean I should ignore small studies?

No. Small studies are how anything begins, and they are often the only evidence that exists for something new. Read them as a signal worth following, not as a measurement of what you will experience. The mistake is not reading them; it is pricing a purchase off them.

How large does a study need to be?

There is no threshold, and anyone who gives you one is oversimplifying. What matters more than size is whether the measurement happened somewhere the seller did not choose. A modest study in an unselected site tells you more than a large one in a showcase.

What if the vendor has no independent data at all?

That is common and it is not disqualifying, particularly for newer products. It does mean the number they are offering is an upper bound, and you should plan the purchase against something considerably less than it. Ask them what they would expect in a practice that is not an enthusiastic early adopter.

Is the same true of case studies?

More so. A case study is a sample of one, selected specifically because it went well. It is useful for understanding how the thing works and close to useless for predicting what it will do for you.

How do I raise this without souring the relationship?

Ask it as interest rather than challenge, because it is interest. How many, and where? What would you expect in a practice like mine? Where has it not worked well? People who know their product enjoy those questions. The reaction tells you as much as the answer.

Final Thoughts

The habit this week is small, and it compounds.

When a number arrives, do not ask whether it is true. Almost all of them are true, in the narrow sense that somebody measured something and got that answer. Ask what was measured, how many, where, and compared to what.

Then notice something about the person answering. The ones with real results have heard these questions before and reach for the detail happily. The ones without will tell you the question is too technical, or that every practice is different, or that the results speak for themselves.

The results never speak for themselves. Somebody measured them, and you are allowed to ask who.

Ask who measured it.

Sources & Further Reading

Rotenstein, Lisa, and colleagues. Study of ambient artificial intelligence documentation across five academic health systems, 8,581 clinicians, published in JAMA, 2026. The multi-site measurement and the authors' own characterization of the effect as "modest."

Wong, Andrew, and colleagues. External validation of a widely deployed sepsis prediction model, published in JAMA Internal Medicine. The cautionary case for what happens when a model meets a population it was not developed on.

Cluster-randomized trial of generative AI clinical decision support across sixteen primary care facilities in Kenya, 9,347 patients analyzed, published in Nature Medicine, 2026. The best randomized evidence available on large language model decision support, and a sobering one.