What Question Makes a Vendor Uncomfortable?

None of the good ones, which is precisely why the question is diagnostic. Seven questions separate a company with results from a company with a story, and every one of them is polite, answerable, and specific. How many, and where? Who measured it? Compared to what? Is that a reduction or a comparison? What is the denominator? Where has it not worked? And has anyone measured this somewhere you did not choose? A vendor sitting on real evidence finds these questions ordinary, because producing that evidence was expensive, and they are proud of it. A vendor without them will tell you the question is too technical, or that every practice is different. The reaction is the information. The AI Alignment Filter™ cannot run on claims you have not tested.

Introduction

The title is slightly unfair, and I want to correct it before the first section.

Good questions do not make good vendors uncomfortable. They make them animated. Ask a company that has genuinely measured its product how many sites the measurement covered, and you will usually get more detail than you wanted, because somebody spent two years and a great deal of money producing it and almost nobody ever asks.

The discomfort only shows up when the evidence is not there. And that is the entire diagnostic value of the exercise. You are not testing the product with these questions. You are testing whether there is anything underneath the claim.

Which means you can ask them warmly, with no edge at all, and still learn everything you need to know.

Key Takeaways

•     Good questions do not make good vendors uncomfortable. Discomfort appears only where the evidence is absent.

•     Seven questions cover almost every claim you will be shown, and none of them require technical knowledge.

•     A reduction and a comparison are different facts. A great many claims quietly present the second as the first.

•     A percentage with no denominator tells you nothing, and the missing denominator is usually the reason it is missing.

•     "Where has it not worked?" is the single most revealing question available, and it is the one almost nobody asks.

•     The answer matters less than whether the question is familiar to them.

The Seven Questions

How many, and where?

The first question, always, and it does more work than the other six combined. A claim with a sample size and a setting attached can be weighed. Without them, you have a sentence.

Who measured it?

The company, a customer the company selected, or somebody with no financial interest in the answer. All three are legitimate sources of information, and they carry entirely different weight. You simply need to know which one you are holding.

Compared to what?

Every improvement is an improvement over something. Over doing nothing? Over the previous version? Over a comparable product? Over how that particular practice happened to be working before, which may have been unusually bad? The comparator is where most of the effect often lives.

Is that a reduction or a comparison?

These sound identical and are not. A reduction is a change over time in the same population. A comparison is a difference between two groups, measured once. A claim that denials fell by seventy per cent is a reduction. A claim that denials on one type of case run seventy per cent lower than on another type is a comparison, and nothing fell at all.

What is the denominator?

Find what the percentage is a percentage of. Relative figures sound large, absolute figures tell you what happens, and when only the relative one is offered, the absolute one is usually the reason.

Where has it not worked?

Ask this one. It is the most revealing question in the list, and it is almost never asked, which means the answer is rarely rehearsed. Every real product has customers it suits badly, and a company that knows its own product can describe them immediately. A company that says it works for everyone has told you they are not paying attention to their own failures.

Has this been measured anywhere you did not choose?

The sharpest question, saved for last. Independent measurement in an unselected site is expensive and risky, and the companies that have commissioned it will tell you about it at length.

A Worked Example

Here is what these questions do to a real claim, and I am using a real one because the abstract version is less useful.

A robotic dental surgery company reports that total procedure time averaged roughly forty-five per cent lower for partial-implant patients. That is a substantial and interesting claim.

Now run the questions.

The figure appears in a press release. The press release describes a prospective, multi-site clinical study. When you go looking for that study, there is no site count, no patient count, no named authors, and no protocol registration. The venue is a trade magazine.

None of that means the result is false. It may well be true, and the company may have done careful work.

What it means is that the honest sentence you can write is "Neocis reports a forty-five per cent reduction in procedure time," and the sentence you cannot write is "a multi-site study found a forty-five per cent reduction." Those two sentences make very different promises to whoever reads them next, and one of them is not supported.

Notice also what is missing from the claim itself. Procedure time is reported, and placement accuracy is not. For implant surgery, accuracy is the thing that matters clinically, and its absence from an efficiency claim is worth a question of its own.

Attribution Is Not a Hedge

Here is the part that changes how you write and speak about technology inside your own organization.

When you attach a vendor's name to their own number, you are not softening the claim or covering yourself. You are stating what kind of fact it is.

"AI reduces missed caries by a substantial margin" is a claim about the world, and you are now responsible for it.

"VideaHealth reports that in its clearance trial, dentists using the system missed forty-three per cent fewer caries" is a claim about what a company found and published, which is true, checkable, and considerably more useful to whoever hears it.

The attribution is not a caveat. It is the information. It tells the listener exactly how much weight to put on the number, which is something you cannot do by adjusting the adjective.

This matters most when you are the one repeating the claim. Every number you pass along without its source becomes, in the next room, a thing that everybody knows.

Where Numbers Go Wrong Without Anyone Lying

Three failure modes worth recognizing, because they show up constantly and none of them involve dishonesty.

Compression. A finding gets summarized, the summary gets summarized, and qualifiers fall off at each step. One technology brief I reviewed recently contained two figures in its own summary bullets that did not appear anywhere in its body. The research was sound. The compression manufactured the errors.

Cherry-picking by omission. A study reports three numbers and a slide reports the two flattering ones. This is the most common form, and it is nearly invisible, because everything presented is accurate.

Category drift. A figure measured in one setting gets repeated about a different one. A result from a large academic health system becomes a claim about what happens in a two-operatory practice, and nobody ever decided to make that leap. It just happened over three retellings.

The defense against all three is the same, and it is unglamorous. Go one source back. Not all the way to the original paper, usually. Just one step closer than whoever handed it to you.

Making This Normal in Your Organization

The questions only help if they get asked when you are not in the room, which means the habit has to belong to the organization rather than to you.

The cheapest way to install it is a standing question on any purchase proposal. Not a form. One line: what is the evidence for this, how many, and where.

The second cheapest is to model it visibly. When somebody brings you a number, ask where it came from, in front of people, warmly. Do it four or five times, and you will notice team members arriving with the source already attached, because they have worked out that you are going to ask.

And be honest when you cannot answer it yourself. A leader who says "I do not know where that figure came from, let me find out" is teaching the standard far more effectively than one who is never caught without a citation.

Frequently Asked Questions

Is it reasonable to expect independent data from a small company?

No, and you should say so. A startup with a genuinely new product will not have multi-site independent validation, and demanding it would mean never adopting anything new. The point is to know that you are buying on promise rather than on evidence, and to price and pilot accordingly.

What if the vendor cannot answer because the person in front of me is in sales?

Entirely normal and not a mark against them. Ask them to come back with it. A company with the evidence will send it within a day. The follow-through is the signal.

How do I handle a claim I cannot check at all?

Treat it as absent. Not false, absent. Make the decision on the things you can verify, and if the unverifiable claim is the reason for the purchase, that is the finding.

Does this apply to references and site visits?

Yes, with one addition. Ask for a reference at your scale, in your situation, and ask the reference what they would do differently. The second question produces more useful information than the first.

What if my team thinks I am being difficult?

Explain what you are doing and why, once. Most people find it a relief. The alternative is a culture where nobody feels able to ask, and in that culture the expensive purchase always goes through.

Final Thoughts

There is no adversarial version of this that works better than the friendly version.

You are not trying to catch anyone. You are trying to find out what kind of thing you are being told, so that you can decide how much weight to put on it. That is a reasonable thing to want, and every competent vendor knows it.

So ask the seven. Ask them evenly, ask them early, and pay attention to the shape of the answer rather than its content.

And when you repeat a number to your own team, carry its source with it. That habit costs you four words, and it is the difference between informing people and starting a rumor.

Ask who measured it.