The words of the press release, and what they conceal

Press releases are written with care, often reviewed by lawyers. Every cautious turn of phrase is there for a reason, and that reason is almost always instructive.

The glossary of caution

• "Up to" X times faster. The best case measured, under the most favourable conditions. The average is lower, sometimes much lower.
• "Can" do this. It has managed it at least once. Says nothing about consistency.
• "In certain cases." Those cases were chosen, and they are rarely described.
• "Rolling out." A fraction of accounts have it. Often a small fraction, often in the US first.
• "Soon." No temporal value. It is an intention.
• "Expert-level performance." On a given benchmark. A human expert does more than sit benchmarks.
• "We are working on." It is not done, and the difficulty is probably real.

The comparison that isn't one

"Our model beats the previous one by 40%" means nothing until you know 40% of what, measured how, and relative to what baseline. Going from 2% to 2.8% is a 40% increase.

Be especially wary when the comparison targets an older version of a competitor, or the competitor's weakest offering. Both happen, and both are spotted simply by checking which version was used and on what date.

What, conversely, is a good sign

A press release that states where its model is weak, publishes a map of its limitations, gives a failure rate, or describes its measurement protocol precisely enough to be challenged. That transparency costs the person offering it something, and that is exactly why it is worth more than a promise.

Do this right now: open the latest press release from a lab and count the "up to"s, the "can"s and the "soon"s. Then look for a single acknowledged limitation.