I thought I was breaking new ground using AI on an old-school problem. It turns out it amplified my ignorance. I learned to pay attention to its overconfidence (and my own).
We needed to measure a distance on a moving component. I asked an AI model how you’d do it. Time-of-flight sensors, it said. These sensors send out light, measure how long it takes to come back, and use that to compute the distance. Confident, standard, and as far as I could tell, correct. So, we started building.
After several weeks of design work and prototyping, of long nights babysitting an AI tool and creating development plans, of going down debugging rabbit holes with the AI tool by my side (always confident that it had the answer ‘this time’), I asked a question basic enough to be embarrassing. Time of flight measures from a reference point to a target. Ours isn’t fixed. The part moves, and it changes angle as it moves. There’s no datum. And with no datum, there’s no measurement. And the AI tool confidently agreed that we’d found the issue that would kill the project.
The lazy read is that the AI tool failed. It didn’t. It knew sensors, but it did not know my use case well enough. It just didn’t know enough to properly vet whether it was applying the right sensors to the job and didn’t realize I didn’t have the knowledge going in to evaluate the baseline assumptions. I found that AI won’t highlight that boundary for you; it’ll plough confidently ahead even if it’s making dangerous assumptions about your situation.
I walked into the same problem from the other direction while writing this newsletter.
I’d absorbed the idea from somewhere that most of the images online now are AI-generated. Before putting it in a sentence, I did the responsible thing and checked it rather than just asserting it. The search handed back a clean figure in Google’s new AI-driven search tool: seventy percent, unequivocally. Into the draft it went. Then, in editing, I found myself wondering: where did that 70% number actually come from? I opened the primary source. It said images “may involve AI tools.” May involve. If I hadn’t checked it, I’d have taken a speculative number and presented it as gospel. I cut it before the issue went out.
Here’s the part I keep turning over. I didn’t cut a corner. I checked. The check just handed back false confidence, because the layer I checked against had already oversimplified and thrown the necessary context away.
Now set the two side by side. In one, the AI produced the claim and a person with no domain knowledge couldn’t catch it for weeks. In the other, the AI produced the claim, and the catch only came from stepping outside the tool to the primary source. In neither case did expertise do the catching. What did the work both times was the same dull move: go and look at the actual thing. The product. The source.
And notice the real difference between them, because it shows that some situations carry more risk than others. The sensor work had a build step. I couldn’t proceed without putting it against a physical object, and the object refused to cooperate. Reality wouldn’t let the mistake slide. But a number in a sentence has no such step. It sits there looking finished, unless you’re the one forcing the test against reality. The numbers in a deck — cited once, never operated against — are exactly the ones that never get audited, even when those numbers can move real money, or worse.
So, what do you do, if you’re the non-technical person on the hook for the call? The popular line is that AI is ‘a great intern’. But that holds only for work you could do slower yourself. The better comparison is the specialist you hire precisely because you lack the skill: the software developer, the structural engineer, the tax attorney. You can’t grade their work. What you can do, and must, is make sure they know every aspect of the foundations of the problem that isn’t obvious, and that you get a real read on confidence, not just an answer. Every specialist disaster starts with a fact about your situation they were never told and never thought to ask for.
The catch is that “write better prompts” is advice you can only take in hindsight. I thought I did it the right way; I pointed the AI tool to our product, including videos showing how it worked. I didn’t realize that it didn’t key in on one of the vital characteristics of our product, that the part in question moved. What I could have done, with no domain knowledge at all, was make it prove it understood the product before I leaned on the answer. It’s easy to get excited and jump straight for the solution, as I did. Instead, ask what it thinks the situation is. Ask what it’s assuming. Ask what would have to be true for the solution to work here. It would have said: a fixed reference point. And I’d have said: there isn’t one. Twenty minutes, before a line of code. The same check was available on the number before it got worked into my draft.
Interrogate before you depend: you can’t audit its expertise, but you can audit whether it understood your case, and your case is the only place it can hurt you. Then put something that didn’t produce the claim between the AI conclusion and the real world — a primary source, a second model, a physical object you have to touch — because odds are you’ll still under-specify. I did, both times.
The AI tool doesn’t know where its world stops matching yours.
Zain
P.S. What’s the last confident answer you accepted because you had no way to check it? Hit reply. I read every one.