It Will Say It With Total Confidence

A customer asks whether you cover Formby.

The reply goes out in ninety seconds. Friendly, well written, exactly your tone. Yes of course — and we can usually get someone to you within the week, typically around £200 for a job that size.

You don't cover Formby. You have never charged £200 for that. And nobody has the faintest idea where the number came from, least of all the thing that said it.

Nothing errored. Nothing went red. It simply made something up, beautifully, in your name, to a stranger.

Software breaks loudly. This doesn't.

Every piece of software you've ever used fails in a way you can see. The page won't load. The card's declined. Something goes red and says it's gone wrong. It's annoying, but it's honest — you know the moment it happens.

AI fails the other way. Politely, fluently, at full confidence. Ask it something it doesn't know and it will not say “I don't know”. It will produce the most plausible-sounding answer available, in your voice, and hand it to your customer.

It has no way of telling you when it's guessing. That isn't a bug someone will patch next year — it's a property of the thing. Which means the whole question of whether it's safe to use comes down to what you put around it.

What's actually at stake

For an internal task, a confident mistake is a nuisance. You spot it, fix it, move on.

In front of a customer it's different, because the mistake arrives wearing your name. A wrong price is a conversation you now have to have. A promised date you can't hit is a review. A confident answer about something regulated — a certificate, what's covered, whether you're insured for that — is a genuine problem, and “the computer said it” has never once worked as a defence with an angry customer.

So the operators who get burned aren't the ones who were careless with the technology. They're the ones who were careless with the handover — who let something start speaking for their business before deciding what it was allowed to say.

It isn't the machine's fault, and that matters

We get called in to clean these up, and they all have the same shape. Somebody set the thing up, got it eighty per cent working, and left. No checkpoint. No review. No named person who owns it when it says something daft on a Saturday.

The technology wasn't the problem in a single one of them. Nobody finished it. That's a far less exciting story than “the AI went rogue”, and it's the true one nearly every time. The failure is a build decision — usually the decision to stop early — and build decisions have people attached to them.

Which is good news, incidentally. Technology you can't control. Build decisions can be made differently.

The rule: draw the line before you switch it on

One rule, and it's the whole of how we approach this. Decide what it may say on its own and what always comes to a human first — before it speaks to anybody.

First list: the facts that don't move. Your hours. Which areas you cover. Where to park. How the process works. What to do about that noise the boiler's making before someone can get there. These are safe because they're checkable and they don't change week to week.

Second list: anything with a price, a promise, a date or a complaint attached. Those still get drafted instantly — that part is worth keeping, the customer sees movement in seconds — but a person presses send.

That's the whole of what we mean by keeping a human in the loop. Not a bot pretending to be you. A system quick enough to hold the window open, honest enough to know which questions are above its pay grade, and a person on the ones that matter.

And the line moves. After a month you'll have evidence rather than nerves — you'll see the questions it has answered correctly forty times, and those can shift to the first list. Trust gets earned in that direction, never assumed in advance.

This week's homework: the Two Lists

Ten minutes and one sheet of paper — and do it whether or not you're buying anything, because the exercise is just as useful pointed at a new member of staff.

Left column: everything about your business a stranger could safely say on your behalf without checking with you. Right column: everything that always, without exception, comes to you first.

Two things usually fall out of it. If you can't fill the left column, nothing is ready to answer for you yet — and the reason is knowledge captivity, not technology. If the right column is empty, you haven't thought hard enough; go back and look at prices, dates and promises.

Most operators have never drawn this line explicitly. Which is exactly why it gets crossed.

If you want to see what that looks like built properly, how we work walks through where the checkpoints go and why. The shape is the same whether we build it or you do.

And hit reply — we read every one. This week: what's the one thing you'd never let anything say to a customer without you seeing it first? Genuinely useful for us — those answers are how the right-hand column gets designed.

Built by operators, for operators.

— Kane & George