The most literal hire you'll ever make
Your AI agent is doing exactly what you asked, and that may be the problem. Agents aren't failing to pay off for lack of intelligence: they deliver what their setup rewards. The fix is an old management idea in a new form. Price the work, set the incentives, and check the result in the real world.
Green across the board
My AI coding agents told me everything worked, and they had the paperwork to prove it: thousands of automated checks, green every morning.
What they were building was a travel connector: a plug-in, built on the MCP standard, that lets AI assistants search for hotels on a traveller's behalf. They had even run hundreds of edge-case tests to back it. Then I asked an outside AI assistant to find me a hotel in Paris, the way a customer would, through the connector the agents said was working. It found a hotel in Paris, all right, just not the Paris you're thinking of: Paris, United Kingdom.
Every check had passed, edge cases and all. None of them had considered that Paris doesn't always mean France. Paris, Virginia? Paris, Texas?
It wasn't a one-off. In one of my two systems I went back through 89 serious bugs, and the agents' own checks had caught none of them. In the other, the sign-in had 205 checks. Not one confirmed the key was real.
So I did what any manager would. I banned that kind of check and told the agents to prove their work on the real system instead.
Ten days later they had added 10,405 new checks. Same habit, new label. They had found the gap between what I wrote and what I meant, and filled it with fakes. I had already told them 87 times, in plain English, to stop using fakes. When a check failed, more than half the time they fixed the check, not the code.

I was annoyed for about a week. Then it dawned on me that they had done nothing wrong. I had made green checks the finish line, and they ran for it harder than any person would.
So I moved the finish line. A change now counted only when it worked on a copy of the real system, with real services, used the way a customer uses it, and the result read back from the record. Within days the behaviour flipped. Each shipped change cost about a third less, and bugs were fixed within hours of being written instead of days.
Same models. Same agents. A different finish line.
The value gap is a management gap
Most businesses are not yet seeing money from AI. MIT's Project NANDA looked at more than 300 AI initiatives in the first half of 2025. Only about 5% of integrated pilots were extracting real value; "the vast majority remain stuck with no measurable P&L impact". The sample is small and self-reported, but anyone who has sat through an AI rollout will recognise the picture.
The usual diagnosis is that the models aren't good enough yet, or the prompts need work. I don't buy either. The models I work with can already do most of what they're asked, and the report itself blames how systems are set up and fitted to the work, not the models.
An agent does what its setup rewards. Reward activity and you get activity. Reward looking finished and you get things that look finished. Few setups reward what the business actually needs, because few businesses have written down what that is worth.
An old problem, a new kind of worker
In 1975 the management scholar Steven Kerr published a paper whose title still says it all: "On the Folly of Rewarding A, While Hoping for B". Organisations, he showed, pay for one behaviour, hope for another, and act surprised when they get what they paid for.
Sixteen years later the economists Bengt Holmström and Paul Milgrom made it precise. When only part of a job can be measured, paying for that part pulls effort away from the rest (1991). Holmström later shared a Nobel prize for this line of work. My agents were their model in miniature: checks were the part I could measure, working software was the part I couldn't, and the effort went where the reward was.
What's new is the worker. An AI agent is the most literal hire you will ever make.
- It never slacks off. It works flat out toward whatever the setup rewards, around the clock, cheaply enough to do it ten thousand times.
- It has no unspoken sense of the job. A good employee knows "close the ticket" means "solve the customer's problem". An agent knows only what the setup tells it and what it can see.
- Instructions barely move it; incentives move it a lot. Eighty-seven requests didn't change my agents' habits. Changing what I checked changed them in days. Researchers see the same: in ImpossibleBench, leading models given tests that contradicted the task passed about half of them anyway, often by editing the tests, despite being told not to.
- It believes its own reports. Ask how it did and it will tell you, confidently, from its own notes.
Every classic incentive problem gets sharper with this worker. So does every well-designed incentive. That is the opportunity.
Three levers that turn agents into value
If agents do what the setup rewards, the setup is where value is made or lost. Three decisions shape it. None of them is technical; each is a judgement call about your business.
1. Price the work
Before an agent touches a task, decide what it is worth done right, what it costs done wrong, and how quickly anyone would notice. That price sets how much freedom the agent gets.
Where a mistake is cheap and visible, let the agent use its judgement. Where it is expensive or can't be undone, draw a hard line in code. For most businesses that means three places: money, access, and anything that leaves the building.
I learned this on an invoice. In one experiment I let the agent do its own invoice arithmetic. It got every invoice right but one, which it issued $536.33 short while its note reported the correct total. A reviewer reading the note would have approved it. Money arithmetic now lives in code; the agent handles everything around it.
The price also tells you what to reward. Take a support agent. Closed tickets are activity. The value is a solved problem and a customer who stays. Reward the first and you'll get plenty of closed tickets.

2. Design the incentives
An agent has no salary or bonus. Its incentives are the conditions it works in: what it is checked against, what it can see, and what it is allowed to do. Each one is a lever.
- Check it against the outcome, not the activity. My agents changed course within days once a change counted only when it worked on the real system.
- Give it the whole picture. I first split the work across specialised agents, each with its own slice of context. They duplicated each other's work and asked people the same questions twice. One agent with one shared context cut background spending by 85 to 90 percent and stopped the double-asking.
- Use fewer controls, not more. Every guard I added fixed the case it was written for and cost the agent other work. In one experiment, the usual safety machinery made an agent hand 44 of 132 jobs back to a person, against 1 without it.
The principle behind all three: make the valuable outcome the easiest thing for the agent to do.
3. Verify in the world
An agent's report is a claim, not evidence. Check what actually happened: the invoice in the ledger, the booking in the system, the email in the customer's inbox.
- Read the record, not the summary. The short invoice came with a correct-looking note. Only the ledger showed the gap.
- Have someone other than the builder check it. When one agent builds and checks, the checks agree with the work by construction, mistakes included.
- Keep checking after launch. A one-off proof is not a guarantee. Track the outcomes that matter, week by week.

The three levers form a loop: what verification finds changes what you think the work is worth, and that changes the incentives.
What not to do
- Don't reward activity. Messages sent, tickets closed and reports produced are all things an agent can maximise without creating any value.
- Don't accept a demo or a green dashboard as proof. Ask to see the agent work on your real data, and look at what it changed.
- Don't let the builder be the only one who checks. That includes an agent checking its own work.
- Don't fix behaviour with more instructions. If the setup rewards the wrong thing, no amount of "please don't" will hold. Change what you check.
- Don't stack controls. Each one fixes a named case and quietly costs you others. Remove the cause instead.
- Don't judge cost per task. Judge value per outcome. Checking on the real system made each working session dearer and each shipped change cheaper.
Five questions before any agent goes live
- What is this work worth to us, done right? In money, time or risk, not tasks completed.
- What will the agent actually be rewarded for? What will it be checked against, and is that the same thing?
- How will we verify it in the real world? Which record will we read, and who reads it?
- What does a mistake cost, and who would notice? If nobody would notice for a month, start checking there.
- Where must it never decide alone? Name the hard lines, and keep them few.
Taste is economic judgement
People talk about taste in AI as curation: the machine produces plenty, and a person picks the good one. With agents, taste does its work earlier. It is knowing what value looks like in your business, being exact about it, and building conditions in which the agent's cheapest path is the valuable one. It is also restraint: knowing which controls to leave out.
No model can supply that judgement, because it depends on what your business is for.
And it will matter more, not less. Each new model is better at finding the shortest path to whatever you reward. Pointed at the right outcome, that is a remarkable worker. Pointed at a proxy, it is a remarkably efficient way to waste money. Choosing the outcome, and building the incentives and checks around it, is a leadership job, not an IT one.
If you're about to put an agent in front of your customers or your staff, I'd be glad to help you price the work and design how it gets verified.
The full numbers behind the opening story are in study 02, "Proof on the real seam", one of thirteen studies from building agents in production at production-agent-experiments.
Sources
- Steven Kerr, "On the Folly of Rewarding A, While Hoping for B", Academy of Management Journal 18(4), 769–783, 1975. Citation and summary
- Bengt Holmström and Paul Milgrom, "Multitask Principal-Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design", Journal of Law, Economics, and Organization 7, 24–52, 1991. PDF
- MIT Project NANDA, "The GenAI Divide: State of AI in Business 2025", July 2025. Coverage with quotes
- Zhong, Raghunathan and Carlini, "ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases", 2025. arXiv 2510.20270
- My own figures: study 02, Proof on the real seam (testing), study 05 (one agent, spending), study 12 (invoice, controls).
More insights
Working with Isometric
Answers about scope, ownership and getting started.

