DIGNALEGI

Worth reading. A personal index selected by Ratipcan Uysal.

Digna Legi guide / AI ROI

How to Measure AI ROI Without Fooling Yourself

Measure AI ROI across the whole workflow: establish a baseline, include review and rework, track total lead time, and decide when to scale, redesign, or stop.

By Ratipcan Uysal11 min read

Paper work units pass through a mechanical bottleneck toward a balance, where one yellow unit reaches the value side.

Ask the person presenting an AI productivity win one question:

Where did the saved time go?

If a team says a task became 40 percent faster, but the same number of people ship the same amount of work at the same quality and cost, the business has not captured a 40 percent return. It has a faster task inside an unchanged system.

That distinction is where most AI ROI claims fall apart. A model produces an answer quickly. A worker then checks it, repairs it, waits for approval, moves it into another tool, explains it to a colleague, and absorbs the failures that escaped review. The demo measures generation. The business pays for the workflow.

A faster task is not yet a better business.

AI creates a return only when it changes a valuable outcome: more accepted work, shorter end-to-end lead time, higher conversion, lower loss, better quality, or capacity deliberately moved to something more valuable. The gain must survive review and rework. It must clear the new bottleneck it creates. And the organisation must decide how to capture it.

This guide is a method for finding that return without pretending every benefit can be reduced to token cost—or that every uncertain benefit should be ignored.

Measure the system, not the model

The cheapest part of an AI system is often the visible inference. The expensive parts live around it: finding the right context, integrating tools, changing permissions, training people, checking outputs, recovering from errors, handling exceptions, and coordinating the work that becomes possible when one step accelerates.

Why AI is booming, but productivity is not makes the central problem clear: intelligence can become cheap while coordination remains expensive. An organisation does not receive the model's theoretical productivity. It receives whatever its operating system can absorb.

A useful ROI account begins with two quantities:

Captured value
The revenue, useful throughput, quality improvement, risk reduction, or released capacity that the organisation actually realises.
Total system cost
Model and licence fees plus integration, data preparation, training, review, rework, governance, change management, and the opportunity cost of maintaining the new workflow.

ROI is not “hours claimed as saved multiplied by salary.” It is captured value minus total system cost, compared with a credible baseline. If time is saved but nobody removes a queue, increases useful throughput, avoids hiring, reduces loss, or reallocates capacity, the saving remains potential rather than realised value.

This is why a local success can coexist with a global disappointment. A team generates specifications twice as fast, but engineering remains the constraint. Engineers produce more code, but review and release queues lengthen. Support agents close more tickets, but complex cases are reopened. Marketing creates more variants, but distribution and learning do not improve.

Measure from the request entering the system to the valuable outcome leaving it.

Task time matters. It is simply not the finish line.

Find the bottleneck before you automate

Automation applied before diagnosis usually makes one of three things happen faster: work nobody needed, work that waits elsewhere, or work that creates more checking.

Start by mapping one complete unit of work. Mark the time spent doing, waiting, reviewing, correcting, transferring, and deciding. Then ask which step limits the valuable output of the system.

That bottleneck may be production. It may also be approval authority, missing customer context, unclear ownership, an overloaded reviewer, an integration boundary, or a decision nobody wants to make. Generating more material upstream will not dissolve those constraints. It may bury them.

You Cannot Mandate an AI Transformation is relevant here because adoption orders do not redesign work. Tokens, Hours, Points, and Other Curious Proxies is the corresponding measurement warning: an easy count is not automatically a valuable one.

Before testing a tool, ask:

  • Would we still want this work if producing it were free?
  • What downstream action makes the output valuable?
  • Who verifies it, and how expensive is that verification?
  • Which queue grows if this step becomes ten times faster?
  • What will we stop doing if the tool works?

The last question forces capture into the design. Without it, the pilot can produce a productivity story but no operational consequence.

Choose work where verification is cheaper than generation

The best initial candidates are not simply tasks a model can perform. They are tasks where useful output can be recognised cheaply and mistakes are contained.

Four conditions make an AI workflow easier to justify:

  • The work occurs often enough for learning and integration costs to be recovered.
  • Inputs and outputs are sufficiently bounded to compare cases.
  • A qualified person or reliable test can verify the result faster than producing it from scratch.
  • A better result moves the system's current constraint rather than feeding a downstream queue.

This creates four practical deployment modes:

Scale
Frequent, bounded work; cheap verification; contained errors; downstream capacity exists.
Assist
Judgment remains important, but a reviewer has the context to accept, reject, or repair the output efficiently.
Redesign first
The model can accelerate a step, but the real constraint is approval, integration, incentives, or handoffs.
Do not automate yet
The value is ambiguous, errors are consequential, verification is as expensive as creation, or no one owns the outcome.

This is also why tool quality cannot be inferred from a benchmark alone. How to evaluate LLMs before production argues for evaluations that resemble the actual environment. Your Agent Isn't Dumb. Your Tools Are. shows the other half of the system: an agent's result depends on the interfaces, context, and feedback it can use.

The economic question is not “Can the model do it?” It is “Can the system verify and use it?”

Establish the baseline before the tool arrives

Once a team likes a tool, memory becomes generous. Old work feels slower and worse than it was. Exceptional demonstrations become typical cases. Time spent prompting, reviewing, and correcting disappears from the story.

Record the baseline first. For a defined unit of work, capture:

  • total lead time from request to accepted outcome;
  • hands-on production time;
  • waiting and handoff time;
  • first-pass acceptance and rework rate;
  • defects, reversals, escalations, or reopenings;
  • the downstream outcome the work is meant to change;
  • the fully loaded cost of producing and governing it.

Segment the results by task type and worker experience. Averages conceal where the tool helps and where it interferes.

That is not a theoretical concern. An NBER study of 5,179 customer-support agents found a 14 percent average productivity increase from AI assistance, but a 34 percent improvement for novice and lower-skilled workers and minimal impact on experienced, higher-skilled workers. The useful conclusion is not “AI improves support by 14 percent.” It is that impact can vary dramatically across the people and cases inside one workflow.

The opposite result can also be real. A 2025 randomised study by METR observed 16 experienced open-source developers completing 246 tasks in mature repositories. With early-2025 AI tools available, they took 19 percent longer—even though they expected to be faster and later believed they had been faster. METR explicitly warns against generalising that result to most software work. Its later experiment could not produce a reliable current estimate because participation and measurement had changed.

Together, the studies justify a stricter rule: measure the people, tasks, tools, and period you actually intend to change. Do not import a universal ROI number from somebody else's setting.

Run a pilot that is allowed to lose

A pilot designed only to demonstrate adoption will demonstrate adoption. A useful pilot is designed to change a decision.

Compare similar units of work with and without the AI-assisted workflow. Random assignment is best when practical; a staggered rollout or carefully matched comparison is better than an unexamined before-and-after story. Run long enough to include ordinary work, edge cases, learning effects, and the novelty wearing off.

Write the experiment brief before the result is visible:

Decision
What will this evidence decide: scale, narrow, redesign, or stop?
Unit of work
What repeatable object moves through the workflow?
Baseline
What does performance look like now, including variation?
Treatment
Which tool, model, instructions, context, integrations, and human roles change?
Outcome
Which business or workflow measure must improve?
Guardrails
Which quality, safety, customer, or employee measures must not deteriorate?
Capture plan
What queue, cost, capacity, or decision changes if the pilot succeeds?
Kill condition
What result means the workflow should not scale in its current form?

Do not use logins, prompts, generated words, accepted suggestions, or token volume as the primary success measure. Those reveal usage. They do not reveal return.

A pilot without a kill condition is a sales demonstration.

Calculate value at three layers

AI ROI should be examined at three layers, in order. Jumping from the first to the third is how a short demo becomes an implausible company forecast.

Task layer
Did the person complete a defined task faster or better after including prompting, verification, and correction?
Workflow layer
Did accepted throughput, total lead time, rework, reliability, or queue length improve across the complete process?
Business layer
Did the organisation earn revenue, prevent loss, improve retention, avoid a cost, reduce risk, or redeploy capacity to a higher-value constraint?

A strong task result with a weak workflow result means the bottleneck moved. A strong workflow result with no business result means the output may not matter, the experiment is too short, or the organisation has not captured the gain. A business result with no credible task or workflow mechanism may be real, but the team should not automatically attribute it to AI.

The distinction also prevents double counting. Suppose an assistant saves each of ten people two hours a week. The organisation cannot simultaneously claim the full salary value of those hours, the value of extra output produced during them, and avoided hiring for the same capacity. Choose the value that was actually captured and show the mechanism.

Five ways AI ROI reports lie

Most bad ROI reports are not fabricated. They are built from measurements that cannot support the conclusion.

  1. They count token cost and ignore system cost. Cheap generation can create expensive review, integration, and exception handling.
  2. They treat self-reported time saved as observed time saved. People are poor stopwatches, especially when a tool feels fast. Self-report is evidence about experience, not a substitute for elapsed-time measurement.
  3. They call output volume productivity. More code, copy, tickets, or documents can be negative productivity if acceptance, use, or outcomes do not improve.
  4. They compare before and after without a counterfactual. Work mix, staffing, seasonality, learning, and unrelated process changes can explain the difference.
  5. They average away the deployment decision. A positive mean can conceal a large gain for novices, no gain for experts, or failure on high-consequence cases.

There is a sixth trap worth naming: counting potential capacity as cash. Time released is not automatically money saved. It becomes valuable when the organisation removes cost, avoids cost, increases a valuable output, or deliberately reallocates the capacity.

Never monetise a saved hour until you can name what changed because it was saved.

Decide how the organisation will capture the gain

The result of a pilot is not merely positive or negative. It should produce an operating decision.

If task time falls but end-to-end time does not, redesign the downstream constraint. If throughput rises but quality falls, narrow the task, strengthen verification, or stop. If novices gain and experts do not, deploy selectively rather than imposing one workflow on everyone. If the tool helps only after extensive context preparation, include that preparation in the product and cost model. If capacity is released but nothing changes, assign it explicitly or remove it from the ROI claim.

The largest returns may require more than inserting a tool into the old process. Research on the productivity paradox argues that general-purpose technologies often need complementary investments in skills, organisational change, and new processes before their full effects appear. That does not excuse a weak pilot. It explains why a good local result can still require a second decision about redesign.

This is the moment to choose among four actions:

  • Scale when the complete workflow improves, guardrails hold, and the capture plan is credible.
  • Narrow when the gain is concentrated in particular people, task types, or risk bands.
  • Redesign when the tool reveals a different bottleneck or new coordination cost.
  • Stop when verification, failures, or weak outcomes consume the apparent gain.

Stopping is not evidence that the experiment failed. It is evidence that the experiment performed its function before the organisation paid to scale the mistake.

Use a one-page AI ROI scorecard

For each workflow, keep one record that can be read without the vendor deck.

Valuable outcome
The customer or business result this work is meant to change.
Unit and boundary
The repeatable unit of work and the start and end of the measured system.
Current constraint
The step that presently limits useful throughput.
Baseline
Lead time, hands-on time, acceptance, rework, outcome, and fully loaded cost before AI.
AI intervention
The exact model, tools, context, permissions, and human responsibilities introduced.
Observed change
The difference at task, workflow, and business layers, with uncertainty and segment variation visible.
Total system cost
Technology, integration, data, training, review, rework, governance, and maintenance.
Capture mechanism
The queue removed, capacity reassigned, cost avoided, risk reduced, or valuable output increased.
Guardrails and exceptions
What must not worsen and which cases stay outside the workflow.
Next decision
Scale, narrow, redesign, or stop; owner and review date included.

The scorecard is deliberately operational. It replaces a universal ROI promise with an inspectable chain of evidence.

The unit of AI ROI is the changed system

AI can make a task dramatically faster. It can also make experienced people slower, move work into review, increase low-value output, or release capacity an organisation never uses. All of those outcomes can be true because “AI productivity” is not one property of a model. It is a result produced by a particular tool, task, person, workflow, and operating decision.

The serious question is therefore not how much intelligence costs. It is whether the organisation can turn that intelligence into a verified outcome and capture the gain without creating a more expensive constraint elsewhere.

The unit of AI ROI is not the token. It is the changed system.

Measure the baseline before belief changes it. Test the whole workflow. Let the pilot lose. Count verification and coordination. Segment the result. Then make the gain operational—or leave it out of the claim.

Sources and further reading