The most common measurement mistake we see is trying to measure the value of an AI project after it’s been running for six months. By that point the baseline is gone, the context has changed, and you’re left with a vague sense that things feel faster — which is not something you can take to a board.
Measurement needs to be designed before the project starts. Here’s how to do it.
Define what success looks like before you build
This sounds obvious and is almost universally skipped.
Before you commission any AI work, write down in plain language what the project is supposed to do. Not “improve efficiency” — something specific: “reduce the time a claims assessor spends processing a standard application from 25 minutes to under 8 minutes” or “reduce the monthly volume of repeat customer queries by 30%.”
Vague goals produce vague outcomes that are impossible to evaluate honestly. Specific goals tell you clearly whether the project worked.
Establish a baseline
Whatever you’re trying to improve, measure it now, before anything changes.
If you want to reduce processing time, time your current process. If you want to reduce error rates, count your current errors. If you want to generate more revenue from a dataset, understand your current baseline revenue from that source.
You’ll be surprised how often organisations don’t have this data. In those cases, a few weeks of careful manual measurement before starting a project is worth more than months of post-implementation guesswork.
Pick a small number of meaningful metrics
More metrics is not better. Three clear metrics you can measure reliably are more valuable than fifteen metrics you’re not sure about.
Good AI metrics are usually one of three types:
Efficiency metrics: time saved, volume processed per person per day, cost per transaction. These are the easiest to measure and usually the fastest to show movement.
Quality metrics: error rates, exception rates, rework rates, customer satisfaction scores. These often take longer to demonstrate improvement but are important because speed gains that come with quality degradation aren’t real gains.
Business outcome metrics: revenue, margin, churn, conversion. These are what you ultimately care about, but they’re often several steps removed from what the AI is actually doing, which makes attribution harder. Be careful about claiming causal links you can’t demonstrate.
Be realistic about the measurement window
Different types of AI projects show results on different timescales.
Document processing or task automation improvements are typically visible within weeks — the volume goes up, the time goes down, and you can see it clearly.
Projects that improve decision quality (better pricing, better risk assessment, better customer targeting) may take months before the downstream business effects are visible in your numbers.
Set expectations accordingly. A project that produces no visible results in 90 days is probably not working. A project that needs 90 days to demonstrate quality improvements is not necessarily failing.
Account for the costs honestly
The cost of an AI project isn’t just the development or licensing fee. Include:
- The time your staff spend working with the vendor or consultant
- Any new infrastructure or tooling costs
- The ongoing cost of maintaining, monitoring, and updating the system
- The cost of the human review layer (most production AI systems still need one)
A project that saves 10 hours of staff time per week but requires 4 hours of weekly oversight is a 6-hour net saving, not a 10-hour one.
Know when to cut losses
Not every AI project works. Some don’t work because the technology isn’t ready for the problem. Some don’t work because the underlying data wasn’t good enough. Some don’t work because the problem was less well-defined than it seemed.
The right time to make this call is when you’ve had enough time to see meaningful signal, you’ve ruled out fixable implementation problems, and the data clearly shows the project isn’t delivering against the metrics you defined upfront.
Killing a project that isn’t working is a success, not a failure — it frees up resources for something that will. The failure is running a project for two years without ever honestly evaluating whether it’s working.
The short version
Define what good looks like before you start. Measure where you are now. Pick a small number of metrics and track them honestly. And be willing to call it if the numbers aren’t there.
That’s it. It’s not complicated — it just requires doing it before you start, not after.