How to measure automation results without BI analytics

Measuring automation results without complex analytics means assessing whether processes, after a change, achieve better predictability and lower handling costs through simple operational measures:
(1) cycle time and working time;
(2) quality and number of errors;
(3) volume stability and timeliness.

Last updated: 2026-02-20

Quick facts

• The simplest set of checks covers turnaround time, the number of corrections and timeliness.
• A “before/after” comparison requires a fixed definition of a case and the same unit of measure.
• A lack of financial data can be replaced with time costs and quality indicators.

The short answer

The results of automation can be assessed without elaborate tools if measurement rests on a few fixed signals from the process that are easy to obtain. What matters is a consistent recording method and unambiguous interpretation thresholds.

• Identify one “healthy” run of the process as the reference for comparison.
• Record exceptions and manual workarounds as a measure of how mature the automation is.
• Check for side effects: work shifting to other roles or channels.

Assessing automation is often confused with advanced analytics, whereas in many organisations disciplined definitions and regularly noting a few measures are enough. The biggest source of errors turns out to be inconsistent counting of “cases”, mixing exceptions with typical cases, and failing to distinguish system time from human working time. Correct measurement requires deciding what the unit of the process is, when a process counts as finished, and which states count as an error or a correction. Only then does a “before/after” comparison of periods show whether automation shortens the cycle, lowers the number of errors and improves predictability. Without this, even a successful improvement can look like a miss, because the data will have been collected in different ways. The rest of this article sets out the measures, thresholds and typical pitfalls.

Setting the goal and the unit of comparison

Measurement without elaborate analytics starts with one decision: which unit of work will be counted the same way every week. Most often this is a ticket, order, lead, invoice, booking or service case, but each of these labels needs its boundaries spelled out.

The unit of comparison should have three attributes: an unambiguous start (the moment of intake), an unambiguous end (the moment of approval or dispatch) and an exception status (when a case is “atypical” and must not go into the main count). In practice, a minimal register can be kept in a spreadsheet or in a list view in the operational system, as long as the fields are fixed.

This is where it helps to distinguish “cycle time” from “working time”. Cycle time describes how long the path from start to end takes, and working time describes how many minutes of actual handling a case required. Automation often shortens working time, but cycle time can stay unchanged if there are approval queues or dependencies between departments.

If the definition of a case is blurred, results are most likely to appear worse after automation, because of a different way of classifying cases.

A set of 5 operational measures without BI tools

The simplest dashboard of automation results fits into five operational measures that you can count by hand or from a system view. The key is regularity: the same measures at the same intervals.

1) Case volume – the number of process units per week or month. Interpretation: if volume rises with unchanged staffing, automation may be freeing up time or shifting the load to another stage.

2) Cycle time (median, not average) – the median is resistant to individual extreme cases. In simple conditions, it is enough to calculate the difference between start and end dates for a sample of 20–30 cases.

3) Working time – the sum of minutes of human work assigned to a case. If there are no time records, a fixed estimation method is enough: a short survey after each case is handled, for 1–2 weeks.

4) Quality – the number of corrections, complaints or returns per 100 cases. A quality indicator is often quicker to interpret than costs.

5) Timeliness – the share of cases completed within the agreed SLA. Even a simple limit such as “within 24 hours” or “within 3 days” lets you assess stability.

A definition consistency test lets you tell a real improvement in process capability from fluctuations caused by seasonality, without increasing the risk of errors.

Thresholds and interpretation: when is a result “good”

Without thresholds, numbers become a description rather than a decision. Sensible thresholds come from operational requirements: how many cases must go through each day, what error rate is acceptable, and what timeliness is the minimum for customers or an audit.

Thresholds can be set in three layers. Layer 1 is the “safety” threshold: no increase in errors or complaints after automation. Layer 2 is the “efficiency” threshold: a drop in working time per case by a set percentage or a set number of minutes. Layer 3 is the “predictability” threshold: a rise in the share of cases within SLA and a drop in the number of exceptions.

In interpretation, what counts is the direction and stability, not a one-off jump. A change considered favourable should persist for at least 3–4 consecutive measurements; otherwise the risk of random factors is high. With small volumes, it is better to assess trends quarterly but measure weekly.

If timeliness rises without an improvement in quality, cases are most likely being closed too aggressively without verifying the required fields.

A minimal “before/after” method without chaos in the data

A “before/after” comparison works only when the periods have similar working conditions and identical measure definitions. The minimal method assumes two measurement windows: a baseline from before automation and a stabilisation period after launch.

The “before” window should cover at least 2–4 weeks of typical work, with no unusual promotional activity or team changes. The “after” window should start after the learning period, when the process is no longer being “tested” by hand. If automation is introduced in stages, conclusions should be attributed to a specific stage, not to the whole project.

In practice, one table with baseline and target values and a “notes” column is enough, where you record disruptive events: holidays, outages, changes to approval policy. Such annotations are sometimes more important than the figure itself, because they explain jumps in the indicators.

To structure how you observe results in real implementations, it can help to read the Tacho App case study.

If the measurement windows include different types of cases, the comparison will show a change that comes from the work mix, not from automation.

A list of typical pitfalls when measuring automation

The most common measurement errors come from an organisation seeing only what is easy to count. Left uncorrected, these errors lead to wrong conclusions about effectiveness.

Pitfall 1: hidden manual workarounds. If employees carry out some steps outside the system, working time appears to fall and the number of errors rises later. It is worth counting exceptions and manual corrections as a separate maturity indicator.

Pitfall 2: shifting work between departments. Automation in one team can move the load onto approvals, accounting or complaint handling. Without a simple process map, measurement can reward local optimisation at the expense of the whole.

Pitfall 3: the wrong measure of time. Counting cycle time alone hides the fact that human working time has fallen. Counting working time alone ignores queues. Both measures are needed, though they may come from different sources.

Pitfall 4: no segmentation. Mixing simple and difficult cases blurs the effect. Minimal segmentation is a split by case type or intake channel.

A segmentation test by case type lets you tell the real effect of automation from an improvement caused by a smaller share of difficult tickets, without increasing the risk of errors.

What data sources are sufficient without analytics?

Sufficient data sources are those that are repeatable, complete and auditable without specialist tools. In practice these are: status lists from the operational system, event logs, complaint registers and simple time-tracking spreadsheets.

System data should contain at least the case identifier, intake date, completion date and final status. If the system does not provide human working time, a sample-based estimate is acceptable, provided the method is fixed and the same role makes the estimate.

Quality is easiest to measure through secondary data: the number of returns, complaints, repeat contacts and requests to correct data. In many processes, these are the signals that show the effects of automation fastest, even when volume is growing.

An important element is a change log: when a rule was introduced, when a form was changed, when validation was added. Without such a log, interpreting trends becomes prone to cognitive bias and to the team’s memory.

If data completeness falls below the level acceptable for an internal audit, the measurement will most likely become a description of individual cases rather than a picture of the process.

Automation without analytics versus with analytics: how to choose sources?

Without analytics, source selection rests on a format that can be verified by hand: case lists, simple exports and exception registers that can be reviewed and spot-checked. With analytics, technical verifiability matters more: consistent identifiers, time-stamped events and metric definitions in one data model. In the first variant, trust signals come from field completeness and a fixed measurement procedure; in the second, from tracking plan documentation, data quality control and auditable logs.

A simple table for checking automation results

Questions and answers

Can you calculate the return on automation without financial data?

You can prepare an approximation based on time costs, meaning minutes of work per case and the number of cases. Combined with quality and SLA, this shows whether automation genuinely relieves the process without lowering the standard.

How long do you need to measure for the result to be reliable?

Reliability grows when measurement covers at least several repeatable work cycles and excludes the learning period after changes go live. With small volumes, it is better to look at the trend across several consecutive measurements than at a single week.

Which indicators are best to start with?

Cycle time, working time, the number of corrections and timeliness against a simple SLA are usually enough. Adding an exceptions counter lets you quickly detect manual workarounds that distort the picture.

What should you do if the indicators have improved but the team reports a heavier workload?

This kind of situation is often the result of work shifting to other activities that the measures do not record, or of a rise in exceptions. It helps to split working time into categories and to count corrections and workarounds as a separate measure.

Does it make sense to compare averages?

The average is sensitive to extreme cases and can fake a deterioration or improvement depending on a few atypical cases. The median cycle time and the share within SLA describe the typical course of the process better.

Sources

• ISO 9001: Quality management systems — Requirements, International Organization for Standardization, 2015
• ITIL 4 Foundation: IT Service Management, AXELOS, 2019
• Process Mining Manifesto, IEEE Task Force on Process Mining, 2011
• Lean Thinking: Banish Waste and Create Wealth in Your Corporation, Womack and Jones, 2003

Summary

The results of automation can be assessed with simple measures if the process unit and the measurement boundaries are unambiguous. The least labour-intensive indicators are cycle time, working time, quality, timeliness and the number of exceptions. Interpretation thresholds and fixed “before/after” windows protect against confusing the effect of automation with a change in case mix or with seasonality.