top of page

Measure the Rework, Not Just the Automation Count

Writer: Branden Bell
Branden Bell
6 days ago
4 min read
Count the rework. A terracotta correction strip crosses an olive line on a cream working copy.

The AI drafts a client update in a minute. The coordinator spends the next eight checking the job record, fixing the wrong date and putting a missing decision back in. The draft counter counts one success. The coordinator still has work to do.


Measure rework as the share of eligible outputs that need a material correction before acceptance. Count each output once, then add the time spent drafting, reviewing and correcting it. At a reporting cutoff, divide flagged outputs by all eligible outputs, including pending ones, and call it the observed rate so far. Show the pending count beside it. Track errors discovered after release separately.


Define what counts as a second pass


Write down the criteria for a material correction before the trial. For a client update, that could be a wrong commitment date, an unsupported status or a missing decision the customer needs to make. Fixing an optional turn of phrase can still take time. Put that time under review without treating phrasing as a material error.


ASQ’s cost-of-quality guidance treats rework as a cost of failures found before delivery. The NIST AI Risk Management Framework Measure playbook recommends realistic tests, documented methods and attention to human work alongside AI. Use those ideas to choose records your team can inspect and to give both versions of the workflow the same acceptance rule.


A faster draft with a larger bill


Imagine a small service team running two fictional batches of 50 client updates of comparable complexity. The same reviewer uses the same acceptance rules. Both batches have 50 accepted updates and no pending cases at the cutoff. The figures below are sample data.


Manual batch. Drafting took 360 active staff minutes, normal review took 75, and correction took another 36 across six updates. The team spent 471 minutes in total, or 9.42 minutes per accepted update. The rework rate was 6 ÷ 50, or 12%.


AI-assisted batch. Drafting took 210 active staff minutes, normal review took 125, and correction took another 150 across 15 updates. The total was 485 minutes, or 9.70 per accepted update. The rework rate was 15 ÷ 50, or 30%.


The team holds expansion. It saved 150 drafting minutes, spent 164 additional minutes on review and correction, and finished 14 minutes behind. On inspecting the cases, the team groups errors by type. Suppose the most common problem is an old milestone date. The next trial tests one change: give the assistant the current approved milestone and require a source beside every commitment date.


Keep one measurable record per output


  • Eligible output. Give the job an ID, a type and a complexity level. Count it in the trial even if it stays open.

  • First-pass result. Record accepted, needs correction, or pending at the first review. Keep a separate final status, so an output that was corrected and accepted stays flagged. Count one material correction flag per output.

  • Staff time. Keep draft, review and correction minutes in separate fields with no overlap. Record waiting time separately if speed to delivery matters.

  • Correction reason. Pick one primary reason and link to the missing or conflicting record. Record any after-release error separately.

  • Decision. Pair the rework rate with total staff minutes, accepted volume, and remaining open cases. Investigate high-risk errors before expanding the pilot.


Put one row in the tool your team already reviews. The definition of done guide helps the reviewer name the evidence an output needs. If the same missing input causes repeat corrections, the form-question guide gives you a way to test an upstream fix. For a wider view of timeliness and quality together, use the delivery speed scorecard.


Choose the check before you run it


Use a small batch of redacted examples with a documented mix of straightforward and difficult cases. Have the reviewer score them using the same rule and time categories. Log pending outputs and their minutes to date. Use the same follow-up period for both groups. If materially different numbers of cases are still unresolved at the cutoff, defer the final rework-rate and time-per-accepted comparisons until the batches are comparable.


Decide in advance what would let you expand. In the fictional team, a reasonable next step is another limited batch if the rate falls and reviewers find no unsupported commitments. If a verifier misses a wrong commitment in the trial, pause and repair the check before trying a larger batch.


Questions about measuring AI rework


Is time spent reviewing a draft rework?


Ordinary review applies to every output, so track it as review. Record the extra time to fix and recheck a material problem as correction. Keep both in total staff time.


What if the same update needs three corrections?


Count one affected output in the rework rate and all correction minutes in the workload. Keep the individual reasons if they help you find a pattern.


What if I only have ten examples?


Use the results to look for repeated failure types. Record the actual counts and test more cases before judging whether a small change in rate will hold up.


Start with a batch your team knows well


Bring a redacted output, how your team accepts it and where people spend time fixing it. Ask Bell Consulting Solutions to review the workflow, or start with the AI process optimization guide.

 
 
 

Comments


bottom of page