How to link AI retries to business outcomes

A retry and a duplicate usage event need opposite treatment. A retry consumes more work and may add cost. A duplicated event reports the same work twice. Counting both as retries inflates spend; dropping both hides it.

Start with three identifiers:

One attempt can contain several model calls. In that case, keep a call ID as well, and use a distinct cost event ID for each charge. A retry of the telemetry upload preserves its event ID. A retry of the model call gets a new call ID and new charge event, even when the prompt is unchanged.

Keep the identity across queues

Create the work item ID where your application first accepts the task. Pass it through queued jobs and worker handoffs. A queue delivery ID is not necessarily the business task ID: redelivery must not create a second ticket in the outcome count.

When a worker starts a new execution, create an attempt ID. Attach the work item and attempt IDs to the cost records produced by that execution. If the mapping is missing, keep the charge in an unassigned bucket. Do not guess which ticket it belongs to just because the timestamps are close.

Deduplicate records, not work

Consider this deliberately small, hypothetical ledger:

The second e1 is repeated delivery and counts once. The e2 charge is a different attempt and stays. Total recorded cost is $0.10, not $0.13 or $0.08.

If ticket-1 succeeds and ticket-2 fails after the declared outcome window, cost per successful ticket is $0.10. The failed ticket's cost stays in the numerator. These numbers are an illustration, not a valuemaxx benchmark or a customer result.

If two records share an event ID but disagree on amount or task, stop that rollup for reconciliation. Choosing whichever arrived last makes the total depend on delivery order. For legitimate billing corrections, use an explicit revision or adjustment convention rather than silently overwriting history.

Record outcomes separately

A model returning a response does not establish that a ticket was resolved. Take the outcome from the business system that owns the task. Keep the outcome rule and its version alongside the observation window.

For example, a support team could require resolution within three days and no reopen for seven days afterward. That requires up to ten days before the cohort can be compared fairly. The appropriate window depends on the workflow; ten days is not a default recommendation for every product.

A missing outcome is unknown. If collection failed, do not record failure for every missing task. Show the cohort size, observed successes, observed failures and unknown outcomes separately. Delay the final comparison when coverage is incomplete.

Try the local example

The accompanying Python example uses only the standard library. It deduplicates identical charge events, rejects conflicting duplicates, and withholds cost per success while an outcome is unknown or there are no successes.

Download the Python example. Save it, then run python3 workflow_cost_ledger.py.

It assumes that the supplied charge stream is complete, amounts have already been normalized to USD, and True/False outcomes have matured. Those assumptions must be verified in a real integration. It is not a billing reconciler or a valuemaxx SDK example. Zero recorded charges do not prove zero actual spend.

Run it with Python 3.10 or newer. The hypothetical input above returns $0.10 total cost and $0.10 per successful item. Change ticket-2's outcome to None and the final ratio becomes unavailable instead of treating the missing outcome as a failure.

Before comparing versions

Use the same cost boundary, work-item cohort definition and outcome rule for both versions. Keep estimated prices separate from billed charges, and report unassigned costs and collection gaps. A cheaper model call is only one part of a cheaper completed workflow.

For the measurement formula and comparison example, read How to measure AI cost per successful workflow.