Skip to main content

Batch Recovery

When you submit 1,000 records to a batch API, what happens when only 985 come back? Or when 40 of those 985 have outputs that fail validation? Batch recovery handles both problems automatically through a two-phase process.

The Two Phases​

Batch recovery runs after the initial batch completes. It addresses two distinct failure modes in sequence:

Phase 1 (Retry) recovers records the provider did not answer — ones it dropped outright (network errors, timeouts, silent failures) and ones it returned with a per-record error and no content. The same request is resubmitted unchanged.

Phase 2 (Repair) fixes records where the LLM responded but produced output that fails the action's expect: suite — schema violations, failed rules. The prompt is modified with the failure feedback before resubmitting.

Phase 1: Retry Missing Records​

After retrieving batch results, the system compares the expected record IDs against the ids the provider actually answered. A record returned with an error, or with null content, was not answered. When the action has retry on, any gap triggers the retry loop. Retry is off until a retry: block turns it on (see Configuration); without it, a record the provider did not return is failed at once (see A record nothing sends again).

How It Works​

  1. Collect all custom_id values from the context map (expected)
  2. Collect the custom_id values of results the provider answered successfully. A result returned with an error carries the record's real custom_id but no content, so it does not count as answered
  3. Compute the difference — these are the missing records
  4. For each retry attempt (up to max_attempts):
    • Build new records from the context map for missing IDs
    • Submit as a new batch
    • Poll until complete
    • Merge the results back, replacing any earlier answer for the same record
    • Update the missing set
  5. If records remain missing after all attempts, mark them with exhaustion metadata

A retry is prepared again from each record's stored context, with the action's dependencies and version, so its prompt reads the same upstream answers and its guard decides as it did for the first request. The run's workflow.* values are not handed to it yet. A record whose prompt reads them is not retried: nothing is sent for it, and it is failed as below. A guard that reads them cannot be evaluated on a retry, and the collect run fails.

A record nothing sends again​

A record the provider left out of its results that no retry will send again is failed, with the reason batch_not_returned. That is a record of an action with retry off, or one whose retry batch did not go out, because its prompt could not be rendered (as for a prompt that reads workflow.*) or the provider refused it. The action completes with failures (completed_with_failures), the actions that read it skip the record, and agac retry sends it again. Once its answer is collected, the actions that read it answer it too.

Configuration​

Retry is configured at the action or defaults level. It is off for an action when neither the action nor defaults: gives a retry: block:

yaml
YAML
defaults:
  retry:
    enabled: true
    max_attempts: 3        # Retry up to 3 times
    on_exhausted: return_last  # or "raise"

Per-Record Tracking​

The system tracks failure counts per record, not globally. If record A succeeds on retry attempt 1 but record B needs all 3 attempts, each gets its own count:

json
JSON
{
  "_recovery": {
    "retry": {
      "attempts": 2,
      "failures": 1,
      "succeeded": true,
      "reason": "missing",
      "timestamp": "2024-06-15T10:30:45Z"
    }
  }
}

Exhaustion​

When a record exhausts all retry attempts:

  • on_exhausted: return_last — record is marked with exhaustion metadata, workflow continues without it
  • on_exhausted: raise — the output file is written first, then the run stops. Batch writes once at the end, so halting mid-conversion would discard every record that had already converted cleanly; the halt is carried to the caller and raised after the write

Both policies apply to every record that spends its attempts, but the two exhausted shapes do not land in the same disposition. The split is whether a result row came back at all. A record the provider never returned has nothing to report, so it becomes an EXHAUSTED tombstone dispositioned exhausted_after_N_attempts. A record that came back — with an error, or with null content and no message at all — is dispositioned FAILED, carrying whatever the provider said (or a generic failure when it said nothing) plus the same exhausted retry metadata under _recovery.retry. The provider's message is the more useful signal of the two, so it is preserved rather than replaced by an empty tombstone. Counting retry exhaustion means counting both — filter on _recovery.retry.succeeded == false, not on the disposition alone.

Phase 2: Validate and Repair​

After Phase 1 ensures all recoverable records are present, Phase 2 checks whether the outputs are actually valid. This runs when the action carries an expect: block with a repair policy.

How It Works​

A repair round is a whole provider batch, so it defers rather than blocking:

  1. Run the action's expectation suite over every returned record, including the structural gate that checks schema conformance
  2. Records that pass graduate into a pool and are never evaluated again — a later round does not re-spend the judge budget on them
  3. Records that fail are collected into a repair batch, each carrying its own regeneration prompt
  4. The batch is submitted and the run defers; the next pass resumes into the same loop
  5. When max_iterations rounds are spent, on_exhausted decides what ships

A record the provider never returns is reconstructed as a failure rather than left to become a silent passthrough, and a record with no content at all failed at the provider rather than at its expectations, so it keeps its own error.

Recovery Metadata​

Every record that goes through recovery gets a _recovery field in its output. This field is automatically excluded from content extraction — downstream actions never see it, but it's available for auditing and debugging.

Structure​

json
JSON
{
  "_recovery": {
    "retry": {
      "attempts": 3,
      "failures": 2,
      "succeeded": true,
      "reason": "missing",
      "timestamp": "2024-06-15T10:30:45Z"
    },
    "expectations": {
      "attempts": 2,
      "failed": []
    }
  }
}

Fields​

Retry metadata (present when transport-layer recovery occurred):

FieldTypeDescription
attemptsintegerTotal attempts made (including initial)
failuresintegerNumber of failed attempts before success
succeededbooleanWhether retry ultimately succeeded
reasonstringWhy retry was needed: missing, network_error, rate_limit, timeout
timestampstringISO 8601 timestamp of the recovery

Expectations metadata (_recovery.expectations, present when the repair loop exhausted):

FieldTypeDescription
attemptsintegerIterations spent, counting the first generation
failedlistIds of the expectations still failing when iterations ran out

Evaluation metadata (_recovery.evaluation, present when a batch record graduated through the loop):

FieldTypeDescription
passedbooleanWhether the record graduated
strategy_namestringWhich evaluation strategy judged it

Serialization​

Recovery metadata survives serialization. When batch results are saved to disk and reloaded (e.g., for agac batch retrieve), both retry and expectations metadata are preserved through the round-trip.

Example: Full Recovery Flow​

Submit 100 records. 95 come back. 10 of the 95 fail validation.

text
TXT
Submit 100 records
├── Phase 1: Retry
│   ├── 5 missing → resubmit
│   ├── Attempt 1: 4 recovered, 1 still missing
│   ├── Attempt 2: 1 recovered
│   └── Result: 100 records (all recovered)
│
└── Phase 2: Repair
    ├── Validate 100 results → 10 fail
    ├── Attempt 1: resubmit 10 with feedback → 8 pass
    ├── Attempt 2: resubmit 2 with feedback → 1 passes
    └── Result: 99 passed, 1 exhausted (on_exhausted: return_last)

Final output:
  94 records — clean (no recovery needed)
   5 records — _recovery.retry present
   1 record  — _recovery.retry + _recovery.expectations (both phases)

See Also​