Skip to main content

Troubleshooting

Storage backend: SQLite as of v0.2.6. File paths under agent_io/target/.../data.json in older snippets are stale; this page uses the current SQLite layout (agent_io/store/<workflow>.db, table target_data).

What happens when something goes wrong in your agentic workflow? This guide helps you debug and fix common errors. Let's explore the error types, their causes, and how to resolve them.

Error Types​

SchemaValidationError​

The most common error. Occurs when data fails JSON Schema validation.

text
TXT
SchemaValidationError: Input schema validation failed for tool 'add_answer_text'
at target_word_counts -> correct_answer_words: 18 is not of type 'string'
[Context: function=add_answer_text, validation_type=input,
 error_path=target_word_counts -> correct_answer_words,
 failed_value=18, schema_constraint={'type': 'string'}]

Context Fields:

FieldDescriptionExample
error_pathDot-path to failing fieldtarget_word_counts -> correct_answer_words
failed_valueThe actual value received18, 'some text', ['a', 'b']
schema_constraintExpected schema rule{'type': 'string'}
functionTool that failedadd_answer_text
validation_typeInput or output validationinput, output

ProcessingError​

Wraps lower-level errors with record context. Think of this as a breadcrumb trail—it tells you which specific record caused the problem.

text
TXT
ProcessingError: Failed to process item
[Context: source_guid=37812c37-80a2-596b-8747-8f93e7a34e7f,
         agent_name=add_answer_text]

Context Fields:

FieldDescription
source_guidUUID of the record being processed
agent_nameAction that failed

RecordContextError (Record Namespace)​

Occurs when a downstream action tries to read fields from an upstream action's output, but the expected fields are missing. This typically means the upstream action stored corrupted or empty data.

text
TXT
[RECORD NAMESPACE] 'generate_optimal_code': declared fields ['optimal_code'] not found.
Available: ['title', 'type', 'properties', 'required', 'additionalProperties']

Common causes:

CauseWhat happenedFix
Schema-echoLLM returned the JSON Schema definition instead of dataAdd expect: {repair: auto} — the validator detects schema-echo payloads and the repair loop regenerates
Empty objectLLM returned {}Same fix — empty objects are now rejected when the schema declares fields
Existing poisonBad data already in target_data from a previous runRe-run the upstream action with a repair policy, or manually clean the SQLite row

Behavior: When a single record has corrupted namespace data, that record is skipped and processing continues for the remaining records in the file. The skipped record is logged at WARNING level. This prevents one bad record from failing an entire file of valid records.

AgentActionsError​

Top-level agentic workflow failure. This wraps other errors to provide the full context of what went wrong.

text
TXT
AgentActionsError: Error generating target: Failed to process content
[Context: file_path=/path/to/data.json,
         agent_name=my_workflow]

Common Errors & Fixes​

Let's walk through the most common errors and how to fix them.

Template Variable Error​

Error:

text
TXT
Template rendering failed for agent 'write_question'

  Reference: classify.question_type
  Namespace 'classify' exists: YES
  Field 'question_type' in namespace: NO
  Available in 'classify': question_category, difficulty_level, tags

  Did you mean 'classify.question_category'?

Fix: Check that 'classify' produces the referenced field.

Cause: Template references a field that doesn't exist in the namespace. The error message shows:

  • Whether the namespace (e.g., classify) exists
  • Whether the field exists within that namespace
  • Available fields you can use instead
  • A suggestion if there's a similar field name (typo detection)

Fix options:

  1. Check the field name - Use the suggested alternative if it's a typo
  2. Verify the dependency - Ensure the action producing this field is in your dependencies list
  3. Check the upstream action's output - Verify the action produces the expected field

Source Data Structure Mismatch​

Error:

text
TXT
A staged input must be rows; found dict. Wrap it in an array: [ ... ].

Cause: The staging file's top level is a wrapper object, not the list of records staging reads. This happens when your JSON file has metadata alongside the actual records.

Wrong format:

json
JSON
{
  "exam_name": "My Exam",
  "items": [
    {"id": "1", "content": "..."},
    {"id": "2", "content": "..."}
  ]
}

Prompt expects {{ source.content }} but source is the wrapper, not individual items.

Correct format:

json
JSON
[
  {"id": "1", "content": "..."},
  {"id": "2", "content": "..."}
]

Fix options:

  1. Restructure input - Extract the array into the staging file. This is usually what you want: one record per item, so {{ source.content }} resolves.

    python
    PY
    data = json.load(open("wrapper.json"))
    records = data["items"]
    json.dump(records, open("staging/data.json", "w"))

  2. Keep the wrapper as one record - If the document really is a single record, put it in an array. It then stages as one record whose fields are exam_name and items, and the prompt loops over the array inside it:

    json
    JSON
    [
      {
        "exam_name": "My Exam",
        "items": [{"id": "1", "content": "..."}]
      }
    ]
    jinja2
    JINJA2
    {% for item in source.items %}
      {{ item.content }}
    {% endfor %}

  3. Add a preprocessing action - With the file wrapped as in option 2, a first action can expand the one record into many:

    yaml
    YAML
    - name: extract_items
      kind: tool
      impl: extract_items_from_wrapper   # returns a list: one row per item

Type Mismatch​

Error:

text
TXT
18 is not of type 'string'

Cause: Integer value where string expected.

Fix: Convert in your tool:

python
PY
data['field'] = str(data['field'])

Or fix the TypedDict:

python
PY
# If the field can be int or string
field: Union[int, str]

Array vs String Mismatch​

Error:

text
TXT
'Audit client code...' is not of type 'array'

Cause: String value where array expected.

Fix: Normalize in your tool:

python
PY
answer_text = data["upstream_action"]["answer_text"]
if isinstance(answer_text, str):
    answer_text = [answer_text]

Unexpected Field​

Error:

text
TXT
'new_field' was unexpected

Cause: Field exists in data but not declared in TypedDict.

Fix: Add field to TypedDict:

python
PY
class MyInput(TypedDict, total=False):
    existing_field: str
    new_field: str  # Add the missing field

Or use total=False to allow any fields.

Mixed-Type Dict Values​

Error:

text
TXT
'greater_than' is not of type 'integer'

Cause: Using Dict[str, int] but values include strings.

Fix: Use plain dict for mixed-type dictionaries:

python
PY
# BAD - Fails if values include strings
target_word_counts: Dict[str, int]

# GOOD - Allows any structure
target_word_counts: dict

Missing Required Field​

Error:

text
TXT
'question' is a required property

Cause: Schema requires field but tool didn't return it.

Fix: Ensure tool returns all required fields:

python
PY
@udf_tool()
def my_function(data: dict) -> dict:
    return {
        'question': data["upstream_action"]["question"],
        'processed': True
    }

Or enable repair for LLM actions:

yaml
YAML
expect:
  repair: auto
  max_iterations: 4
  on_exhausted: return_last

Debugging with Prompt Traces​

When an LLM action produces unexpected output, the fastest path to understanding "why" is inspecting the compiled prompt and raw response. Agent Actions captures both automatically.

warning

Prompt traces only exist for LLM actions. Tool (UDF) actions do not generate prompt traces. If a tool action produces empty or wrong output, debug the tool function directly — check the input data it receives, not prompt_trace.

Using the Data Explorer​

  1. Run agac docs to generate the documentation catalog
  2. Open the Data Explorer in your browser
  3. Navigate to the action's output in the Data tab
  4. Find the record with unexpected output
  5. Click the Prompt Trace accordion below the record — it shows:
    • Compiled Prompt: The exact text the LLM received (with all template variables resolved)
    • LLM Response: The raw text the LLM returned (before parsing)

What to Look For​

  • Missing context: Template variables resolved to empty strings or wrong values — check your context_scope and dependency chain
  • Ambiguous instructions: The prompt doesn't clearly constrain the output format — tighten the prompt template
  • Schema mismatch: The LLM response doesn't match the expected JSON structure — consider adding an expect: block with a repair policy
  • Model badge: Check if the model name matches what you expected — a misconfigured provider can route to the wrong model

Querying Traces Directly​

For bulk analysis across many records, you can query the storage backend directly. With the default SQLite backend:

bash
BASH
sqlite3 my_workflow/agent_io/store/my_workflow.db \
  "SELECT record_id, response_text FROM prompt_trace WHERE action_name = 'classify_issue' LIMIT 10"

See Prompt Traces reference for the full table schema and query examples.

Inspecting Failed Record Input Snapshots​

When records fail or exhaust retries, the framework captures the input record at the moment of failure. Query the disposition table to see what data was being processed:

sql
SQL
-- View failed records with their input snapshots
SELECT action_name, record_id, reason, input_snapshot
FROM record_disposition
WHERE disposition = 'failed' AND input_snapshot IS NOT NULL;

-- View exhausted records (retries exceeded)
SELECT action_name, record_id, reason, input_snapshot
FROM record_disposition
WHERE disposition = 'exhausted' AND input_snapshot IS NOT NULL;

Debugging Agentic Workflows​

When an error occurs, resist the urge to start changing code immediately. Follow this systematic approach to understand what went wrong before fixing it.

Step 1: Parse the Error​

Extract key information from the error message. Agent Actions provides structured error context—use it:

  1. What type? SchemaValidationError, ProcessingError, etc.
  2. Which field? Look at error_path
  3. What value? Look at failed_value
  4. What expected? Look at schema_constraint
  5. Which action? Look at function or agent_name

Step 2: Find the Source Record​

Use source_guid to trace the record. Each action's output is a JSON array stored in the data column of the target_data table, keyed by action_name. Use json_each(data) to scan records and json_extract to filter by source_guid:

bash
BASH
# Find every action that produced a record with this source_guid
sqlite3 agent_io/store/<workflow>.db "
  SELECT DISTINCT t.action_name
  FROM target_data t, json_each(t.data) r
  WHERE json_extract(r.value, '\$.source_guid') = '37812c37-80a2-596b-8747-8f93e7a34e7f'
"

Step 3: Check Node Outputs​

Compare data at each stage. Filter the JSON blob in the data column by source_guid to inspect the same record at each step:

bash
BASH
# Input to failing action — fetch the matching record
sqlite3 agent_io/store/<workflow>.db "
  SELECT json_extract(r.value, '\$')
  FROM target_data t, json_each(t.data) r
  WHERE t.action_name = '<failing_action>'
    AND json_extract(r.value, '\$.source_guid') = '37812c37-80a2-596b-8747-8f93e7a34e7f'
"

# Previous action's output for the same record
sqlite3 agent_io/store/<workflow>.db "
  SELECT json_extract(r.value, '\$')
  FROM target_data t, json_each(t.data) r
  WHERE t.action_name = '<previous_action>'
    AND json_extract(r.value, '\$.source_guid') = '37812c37-80a2-596b-8747-8f93e7a34e7f'
"

Step 4: Enable Debug Mode​

Get detailed tracebacks:

bash
BASH
AGENT_ACTIONS_LOG_LEVEL=DEBUG agac run -a my_workflow

Step 5: Enable Prompt Debug​

See rendered prompts for LLM actions:

yaml
YAML
- name: my_action
  prompt: $workflow.My_Prompt
  prompt_debug: true  # Logs full prompt

Data Lineage​

Here's where it gets interesting: every record in Agent Actions maintains tracking fields that let you trace its journey through the agentic workflow. This lineage data is invaluable for debugging.

source_guid​

Original content UUID. Never changes through the agentic workflow.

json
JSON
"source_guid": "37812c37-80a2-596b-8747-8f93e7a34e7f"

Use this to trace any record back to its source, regardless of how many actions it has passed through.

lineage​

Array of all node_ids visited. Grows at each node.

json
JSON
"lineage": [
  "node_0_693094fb-53d1-48d6-bdc9-781a4989d35c",
  "node_1_361c54c6-7080-4527-9a00-aaeccfd0e6ba_0",
  "node_2_e546c260-de10-4f20-8950-e09f01ea468f"
]

The _0 suffix indicates this was the first record from a flattening operation.

node_id​

Identifies the specific processing node:

text
TXT
node_0_693094fb-...          # Single output
node_1_361c54c6-..._0        # Flattened, index 0
node_1_361c54c6-..._1        # Flattened, index 1

Action Output Storage​

Action outputs are stored as JSON arrays in the target_data table of agent_io/store/<workflow>.db, keyed by (action_name, relative_path). List what each action produced:

bash
BASH
sqlite3 agent_io/store/<workflow>.db "
  SELECT action_name, relative_path, record_count
  FROM target_data
  ORDER BY action_name, relative_path
"

Typical layout for a multi-action workflow:

text
TXT
target_data
├── action_name='node_0_extract_raw_qa'      relative_path='input.json'  record_count=42
├── action_name='node_1_flatten_questions'   relative_path='input.json'  record_count=128
├── action_name='node_2_classify_type'       relative_path='input.json'  record_count=128
└── action_name='final_workflow_output'      relative_path='input.json'  record_count=128

TypedDict Best Practices​

Use total=False for Optional Fields​

python
PY
class MyInput(TypedDict, total=False):
    required_field: str  # Still works, just not enforced
    optional_field: str

Use Plain dict for Mixed Types​

python
PY
# When dict values can be int, string, or other types
metadata: dict  # Not Dict[str, int]

Use Union for Polymorphic Fields​

python
PY
# When a field can be different types
answer_text: Union[str, List[str]]

Access Namespaced Fields​

python
PY
@udf_tool()
def safe_function(data: dict) -> dict:
    value = data["upstream_action"]["field"]
    items = data["upstream_action"]["items"]
    return {'result': value}

The repair loop​

What happens when an LLM returns invalid JSON, or output that breaks a rule you declared? Rather than failing immediately, Agent Actions can regenerate with feedback about what went wrong. This is the repair loop, and it is driven by an action's expect: block.

Configuration​

Repair requires explicit configuration:

yaml
YAML
expect:
  repair: auto              # how a rule failure is regenerated
  structural: retry         # how a schema failure is regenerated
  max_iterations: 3         # total generations, counting the first
  on_exhausted: return_last # return_last | fail | raise

To disable: omit the expect: block.

Configuration Options​

OptionDescription
repairnone (observe only), retry (re-send the original prompt), or auto (send the failure detail and hint)
structuralSame choices, for a response the schema rejects (default: retry)
max_iterationsTotal generations, counting the first (1-10, default: 3)
on_exhaustedBehavior when iterations run out: return_last, fail, raise

When to Use​

Consider what your agentic workflow needs:

  • Simple schemas — Low max_iterations (2-3)
  • Complex schemas — Higher max_iterations (4-5)
  • Critical outputs — More iterations, on_exhausted: raise
warning

Repair adds latency and token cost. For high-volume agentic workflows, consider fixing schema issues at the source rather than relying on regeneration.

Log Analysis​

Log Location​

text
TXT
agent_io/logs/events.json

Search Patterns​

Events are JSON Lines. Structured fields live under the .data object (e.g. .data.action_name); message and level are top-level.

bash
BASH
# Find all schema validation errors
jq -c 'select(.message | test("SchemaValidationError"))' agent_io/logs/events.json

# Find events for a specific action
jq -c 'select(.data.action_name == "add_answer_text")' agent_io/logs/events.json

# Find events that mention a specific field
jq -c 'select(.message | test("answer_text"))' agent_io/logs/events.json

Execution History​

Check artefact/runs.json for execution metrics:

bash
BASH
# Count failed runs
grep '"status": "FAILED"' artefact/runs.json | wc -l

# Find recent failures
grep -A5 '"status": "FAILED"' artefact/runs.json | tail -20

Quick Reference​

These commands and patterns are your debugging toolkit.

Debug Commands​

bash
BASH
# Debug mode (full tracebacks)
AGENT_ACTIONS_LOG_LEVEL=DEBUG agac run -a workflow

Common Fixes Cheatsheet​

Error PatternQuick Fix
X is not of type 'string'str(value) or Union[int, str]
X is not of type 'array'[value] if isinstance(value, str) else value
X was unexpectedAdd field to TypedDict
X is a required propertyReturn field from tool or use expect: {repair: auto}
Dict[str, int] failsUse plain dict

Debug Checklist​

When debugging agentic workflow errors, work through this checklist:

  1. Read full error message, note error_path and failed_value
  2. Find record by source_guid in node outputs
  3. Compare data before/after failing action
  4. Check TypedDict matches actual data shape
  5. Enable prompt_debug: true for LLM actions
  6. Run with AGENT_ACTIONS_LOG_LEVEL=DEBUG for tracebacks
  7. Consider expect: {repair: auto, max_iterations: 3} for LLM schema failures

Most errors fall into one of two categories: schema mismatches (the data structure doesn't match expectations) or missing fields (required data wasn't provided). The checklist above helps you identify which category you're dealing with.

Dependency Patterns​

Understanding Fan-in vs Parallel​

When using multiple dependencies, understanding the pattern detection is crucial:

yaml
YAML
# Pattern 1: Parallel Branches (MERGE)
dependencies: [classify_1, classify_2, classify_3]
# Same base name "classify" → outputs are merged
# Execution count: N (from merged outputs)

# Pattern 2: Fan-in (PRIMARY + CONTEXT)
dependencies: [extract, enrich, validate]
# Different actions → first is primary, others via context
# Execution count: N (from extract only)

To aggregate several records into one — a roll-up, a deduplication, a count across the file — give the consuming action granularity: File. A kind: tool action at File granularity is handed every record of a file in one call, so it can group them by whichever field it likes and return as many rows as the grouping produces. examples/map_reduce_fanin_check shows the shape.

Missing Context Data in Fan-in​

Symptom: Action only sees data from first dependency, not others.

Cause: Fan-in pattern requires context sources to be in context_scope:

yaml
YAML
# WRONG - enrich and validate data not accessible
- name: generate_report
  dependencies: [extract, enrich, validate]

# CORRECT - all dependencies in context_scope
- name: generate_report
  dependencies: [extract, enrich, validate]
  context_scope:
    observe:
      - extract.*
      - enrich.*     # Loaded from record namespaces
      - validate.*   # Loaded from record namespaces

Unexpected Execution Count​

Symptom: Action executes more/fewer times than expected.

Debug:

  1. Check if dependencies are parallel branches (same base name) or different actions
  2. For fan-in: first dependency determines execution count
  3. For parallel: merged outputs determine execution count
  4. For granularity: File: one execution per input file, whatever its record count