Investigate recurring agent failures
Scan trace patterns, review defects and insights, then turn a recurring failure into a guarded eval.
Use afk improve to turn recurring patterns in witness traces into failure cards, proposals, eval cases, and deterministic eval runs. For a quick overview before triaging individual traces, generate a local insights report or list defects previously identified as pre-existing.
Find a pattern
Get an overview, then scan recent sessions:
afk insights --days 30 --no-open
afk defects --days 90 --top 10
afk improve scan --since 7dafk insights writes a self-contained HTML analytics report to ~/.afk/cache/insights.html by default. Omit --no-open to open it in your browser; use --output for a different path. It aggregates local telemetry and recommendations, not live subscription headroom. afk defects groups recurring defect loci by repo and category from the pre-existing defect ledger; --json emits structured clusters. It reads up to the latest 1 MB of ledger records and warns if older entries were omitted. afk improve scan is dry-run by default.
Save cards, then copy a real slug from the list:
afk improve scan --since 7d --write
afk improve cards list --status openRead afk improve cards show with that slug to inspect evidence before changing status or generating a proposal.
Work a card through the pipeline
| Stage | Command | Result |
|---|---|---|
| Scan | afk improve scan --since 24h --write | Runs registered deterministic detectors and persists cards. Without --write, prints a preview only. |
| Inspect | afk improve cards list --regressed | Finds deferred or resolved cards that fired again after their latest triage note, without changing status. |
| Triage | afk improve cards triage card-slug --note "Reproduced in CI" --status deferred | Appends a note and changes status while retaining prior evidence and notes. |
| Propose | afk improve propose card-slug | Writes a deterministic starter proposal for human refinement. --no-write previews it. |
| Evaluate | afk improve eval-gen card-slug | Slices a byte-identical fixture from the card's witness evidence and writes an eval-case contract. --no-write previews it. |
| Run | afk improve eval-run card-slug | Runs the latest eval case's smallest available deterministic validation contract against the current codebase. |
Replace card-slug with the actual value shown by cards list. afk improve proposals list and afk improve eval-cases list let you inspect generated artifacts; both groups have a show command. afk improve approve <slug> marks a proposal or eval case approved and records who approved it. Approval is an artifact state change, not an automatic patch or deployment.
The registered scan patterns include repeated-tool-use, closure-anomaly, and tool-failure-density by default. subagent-block and subagent-read-denial are opt-in because legitimate work can trigger them; select one with --only subagent-read-denial or use --include-disabled. The read-denial pattern checks repeated forked-child path approval refusals. scan --since all includes all retained traces. See the witness guide for inspecting individual events.
eval-gen --proposal records a validated proposal back-reference without changing the proposal. --evidence-row <index> selects a card's 0-based evidence row, defaulting to the newest. The command refuses a duplicate eval case for that row unless you explicitly pass --force. Use --no-write to preview proposal, eval-case, or eval-run output without saving it; --json is available for scripting. The CLI reference lists the remaining flags.
A generated eval case is not a full replay of the agent's original run. eval-run rechecks the fixture checksum and executes a registered deterministic guardrail contract; it does not feed the fixture through a model or detector. A passing run exits 0, fail or error exits 1, and an unsupported pattern exits 3 because no card-specific behavioural proof was obtained (no contract, no replay handler, or a replay handler that ran inconclusively). Do not treat unsupported as a passing CI gate. --no-write runs without persisting a result. The pipeline never invokes an LLM or applies code changes.
Check previously reported defects
The pre-existing defect ledger records structured loci, not transcript prose. At session end, a detector looks for assistant statements that identify a pre-existing defect or a deferred item with a concrete file, test, or gate. Records include the session, repo, coarse category, signal, and locus under ~/.afk/agent-framework/preexisting-ledger.jsonl. Seeing a locus there does not prove it is still broken: verify current code or tests before relying on an old report.
afk defects --days 30 --top 20
afk defects --jsonThe default window is 90 days and the default limit is ten clusters. Set AFK_PREEXISTING_LEDGER_DISABLE=1 if you do not want session-end collection. A separate optional historical backfill exists in the source project (pnpm backfill:preexisting), but normal afk defects only reads the running ledger; it does not scan all old transcripts itself. For full-text conversation search, use observability.
Test a proposed behavior change
If you already have a candidate rule, model, or memory change, what-if compares an isolated candidate with your current setup. afk whatif --append "Ask before editing" --predict "agent asks before editing" supplies your hypothesis directly and skips the analyst prediction call; add --verify only when you intend to pay for sandbox episodes. This is a separate experimental workflow, not a stage in afk improve. There is no afk daemon --predict flag.