Troubleshooting
OpenFlows is a distributed system: a Coder control plane, a Redis state store, a Coder AI
Gateway, an orchestrator process running in a NEXUS workspace, and a fleet of ephemeral
worker workspaces. When something fails, the failure can appear anywhere in that chain.
This guide walks through common failures, diagnostics, and how to read the artifacts the
system leaves behind, especially STATUS.json.
The first rule of troubleshooting is to check the controller log. The NEXUS workspace
writes to /tmp/openflows-controller.log, and that log contains structured
events from the controller and every worker. The second rule is to check Redis, which is
the source of truth for ticket and worker state. The third rule is to read
STATUS.json when a worker writes one; it is the worker's own statement of
what went wrong.
Common Failures
| Failure | Symptom / Cause |
|---|---|
| Coder unreachable | NEXUS cannot provision or destroy workspaces. Check CODER_URL and CODER_NEXUS_TOKEN. |
| GitHub external auth expired | Workers lose git push/pull access. Re-authenticate in Coder admin or user settings. |
| Worker timeout | A FORGE or SENTINEL workspace exceeded its step timeout. Check workspace logs and LLM gateway health. |
| SENTINEL rejection loop | SENTINEL rejects every segment repeatedly. Escalate to AwaitingHuman and review the plan. |
| Merge conflict | PR is no longer mergeable. VESSEL retries; if unresolved, escalate to AwaitingHuman. |
| Redis connection lost | All state operations fail. Controller enters degraded mode and retries. |
| AI Gateway 429 | Rate limit hit. Back off and scale gateway capacity if sustained. |
| STATUS.json blocked | Agent wrote a blocked status. Read the blocker object for the exact question. |
Diagnostics
The openflows doctor command runs a battery of health checks against the
orchestrator, Coder, Redis, and the AI Gateway. Run it from the NEXUS workspace before
digging into manual logs.
openflows doctor --tenant my-team
The doctor command checks:
- Redis connectivity and keyspace prefix integrity.
- Coder API reachability and token validity.
- AI Gateway reachability and model availability.
- GitHub external auth status for the tenant.
- Stale workers and orphaned tickets.
- Disk space and workspace quota on the NEXUS workspace.
If doctor reports a failure, it prints a remediation hint. If it reports all green but the system still misbehaves, the issue is likely in the LLM-driven agent logic, not the infrastructure.
Continue reading