State Machine & Recovery

OpenFlows is a state-driven system. The SharedStore is the single source of truth, and every agent action is a state transition. NEXUS is the only component that is allowed to reconcile the global state: on every poll cycle it runs reconcile(), compares the SharedStore against the real world (GitHub, Coder, worker heartbeats), and fixes any inconsistencies it finds.

This page explains that reconciliation engine in detail: the SharedStore state machine, the anomalies NEXUS detects, the retry semantics for each failure mode, and the AwaitingHuman escalation path that prevents infinite loops when automatic recovery is impossible.

SharedStore as the State Machine

The SharedStore is not just a cache. It is the canonical state machine of the OpenFlows pipeline. Every ticket, every worker, every pending PR, and every recovery action is stored in typed Redis keys. The in-memory backend used during development is a behavior-preserving mirror of the Redis implementation, so tests and production share the same contract semantics.

These keys are the contract surface between agents. An agent never talks to another agent directly; it reads and writes these keys. NEXUS is the coordinator because it is the only role that reads all of them. FORGE only writes to its own ticket and reads the SENTINEL review state for that ticket. SENTINEL only writes review files. VESSEL only reads pending_prs and writes ticket_merged events. LORE only listens for merge events.

Ticket State Machine

A ticket is the OpenFlows representation of a GitHub issue. It moves through a well-defined state machine, and each state tells NEXUS what to do next. The terminal states are merged, awaiting_human, and closed. Every other state is recoverable.

What NEXUS reconcile() Actually Does

reconcile() is the heartbeat of the system. It runs on every NEXUS poll cycle, which defaults to 10 seconds. The cycle is deliberately simple: fetch state, compare state to reality, emit recovery actions, advance tickets, and sleep. The simplicity makes it easy to reason about and easy to debug from logs.

The reconcile loop follows these steps in order:

1. Load global state - read all SharedStore keys into a consistent snapshot.
2. Sync GitHub issues - discover new, closed, or renamed issues; update tickets.
3. Inspect worker slots - compare each slot's working state against the Coder API to detect stale workspaces.
4. Inspect pending PRs - query GitHub for merge status, conflicts, and CI results.
5. Detect anomalies - classify each inconsistency as one of the known anomaly types.
6. Emit recovery actions - write flow_recovery records and dispatch follow-up actions.
7. Assign idle workers - match ready tickets to idle workers and provision workspaces.
8. Write event_ring - record the cycle's actions for observability.

// Pseudocode of the reconcile loop
async fn reconcile(&self) -> Result<()> {
    let state = self.shared_store.snapshot().await?;
    let github_issues = self.github.poll_open_issues().await?;
    let coder_workspaces = self.coder.list_workspaces().await?;

    for ticket in state.tickets.values() {
        match ticket.status {
            TicketStatus::Open => self.try_assign(ticket, &state).await?,
            TicketStatus::Assigned | TicketStatus::InProgress => {
                self.check_worker_health(ticket, &state, &coder_workspaces).await?;
            }
            TicketStatus::PrOpened => self.check_pr_status(ticket, &state).await?,
            TicketStatus::CompletedWithoutPr => self.recover_pr_creation(ticket).await?,
            TicketStatus::AwaitingHuman => self.check_human_resolution(ticket).await?,
            _ => {}
        }
    }

    self.event_ring.push(Event::ReconcileComplete).await?;
    Ok(())
}

Anomaly Detection

NEXUS recognizes a closed set of anomalies. Each anomaly maps to a recovery action, and the recovery action is idempotent. If an anomaly persists across multiple cycles, NEXUS increments a retry counter and eventually escalates. This prevents the system from flapping on transient inconsistencies.

Orphaned Tickets

An orphaned ticket is a ticket that is supposed to be in progress but has no healthy worker attached. This happens when a workspace crashes, a worker process panics, or a Coder auto-stop removes the workspace before NEXUS can update the slot. The ticket is "orphaned" because its state machine says it is assigned, but the worker slot is gone.

NEXUS handles orphans by checking whether the ticket is recoverable. If the ticket has a recent commit on its branch and a clear next state, NEXUS provisions a new workspace and re-assigns the ticket to the same or a different worker slot. If the ticket state is ambiguous, NEXUS writes the reason to flow_recovery and may escalate to AwaitingHuman.

Stale Workers

A stale worker is the opposite of an orphaned ticket: the worker slot is still marked working, but the Coder workspace it references is stopped or deleted. Stale workers are dangerous because they consume a worker slot without doing work. If every slot becomes stale, the pipeline stops assigning new tickets.

NEXUS resolves stale workers by first querying the Coder API for the workspace status. If the workspace is stopped, NEXUS can restart it and resume the ticket. If the workspace is deleted, NEXUS clears the worker slot and either reassigns the ticket or escalates it. The stale-worker check is also the mechanism that enforces workspace TTL: a stopped workspace after TTL expiry is treated as stale.

Unmerged PRs

A PR in pending_prs that remains open after a timeout is classified as unmerged. Timeouts can be caused by long-running CI, failing checks, or a GitHub merge conflict that VESSEL has not yet resolved. NEXUS delegates unmerged PRs to VESSEL, which polls the PR, checks mergeable, and either merges it, resolves the conflict, or routes the ticket back to FORGE for rework.

The unmerged-PR check is the bridge between the SharedStore and GitHub reality. If a PR is merged by a human while OpenFlows is offline, NEXUS detects the merged state on the next reconcile and moves the ticket to merged without requiring a new VESSEL action.

completed_without_pr

completed_without_pr is a specific state that means the worker believes the ticket is done but no PR has been recorded. This can happen when a FORGE workspace is destroyed right after the final commit but before the GitHub MCP PR creation call completes. It can also happen when a human closes the issue while the ticket is in progress.

NEXUS resolves this by looking at the branch. If the branch has commits that are not on the base branch, NEXUS instructs FORGE to open the PR. If the branch is empty, NEXUS checks whether the issue is still relevant and either closes the ticket or escalates. The completed_without_pr state is the safety net that prevents "done but not delivered" tickets from silently disappearing.

Retry Semantics

OpenFlows has three separate retry layers, each with its own budget and purpose. They are designed to avoid both premature escalation and infinite loops. A failure that is not transient exhausts its budget and then escalates.

The retry counters are stored in the flow_recovery key. NEXUS increments them atomically and reads them back on the next cycle to decide whether to retry again or escalate. Because the SharedStore is the source of truth, a NEXUS crash in the middle of a retry does not lose the retry count; the next NEXUS process resumes from the same counter value.

Flow Recovery Record

The flow_recovery key holds a record per ticket. It is the scratchpad for NEXUS's healing decisions. The record includes the anomaly type, the retry count, the last action taken, the expected next state, and a human-readable reason. Operators can inspect these records in the TUI to understand why a ticket is stuck.

{
  "T-001": {
    "anomaly": "StaleWorker",
    "retry_count": 1,
    "max_retries": 3,
    "last_action": "workspace_restart",
    "expected_next_state": "InProgress",
    "reason": "Coder workspace was auto-stopped after TTL; restarting to resume segment 3",
    "escalate_at": "2026-07-20T17:00:00Z"
  }
}

AwaitingHuman Escalation

Not every failure is recoverable. When a ticket has exhausted all retry budgets, when the specification is ambiguous, or when a security concern is raised, NEXUS moves the ticket to awaiting_human. This is a terminal state for automation: no further agent actions are taken until a human operator intervenes.

The escalation is not silent. NEXUS writes a detailed reason to the SharedStore, sends a notification if configured, and marks the worker slot as suspended. The worker's workspace is torn down, because a suspended worker should not hold resources. When the human resolves the blocker, they update the ticket state in the TUI or by pushing a state update, and NEXUS resumes the flow from the appropriate phase.

The AwaitingHuman path is the final safety valve. It prevents OpenFlows from making speculative decisions about ambiguous requirements, security exceptions, or external dependencies. The design philosophy is clear: recover automatically when possible, stop cleanly when it is not, and always tell the human why.

Observability and Operations

The reconcile loop is fully observable. Every cycle writes to event_ring, and every recovery action updates flow_recovery. The TUI displays the current state of every ticket, the health of every worker slot, and the count of pending anomalies. Operators can drill into a ticket to see the full history of state transitions, retry counts, and escalation reasons.

For long-running deployments, the reconcile metrics are also exposed for external monitoring. The most important metrics are: tickets assigned per cycle, reconcile duration, anomaly count, retry count per anomaly type, and tickets in awaiting_human. A sudden spike in anomalies is usually the first sign of an upstream dependency issue (Coder, GitHub, or the LLM provider).