Operations & Monitoring

OpenFlows is an orchestration layer that runs on top of Coder. Operating it means monitoring the controller, the Redis state machine, the Coder AI Gateway, and the fleet of ephemeral worker workspaces. This guide covers production runbooks, TUI usage, log locations, Redis inspection, monitoring NEXUS reconcile, and scaling workers. It is written for operators who need to keep a multi-tenant deployment healthy.

The controller runs inside a per-tenant Coder workspace named openflows-nexus-{tenant}. That workspace is the only long-lived component. All other workspaces are ephemeral and are created and destroyed as tickets move through the pipeline. Operators should treat the NEXUS workspace as the source of truth for a tenant's health.

Production Runbooks

The following runbooks are the most common operational procedures. They assume you have shell access to the NEXUS workspace and the Redis instance, and that you have installed the openflows CLI.

Starting the Controller

Ensure the Coder deployment is reachable, the NEXUS workspace is running, and the Redis URL is correct. Then start the controller process inside the NEXUS workspace.

coder ssh openflows-nexus-my-team

export REDIS_URL=redis://redis.internal:6379
export CODER_URL=https://coder.example.com
export CODER_NEXUS_TOKEN=...

openflows controller start --tenant my-team

Stopping the Controller

Isolating the tenant first stops new work from being assigned. Active workers are allowed to finish their current segment, but they will not receive new segments. After a graceful drain, stop the NEXUS workspace.

openflows tenant isolate my-team

# Wait for active workers to reach a safe point
tail -f /tmp/openflows-controller.log

coder stop openflows-nexus-my-team

Updating Binaries

The ./update-binaries.sh script pulls the latest openflows release, replaces the CLI binary, and restarts the controller. Run it inside the NEXUS workspace.

coder ssh openflows-nexus-my-team
./update-binaries.sh

TUI Usage

The OpenFlows TUI is a ratatui-based dashboard that runs on the NEXUS workspace. It shows live tickets, worker slots, events, and logs. It also provides one-key diagnostics and tenant isolation. Operators should keep a TUI session open during incident response.

openflows tui --tenant my-team

Log Locations

The controller writes a single log file on the NEXUS workspace. Worker logs are also forwarded to the controller, which writes them to the same file with a worker tag. This keeps all operational telemetry in one place.

# Controller log on the NEXUS workspace
tail -f /tmp/openflows-controller.log

# Recent error lines
rg ERROR /tmp/openflows-controller.log | tail -50

# Recent worker-specific logs
rg 'forge-1' /tmp/openflows-controller.log | tail -100

The log format is structured JSON with a human-readable prefix. Each line contains a timestamp, tenant, worker ID, level, and message. The structured fields are stable and can be shipped to a central log aggregator with a simple forwarder.

{
  "ts": "2026-07-20T15:43:21Z",
  "tenant": "my-team",
  "worker": "forge-1",
  "level": "INFO",
  "msg": "segment review APPROVED",
  "ticket": "ISSUE-123",
  "segment": 2
}

Redis Inspection

Redis is the production SharedStore backend. Every tenant's keys are prefixed with ns:{tenant}:. Operators can inspect the live state directly with redis-cli. This is useful when the TUI is unavailable or when you need to run programmatic checks.

# Live event stream for a tenant
redis-cli MONITOR | rg 'ns:my-team'

# Check if a worker is still alive
redis-cli HGET ns:my-team:worker_slots forge-1

Monitoring NEXUS Reconcile

NEXUS runs reconcile() on every poll cycle. It scans the SharedStore for inconsistencies and takes corrective action. Reconcile is the heart of OpenFlows' ability to self-heal. Monitoring it means watching for the categories of problems it detects and the actions it takes.

Reconcile events are emitted to the tenant's event ring and logged at INFO level. Look for lines containing reconcile: in the controller log. Each reconcile event has a category field that describes the type of inconsistency detected.

# Watch reconcile events in real time
tail -f /tmp/openflows-controller.log | rg 'reconcile:'

# Read the last 50 reconcile events from Redis
redis-cli LRANGE ns:my-team:event_ring 0 49 | rg 'reconcile'

Scaling Workers

FORGE workers are the primary parallel resource. Each instance is an independent Coder workspace. Scaling is done by changing the agents.forge.instances value in registry.json. The change is hot-reloaded on the next NEXUS poll cycle.

Scaling considerations:

• Each instance consumes a Coder workspace slot and a GitHub MCP connection.
• Concurrency is bounded by the Coder provisioner's capacity and the AI Gateway rate limits.
• More instances do not make a single ticket faster; they increase the number of tickets that can be processed in parallel.
• Reduce instances during off-hours to lower Coder and LLM costs.

{
  "agents": {
    "forge": { "instances": 4 }
  }
}