The Bots pane was empty while a bot was talking to me

Hermes Desktop has a BOTS tab that lists every agent profile: default, game-night, whatever else lives under ~/.hermes/profiles. One afternoon that list vanished. The pane sat on a spinner. I was already mid-conversation with a bot in the same window.

That mismatch is what made the bug interesting. The gateway was up. Chat streamed. The roster, which is just a profiles.list RPC plus some React Query glue, acted like the backend was dead.

This is the path from that empty sidebar to PR #92793. There were three stacked failures, not one. Each one we “fixed” made the next one visible.

What the pane is supposed to do

Every five seconds the hermes-bots plugin runs useRoster(). That hook calls profiles.list on the active gateway and paints rows from the result: name, last session, the canonical “Bot Chat” preview.

On the server, tui_gateway/methods_profiles.py walks every profile directory. For each one it opens that profile’s state.db, looks up the session titled exactly "Bot Chat", and attaches last_session, worker_session, and canonical_session. The desktop never stores a session pointer. Identity is the title.

That inspect path used to construct a normal SessionDB(). A writable SessionDB is the live-agent handle. It takes write-lock patience (up to 20 seconds) and runs schema init. That is the right constructor for a turn that is about to write messages. It is the wrong constructor for a sidebar that polls.

The first wrong theory

The first guess was a stale gateway. Restart Hermes, the spinner stays. Restart the gateway, same thing. The chat next to the pane kept working, so “the backend is down” was already a bad story. We kept it around too long because the error card, when it finally appeared, said “Roster unavailable” and that sounds like connectivity.

It wasn’t.

Failure 1: a reader that waited like a writer

profiles.list opened every profile state.db as a writer. A live turn already holds that file. SQLite’s BEGIN IMMEDIATE from the agent, plus 20 seconds of SessionDB lock patience on the inspect path, meant one busy profile could stall the whole RPC past the desktop’s request timeout.

The Bots pane polls every 5 seconds. A 20 second wait per profile is already longer than the poll. Several profiles and you blow the WebSocket budget every tick.

Worse, useRoster was configured with React Query retry: true. In TanStack Query v5 that means infinite retries, and isLoading stays true until the first success. The pane’s render rule is roughly: if we are loading and have no snapshot, show a spinner. There is an error card. You never reach it if the query never rejects.

So the UI told a clean lie. The backend was answering other RPCs. The roster RPC was wedged on a lock. The client treated “still trying” as “still loading.” Empty sidebar, live chat.

The fix on this layer is small. Extract _open_profile_session_db(), open with SessionDB(..., read_only=True), share that handle for last/worker/canonical rows, close it in a finally. read_only=True is the inspect constructor SessionDB already documents: no write lock, no DDL, WAL readers can run next to a live writer.

The test that made this real holds BEGIN IMMEDIATE on state.db and asserts profiles.list returns in under 3 seconds with the Bot Chat preview still populated:

holder = sqlite3.connect(str(home / "state.db"), isolation_level=None, timeout=0)
holder.execute("BEGIN IMMEDIATE")
rows = _profiles({})
assert elapsed < 3.0
assert "hello from bot" in canonical["preview"]

A second test spies on SessionDB.__init__ and requires read_only=True on every open. We also bounded the client: ROSTER_QUERY_RETRY = 2. After two failures the error card can appear. The existing 5 second refetchInterval and the gateway-open effect still recover a dropped SSH session. Infinite isLoading cannot.

Failure 2: the function that was never there

Bounded retries did what they were supposed to. The spinner went away. In its place:

Roster unavailable: activeBotRoute is not defined

That is a ReferenceError, not a timeout. useRoster() called activeBotRoute() before profiles.list. The helper was added in a remote-routing commit, then the definition was dropped while the call sites stayed. JavaScript does not care that you meant to feature-detect it.

Once you see the name in the error string the fix is obvious: put the function back. Feature-detect host.profileRoutes. If the host is old, return null and use the unscoped host.request door. If the inventory throws, also return null. A missing route must not take the pane down.

After a rebuild, BOTS listed default, game-night, game-night-dev, and mpowa-swe. The crash was gone. The lock fix was in the same PR. We shipped both.

Failure 3: we put the slow call back in front

Code review caught the next one before it became another support thread.

Restoring activeBotRoute() was correct as a crash fix and wrong as a first-paint dependency. On current Desktop, host.profileRoutes() goes through refreshProfiles() → GET /api/profiles with a 60 second startup timeout. useRoster awaited that before profiles.list.

retry: 2 never fires if the promise has not rejected. A slow or hanging inventory looks exactly like the original spinner, except now the database is fine.

The right door for “list the bots on this window” is the socket already attached to the window. That is host.request('profiles.list'). Other connections still arrive through host.agents(). Route inventory can stay for tests and for callers that actually need a connectionId + profile pair. It does not belong on the critical path of first paint.

useRoster and the hide-sweep now look like this:

const local = await host.request('profiles.list', {})

No await activeBotRoute() in front.

What we did not do

We did not add a timeout wrapper around profileRoutes and keep the serial await. That still makes first paint wait on REST. We did not keep retry: true “because SSH flaps.” Bounded retries plus a refetch interval is the desktop rule: retries end in a recovery affordance, never an infinite spinner.

We also did not try to make the Python lock test cover every journal mode. The regression we hit in production is WAL + a live BEGIN IMMEDIATE writer, which is what a normal backend holds. DELETE-journal and BEGIN EXCLUSIVE (VACUUM, truncate checkpoint) can still cost up to 1 second of sqlite busy timeout per profile. That is a leftover risk, not the 20 second stall.

Tests that actually execute

The first roster test regex-matched plugin.js for retry: ROSTER_QUERY_RETRY. That is banned in this repo for a reason: a source scan stays green if you wire the constant to nothing, or set it to 10000. Review deleted the regex and pinned ROSTER_QUERY_RETRY === 2. The same VM harness now stubs host.profileRoutes and checks match, throw, and no-match. Existence of a function is not a contract.

The Python tests open a real state.db. One spies constructor kwargs. One holds a writer and times the RPC. Those are the tests I would keep if we deleted everything else.

The shape I keep seeing

Three different layers produced the same screenshot.

1. Server: an inspect RPC used a writer constructor.
2. Client policy: infinite retry turned a timeout into a permanent load state.
3. Client wiring: a missing symbol, then a restored symbol that serialized a 60 second REST call in front of the RPC we actually needed.

The user-visible sentence was always “the Bots list is empty.” Debugging it as one bug would have left the other two in place. The lock fix alone still crashes on activeBotRoute. The crash fix alone still hangs on /api/profiles. The inventory skip alone still wedged on state.db.

If a pane can be empty while the thing it lists is obviously alive, look at the query’s loading contract before you look at the network. Then check whether the “read” path is actually taking a write lock. Then check whether first paint is waiting on something that is not the data it paints.