fix: stop the cache index stampede that wipes bots and NPCs on restart #208

Merged

sickday merged 1 commit from fix/cache-index-stampede into main

2026-07-22 12:43:03 +00:00

sickday commented

2026-07-22 12:36:53 +00:00

Owner

A restart with a saved fleet killed 279 bots and the entire NPC
population within 30 seconds of boot, every one of them exiting on

GenServer.call(cache_pid, {:request_index, 5}, 5000) ** (EXIT) time out

Cache.Server served every archive index from handle_call, and archive 5's
index is a list of 1852 group structs — so each of those replies deep-copied
a large term through one serialized process. Region loads take that path,
and at boot 816 resuming bots, the NPC collision preload and the region
warmer all want cold regions at once. Measured on the dev box: 2036us per
call, and 800 concurrent region loads spend 1.8s queueing. On the slower
deployment hardware that clears the 5s default timeout, and the callers die.

Indices are immutable for the life of a revision, so parse them once in
handle_continue and publish to persistent_term, which reads by reference
with no copy and no process involved. request_index/2 keeps its signature
and falls back to a call for anything unwarmed. The server stays the single
writer for its own key. All 16 archives warm in ~100ms at boot.

request_index: 2036us -> 0.18us
800 concurrent region loads: 1831ms -> 94ms wall, p50 1055ms -> 7.1ms

Two changes so this class of stall degrades instead of deleting the world:

NpcSupervisor.preload_spawn_regions/1 ran before the spawn loop with no
guard, so one timed-out region took every NPC with it — the world came up
with zero NPCs and only a task-terminated line to explain it. Preloading is
now per-region best-effort, matching the filters above it: spawning is the
safer failure.

Bot.Manager resumed the whole fleet in one loop, putting every bot on the
same tick needing the same cold regions. It now spawns in batches of 50 with
the queue in state, so demand is spread and the Manager stays answerable.

A restart with a saved fleet killed 279 bots and the entire NPC population within 30 seconds of boot, every one of them exiting on GenServer.call(cache_pid, {:request_index, 5}, 5000) ** (EXIT) time out Cache.Server served every archive index from handle_call, and archive 5's index is a list of 1852 group structs — so each of those replies deep-copied a large term through one serialized process. Region loads take that path, and at boot 816 resuming bots, the NPC collision preload and the region warmer all want cold regions at once. Measured on the dev box: 2036us per call, and 800 concurrent region loads spend 1.8s queueing. On the slower deployment hardware that clears the 5s default timeout, and the callers die. Indices are immutable for the life of a revision, so parse them once in handle_continue and publish to persistent_term, which reads by reference with no copy and no process involved. request_index/2 keeps its signature and falls back to a call for anything unwarmed. The server stays the single writer for its own key. All 16 archives warm in ~100ms at boot. request_index: 2036us -> 0.18us 800 concurrent region loads: 1831ms -> 94ms wall, p50 1055ms -> 7.1ms Two changes so this class of stall degrades instead of deleting the world: NpcSupervisor.preload_spawn_regions/1 ran before the spawn loop with no guard, so one timed-out region took every NPC with it — the world came up with zero NPCs and only a task-terminated line to explain it. Preloading is now per-region best-effort, matching the filters above it: spawning is the safer failure. Bot.Manager resumed the whole fleet in one loop, putting every bot on the same tick needing the same cold regions. It now spawns in batches of 50 with the queue in state, so demand is spread and the Manager stays answerable.

sickday added 1 commit

2026-07-22 12:36:53 +00:00

fix: stop the cache index stampede that wipes bots and NPCs on restart

ci / test (pull_request) Successful in 2m30s

Details

45f7ce8a28

A restart with a saved fleet killed 279 bots and the entire NPC
population within 30 seconds of boot, every one of them exiting on

  GenServer.call(cache_pid, {:request_index, 5}, 5000) ** (EXIT) time out

Cache.Server served every archive index from handle_call, and archive 5's
index is a list of 1852 group structs — so each of those replies deep-copied
a large term through one serialized process. Region loads take that path,
and at boot 816 resuming bots, the NPC collision preload and the region
warmer all want cold regions at once. Measured on the dev box: 2036us per
call, and 800 concurrent region loads spend 1.8s queueing. On the slower
deployment hardware that clears the 5s default timeout, and the callers die.

Indices are immutable for the life of a revision, so parse them once in
handle_continue and publish to persistent_term, which reads by reference
with no copy and no process involved. request_index/2 keeps its signature
and falls back to a call for anything unwarmed. The server stays the single
writer for its own key. All 16 archives warm in ~100ms at boot.

  request_index: 2036us -> 0.18us
  800 concurrent region loads: 1831ms -> 94ms wall, p50 1055ms -> 7.1ms

Two changes so this class of stall degrades instead of deleting the world:

NpcSupervisor.preload_spawn_regions/1 ran before the spawn loop with no
guard, so one timed-out region took every NPC with it — the world came up
with zero NPCs and only a task-terminated line to explain it. Preloading is
now per-region best-effort, matching the filters above it: spawning is the
safer failure.

Bot.Manager resumed the whole fleet in one loop, putting every bot on the
same tick needing the same cold regions. It now spawns in batches of 50 with
the queue in state, so demand is spread and the Manager stays answerable.

sickday merged commit f6c23d6f44 into main

2026-07-22 12:43:03 +00:00

sickday referenced this pull request from a commit

2026-07-22 12:43:04 +00:00

fix: stop the cache index stampede that wipes bots and NPCs on restart (#208)