World.Content.LightTest is seed-flaky: it needs a 65/256 roll to fail #119

New issue

Open

opened 2026-08-05 02:40:00 +00:00 by sickday · 0 comments

sickday commented

2026-08-05 02:40:00 +00:00

Owner

World.Content.LightTest — "a failed attempt is retried on the very next
tick" (test/revenant/world/content/light_test.exs:270) fails on roughly one
run in five, and it is not flaky in the "sometimes the machine is slow"
sense: it is deterministic per ExUnit seed.

Reproduction

mix test --seed 12 test/revenant/world/content/light_test.exs
mix test --seed 15 test/revenant/world/content/light_test.exs
mix test --seed 20 test/revenant/world/content/light_test.exs

3 of the first 20 integer seeds fail. The file alone is enough — no other test
file, no world, no database.

1) test the loop over the real floor a failed attempt is retried on the
   very next tick (World.Content.LightTest)
   Expected truthy, got false
   code: assert World.Timers.has?(failed.timers, Light)

The mechanism

The test drives World.Content.Light.run/2, which calls
Contract.Skilling.dice() for itself. That is &(:rand.uniform(&1) - 1) on
the calling process, and ExUnit seeds :rand from the run seed, so the
roll is a pure function of --seed.

The test needs the roll to fail in order to observe the retry. At
Firemaking 1 a log succeeds 65/256 of the time — about 25% — so about a
quarter of seeds light the fire instead, the loop ends, and the timer the
assertion looks for is gone. The measured 3/20 is that number with sampling
noise.

The file's own moduledoc says it is "driven with the dice handed in so every
branch is reachable without waiting on chance", and every other test in it
is. This one is the exception, because it is the only one exercising run/2
rather than attempt/5.

Why this is worth writing down

It presents as "passes alone, fails in the suite", which is the exact
signature of the World.FiremakingWorldTest hardcoded-tick trap CLAUDE.md
documents — and it is nothing to do with it. Adding any test file to a
selection reshuffles the order and therefore re-rolls the seed lottery, so the
failure appears to be caused by whatever was added last. It was blamed on the
fishing branch on that basis and is not fishing's.

The fix

Light.attempt/5 already takes the dice as an argument for exactly this
reason. run/2 does not, and it is the only entry point that rolls. Either
thread a dice through the timer args the way facts is threaded through
attempt/5, or rewrite this one test to drive attempt/5 and assert the
reschedule separately.

Every other skilling loop has the same shape (Chop, Mine, Fish all roll
inside run/2 with Skilling.dice()), so whoever fixes this should check
whether their own tests avoid run/2 by accident or by design.

`World.Content.LightTest` — *"a failed attempt is retried on the very next tick"* (`test/revenant/world/content/light_test.exs:270`) fails on roughly one run in five, and it is **not** flaky in the "sometimes the machine is slow" sense: it is deterministic per ExUnit seed. ## Reproduction mix test --seed 12 test/revenant/world/content/light_test.exs mix test --seed 15 test/revenant/world/content/light_test.exs mix test --seed 20 test/revenant/world/content/light_test.exs 3 of the first 20 integer seeds fail. The file alone is enough — no other test file, no world, no database. 1) test the loop over the real floor a failed attempt is retried on the very next tick (World.Content.LightTest) Expected truthy, got false code: assert World.Timers.has?(failed.timers, Light) ## The mechanism The test drives `World.Content.Light.run/2`, which calls `Contract.Skilling.dice()` for itself. That is `&(:rand.uniform(&1) - 1)` on the calling process, and **ExUnit seeds `:rand` from the run seed**, so the roll is a pure function of `--seed`. The test needs the roll to **fail** in order to observe the retry. At Firemaking 1 a log succeeds `65/256` of the time — about 25% — so about a quarter of seeds light the fire instead, the loop ends, and the timer the assertion looks for is gone. The measured 3/20 is that number with sampling noise. The file's own moduledoc says it is *"driven with the dice handed in so every branch is reachable without waiting on chance"*, and every other test in it is. This one is the exception, because it is the only one exercising `run/2` rather than `attempt/5`. ## Why this is worth writing down It presents as *"passes alone, fails in the suite"*, which is the exact signature of the `World.FiremakingWorldTest` hardcoded-tick trap CLAUDE.md documents — and it is nothing to do with it. Adding any test file to a selection reshuffles the order and therefore re-rolls the seed lottery, so the failure appears to be caused by whatever was added last. It was blamed on the fishing branch on that basis and is not fishing's. ## The fix `Light.attempt/5` already takes the dice as an argument for exactly this reason. `run/2` does not, and it is the only entry point that rolls. Either thread a dice through the timer args the way `facts` is threaded through `attempt/5`, or rewrite this one test to drive `attempt/5` and assert the reschedule separately. Every other skilling loop has the same shape (`Chop`, `Mine`, `Fish` all roll inside `run/2` with `Skilling.dice()`), so whoever fixes this should check whether their own tests avoid `run/2` by accident or by design.

sickday referenced this issue

2026-08-05 10:14:55 +00:00

feat(fishing): eleven methods over 66 spots, on the shared skilling clock #120

sickday referenced this issue

2026-08-06 13:05:47 +00:00

feat(runecrafting): twelve altars, and the half of the map no name scan can see #132

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Revenant/Server#119

No description provided.

Rows
Columns