Drive timeouts from a shared Redis deadline queue

Turn auto-play and hand-end auto-continue were process-local asyncio
tasks armed only by client connects and state broadcasts: with no
sockets connected the next turn's timer was never armed, a hand-end
timer died with its worker, and neither survived a pod restart.

Deadlines are now driven by the absolute timestamps persisted on the
game state and enqueued in a shared Redis sorted set. Every worker runs
a consumer that fires due entries under the per-game lock after
revalidating them against the live state, so timeouts no longer depend
on any player being connected and survive the death of any worker.
Delivery is at-least-once: entries are removed only after processing,
and revalidation makes duplicate deliveries no-ops.

Queue entries carry the deadline as integer epoch milliseconds, which
also serves as the revalidation token, and the score derives from the
same value.
This commit is contained in:
2026-09-18 00:59:54 +00:00
parent 22ef689092
commit 5dc3f4cb17
9 changed files with 610 additions and 138 deletions
+12 -2
View File
@@ -60,6 +60,7 @@ All configuration comes from environment variables (see `.env.example`):
| `GAME_TTL_SECONDS` | `86400` | Sliding TTL of a live game in Redis |
| `HAND_ACK_TIMEOUT_SECONDS` | `30` | Seconds the between-hands scoring summary waits for acknowledgements |
| `TURN_TIMEOUT_SECONDS` | `30` | Seconds a player has to play before the server plays a random legal card for them |
| `DEADLINE_HEARTBEAT_MS` | `1000` | Upper bound on the deadline consumer's poll interval (locally enqueued deadlines fire on time regardless) |
| `LOGGING_CONFIG` | unset | Path to a YAML logging configuration file (see below). Unset logs DEBUG to the console |
| `APP_HOST` / `APP_PORT` | `0.0.0.0` / `8080` | Bind address |
@@ -111,6 +112,12 @@ loggers:
- `tavolo:game:<uuid>:events` — a pub/sub channel carrying "state changed"
signals; every open WebSocket reloads the state and pushes the
personalized view to its player.
- `tavolo:deadlines` — a sorted set (score = due timestamp) of pending
timeouts: turn auto-plays and hand-end auto-continues. Every worker runs
a consumer that fires due entries under the per-game lock, so timeouts
do not depend on any player being connected and survive the death of
any worker (delivery is at-least-once; entries are revalidated against
the live state before firing).
### Postgres (statistics, via Tortoise ORM + aerich migrations)
@@ -181,7 +188,9 @@ player on turn does not move before it, the server plays a random legal
card for them (picking one of the legal captures at random when a capture
is required), so a disconnected or idle player cannot stall the match. The
timeout is `TURN_TIMEOUT_SECONDS` (default 30); the auto-played move is
broadcast like any other.
broadcast like any other. Deadlines fire from the shared `tavolo:deadlines`
queue (see above), not from timers tied to client connections, so the
match keeps progressing even with every player disconnected.
### Hand-end summary
@@ -257,7 +266,8 @@ src/tavolo/
├── aerich_config.py # aerich CLI configuration
├── models.py # Match, MatchPlayer (Postgres)
├── stats.py # finished match -> Postgres persistence
├── store.py # Redis / in-memory live-game store
├── store.py # Redis / in-memory live-game store (+ deadline queue)
├── deadlines.py # connection-independent timeout scheduler
├── ws.py # WebSocket live-play endpoint
├── game/
│ ├── state.py # GameState / PlayerState / Card, JSON (de)serialization