diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-08 12:29:22 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-08 12:29:22 +0000 |
| commit | c008bdeca126062b2ab87bb82e7fc9ae05060e5f (patch) | |
| tree | 329b595a9e18733ac559c446c11d3b320c7a31b2 /test/python/test_traits.py | |
| parent | ee939a246a5b67f6650e0759e23fe2731124f8c3 (diff) | |
| download | genenetwork4-c008bdeca126062b2ab87bb82e7fc9ae05060e5f.tar.gz | |
db: recover from a connection the server closed, and report the database
Every database-backed endpoint 500'd with `CodecError { IO error: Broken
pipe }` while the process answered /version quite happily. The cause was
ours: `pool()` turns off the driver's `check_health` (a ping at checkout,
dead connections replaced) to save a round trip -- sound against a
database 130 ms away, worth nothing against loopback, and it costs the
pool its self-healing. One server-side close of an idle session then
becomes one failed request per dead socket: ten in a row, until a
restart.
* src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping;
`is_connection_loss` recognises the errors that mean the peer went away
(broken pipe, reset, went away, lost connection, closed);
`forget_pool` drops the cached pool so the next call rebuilds it;
`health` probes reachability, cached ten seconds. One pool static shared
by pool()/forget_pool (they had drifted into two).
* src/error.rs (ApiError::from_db): on a connection loss, forget the pool
-- the one place every database error passes through, so recovery does
not depend on each caller.
* src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a
service that answers while nothing behind it works is now visible.
* src/config.rs: the switch, with the trade written down.
* tests/db_recovery.rs: kills every pooled session from a second
connection, then asserts the next twelve requests all succeed -- the
scenario that produced the outage, now a regression test.
* tests/...: unit test for the classifier (its own IO error text and the
"this is a bad query, do not retry" cases).
* README.md ("Database connections"), docs/www/operator-checklist.md:
the knob, why it defaults on, and what /version reports.
Diffstat (limited to 'test/python/test_traits.py')
0 files changed, 0 insertions, 0 deletions
