diff options
| author | Pjotr <wrk@bosa.lan> | 2026-11-08 12:29:22 +0000 |
|---|---|---|
| committer | Pjotr <wrk@bosa.lan> | 2026-11-08 12:29:22 +0000 |
| commit | c008bdeca126062b2ab87bb82e7fc9ae05060e5f (patch) | |
| tree | 329b595a9e18733ac559c446c11d3b320c7a31b2 /src/error.rs | |
| parent | ee939a246a5b67f6650e0759e23fe2731124f8c3 (diff) | |
| download | genenetwork4-c008bdeca126062b2ab87bb82e7fc9ae05060e5f.tar.gz | |
db: recover from a connection the server closed, and report the database
Every database-backed endpoint 500'd with `CodecError { IO error: Broken
pipe }` while the process answered /version quite happily. The cause was
ours: `pool()` turns off the driver's `check_health` (a ping at checkout,
dead connections replaced) to save a round trip -- sound against a
database 130 ms away, worth nothing against loopback, and it costs the
pool its self-healing. One server-side close of an idle session then
becomes one failed request per dead socket: ten in a row, until a
restart.
* src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping;
`is_connection_loss` recognises the errors that mean the peer went away
(broken pipe, reset, went away, lost connection, closed);
`forget_pool` drops the cached pool so the next call rebuilds it;
`health` probes reachability, cached ten seconds. One pool static shared
by pool()/forget_pool (they had drifted into two).
* src/error.rs (ApiError::from_db): on a connection loss, forget the pool
-- the one place every database error passes through, so recovery does
not depend on each caller.
* src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a
service that answers while nothing behind it works is now visible.
* src/config.rs: the switch, with the trade written down.
* tests/db_recovery.rs: kills every pooled session from a second
connection, then asserts the next twelve requests all succeed -- the
scenario that produced the outage, now a regression test.
* tests/...: unit test for the classifier (its own IO error text and the
"this is a bad query, do not retry" cases).
* README.md ("Database connections"), docs/www/operator-checklist.md:
the knob, why it defaults on, and what /version reports.
Diffstat (limited to 'src/error.rs')
| -rw-r--r-- | src/error.rs | 25 |
1 files changed, 18 insertions, 7 deletions
diff --git a/src/error.rs b/src/error.rs index 7daa103..a9b01bd 100644 --- a/src/error.rs +++ b/src/error.rs @@ -128,13 +128,24 @@ impl ApiError { detail, operation_id, ), - crate::db::DbError::Query(error) => ApiError::new( - StatusCode::INTERNAL_SERVER_ERROR, - "database_error", - "Database error", - error.to_string(), - operation_id, - ), + crate::db::DbError::Query(error) => { + // A connection that died under the query is not the + // query's fault. Forget the pool so the next request + // cannot draw the same dead sockets -- this is the one + // place every database error passes through, so the + // recovery does not depend on each caller remembering it. + let api_error = ApiError::new( + StatusCode::INTERNAL_SERVER_ERROR, + "database_error", + "Database error", + error.to_string(), + operation_id, + ); + if crate::db::is_connection_loss(&crate::db::DbError::Query(error)) { + crate::db::forget_pool(); + } + api_error + } } } } |
