about summary refs log tree commit diff
path: root/src/error.rs
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-08 12:29:22 +0000
committerPjotr <wrk@bosa.lan>2026-11-08 12:29:22 +0000
commitc008bdeca126062b2ab87bb82e7fc9ae05060e5f (patch)
tree329b595a9e18733ac559c446c11d3b320c7a31b2 /src/error.rs
parentee939a246a5b67f6650e0759e23fe2731124f8c3 (diff)
downloadgenenetwork4-c008bdeca126062b2ab87bb82e7fc9ae05060e5f.tar.gz
db: recover from a connection the server closed, and report the database
Every database-backed endpoint 500'd with `CodecError { IO error: Broken
pipe }` while the process answered /version quite happily. The cause was
ours: `pool()` turns off the driver's `check_health` (a ping at checkout,
dead connections replaced) to save a round trip -- sound against a
database 130 ms away, worth nothing against loopback, and it costs the
pool its self-healing. One server-side close of an idle session then
becomes one failed request per dead socket: ten in a row, until a
restart.

* src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping;
`is_connection_loss` recognises the errors that mean the peer went away
(broken pipe, reset, went away, lost connection, closed);
`forget_pool` drops the cached pool so the next call rebuilds it;
`health` probes reachability, cached ten seconds. One pool static shared
by pool()/forget_pool (they had drifted into two).
* src/error.rs (ApiError::from_db): on a connection loss, forget the pool
-- the one place every database error passes through, so recovery does
not depend on each caller.
* src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a
service that answers while nothing behind it works is now visible.
* src/config.rs: the switch, with the trade written down.
* tests/db_recovery.rs: kills every pooled session from a second
connection, then asserts the next twelve requests all succeed -- the
scenario that produced the outage, now a regression test.
* tests/...: unit test for the classifier (its own IO error text and the
"this is a bad query, do not retry" cases).
* README.md ("Database connections"), docs/www/operator-checklist.md:
the knob, why it defaults on, and what /version reports.
Diffstat (limited to 'src/error.rs')
-rw-r--r--src/error.rs25
1 files changed, 18 insertions, 7 deletions
diff --git a/src/error.rs b/src/error.rs
index 7daa103..a9b01bd 100644
--- a/src/error.rs
+++ b/src/error.rs
@@ -128,13 +128,24 @@ impl ApiError {
                 detail,
                 operation_id,
             ),
-            crate::db::DbError::Query(error) => ApiError::new(
-                StatusCode::INTERNAL_SERVER_ERROR,
-                "database_error",
-                "Database error",
-                error.to_string(),
-                operation_id,
-            ),
+            crate::db::DbError::Query(error) => {
+                // A connection that died under the query is not the
+                // query's fault. Forget the pool so the next request
+                // cannot draw the same dead sockets -- this is the one
+                // place every database error passes through, so the
+                // recovery does not depend on each caller remembering it.
+                let api_error = ApiError::new(
+                    StatusCode::INTERNAL_SERVER_ERROR,
+                    "database_error",
+                    "Database error",
+                    error.to_string(),
+                    operation_id,
+                );
+                if crate::db::is_connection_loss(&crate::db::DbError::Query(error)) {
+                    crate::db::forget_pool();
+                }
+                api_error
+            }
         }
     }
 }