about summary refs log tree commit diff
path: root/src/lib.rs
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-08 12:29:22 +0000
committerPjotr <wrk@bosa.lan>2026-11-08 12:29:22 +0000
commitc008bdeca126062b2ab87bb82e7fc9ae05060e5f (patch)
tree329b595a9e18733ac559c446c11d3b320c7a31b2 /src/lib.rs
parentee939a246a5b67f6650e0759e23fe2731124f8c3 (diff)
downloadgenenetwork4-c008bdeca126062b2ab87bb82e7fc9ae05060e5f.tar.gz
db: recover from a connection the server closed, and report the database
Every database-backed endpoint 500'd with `CodecError { IO error: Broken
pipe }` while the process answered /version quite happily. The cause was
ours: `pool()` turns off the driver's `check_health` (a ping at checkout,
dead connections replaced) to save a round trip -- sound against a
database 130 ms away, worth nothing against loopback, and it costs the
pool its self-healing. One server-side close of an idle session then
becomes one failed request per dead socket: ten in a row, until a
restart.

* src/db.rs: `SQL_CHECK_HEALTH` (default true) decides the ping;
`is_connection_loss` recognises the errors that mean the peer went away
(broken pipe, reset, went away, lost connection, closed);
`forget_pool` drops the cached pool so the next call rebuilds it;
`health` probes reachability, cached ten seconds. One pool static shared
by pool()/forget_pool (they had drifted into two).
* src/error.rs (ApiError::from_db): on a connection loss, forget the pool
-- the one place every database error passes through, so recovery does
not depend on each caller.
* src/lib.rs (/version): `"database": "reachable" | "unreachable"`; a
service that answers while nothing behind it works is now visible.
* src/config.rs: the switch, with the trade written down.
* tests/db_recovery.rs: kills every pooled session from a second
connection, then asserts the next twelve requests all succeed -- the
scenario that produced the outage, now a regression test.
* tests/...: unit test for the classifier (its own IO error text and the
"this is a bad query, do not retry" cases).
* README.md ("Database connections"), docs/www/operator-checklist.md:
the knob, why it defaults on, and what /version reports.
Diffstat (limited to 'src/lib.rs')
-rw-r--r--src/lib.rs9
1 files changed, 9 insertions, 0 deletions
diff --git a/src/lib.rs b/src/lib.rs
index 2bba755..c3efcca 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -154,6 +154,10 @@ struct VersionInfo {
     source_digest: &'static str,
     /// The branch the build's HEAD was on, or "HEAD" when detached.
     branch: &'static str,
+    /// Whether the database answers, checked at most every ten seconds.
+    /// A service that answers `/version` while every data endpoint fails
+    /// is the state this makes visible.
+    database: &'static str,
 }
 
 #[utoipa::path(get, path = "/version", summary = "Build information: the git commit the binary was built from, and a source digest", responses(
@@ -176,5 +180,10 @@ async fn version(ctx: Ctx) -> Json<Envelope<VersionInfo>> {
         dirty: git.ends_with("-dirty"),
         source_digest: env!("GN4_SOURCE_DIGEST"),
         branch: env!("GN4_GIT_BRANCH"),
+        database: if crate::db::health() {
+            "reachable"
+        } else {
+            "unreachable"
+        },
     }))
 }