about summary refs log tree commit diff
path: root/src/compute_gate.rs
AgeCommit message (Collapse)Author
6 dayscompute/bnw: a Bayesian network over traits, called out to BNW's driverPjotr
POST /compute/bnw learns a network over a set of variables and returns one row per candidate edge: from, to, posterior, in_structure. Inputs are published traits of a group (group + traits) or an explicit matrix (values: one entry per variable with per-sample values), which covers the case the database cannot -- a basket's columns, a handful of probe sets. BNW's knowledge stays in the gn-bnw repository. GN4 writes a variables x samples matrix, runs `bnw learn` (BNW_CMD, BNW_BIN_DIR, BNW_CACHE_DIR) and parses the reply: nothing about the three C programs, the private work directory or the type line in the input file lives here, so BNW's pipeline can change without a GN4 release. The driver's failures map onto the API's own: bad input 422, a missing driver 502 bnw_not_installed (its exit 77), a failed computation 502 bnw_failed in the driver's words. Decisions worth recording: * parameters are BNW's own defaults -- maxparents 4, k 1, tiers 3, threshold 0.5, read from its create_tiers_gom_part1.php rather than chosen here -- and the reply names the values used, because a network at 0.5 and one at 0.9 are different answers to the same question; * at most 40 variables and at least 3 samples complete across every variable. The first is the structure search's exponential in-degree; the second is BNW modelling a complete multivariate normal, so a sample missing any value is dropped -- and the note says how many survived rather than silently shrinking the panel; * its own gate, always claimed (threshold 0): there is no such thing as a small BNW run, the cost is the search, not the number of individuals; * the notes say plainly that a posterior is not a p-value and that a network is not a QTL, because that is the easiest thing to over-read. tests/bnw.rs compares the endpoint against the answer the BNW *web application* gives for its own six-trait example (the same fixture as the driver's parity check, gn-bnw/test/golden/): the five edges must match, and the notes must name the parameters and carry the caveat. It skips when the driver or its binaries are absent. Not deployed to production.
7 daysbulklmm: an LMM scan that can be asked for thresholdsPjotr
GEMMA's LOCO scan is the reference for a biallelic panel, but its p-values have no threshold we can compute: a GEMMA permutation repeats the whole scan, which is why /compute/gemma refuses nperm. R/qtl's nperm is the usual stand-in and it is the wrong null -- it shuffles phenotypes as if the individuals were unrelated. BulkLMM.jl fits an LMM per marker with a global kinship matrix and permutes by rotating the data through the kinship's eigen decomposition, so a threshold costs seconds. POST /compute/bulklmm {group, genotype_version, samples|basket, nperm, seed, reml, assumption} Rows are marker, chr, pos (Mb), lod, beta, h2. BXD trait 16627 (BXD.8, 35 phenotyped animals, 21056 markers) in 22 s: peak rsm10000000888 chr2 4.92 Mb, LOD 9.59, and thresholds alpha 0.10 -> 3.51, 0.05 -> 3.81, 0.01 -> 4.47 from 200 rotations with seed 42. That is the trait's published locus and LOD, and a significant peak against a threshold this API could not compute before. Two differences from GEMMA are in every reply, not implied: a global kinship matrix rather than leave-one-chromosome-out, and h2_null saying how much relatedness was found (here ~0, so the fit is effectively fixed-effects -- which is why it agrees with R/qtl where GEMMA differs). * src/bulklmm.rs prepares what Julia reads -- the BIMBAM dosages as a strains x markers matrix (our BIMBAM pair is already the allele dosages BulkLMM wants), a marker map, the phenotype and covariates -- runs the driver through streaming::run_streaming_with_env, and reads the CSV back. A row whose LOD is NaN is dropped: NaN is a valid f64 in Rust and would otherwise be reported as a value. * scripts/bulklmm_scan.jl is the driver (scan/calcKinship/get_thresholds), with the seed recorded so a threshold set is reproducible. * gate::bulklmm (2 slots, GEMMA-sized threshold): a permutation here is cheap, unlike R/qtl's. * Julia comes from the deployment's own store binary with a prepared depot -- 79 packages, 139 MB, no GPU/CUDA stack -- and every request runs JULIA_PKG_OFFLINE=true, so a scan can never download anything. `--project=<path>`, not `--project <path>`: Julia 1.8 takes the space-separated form as a program argument. tests/bulklmm.rs: the peak stays on chr2 above LOD 3, the thresholds and seed are in the notes, and a scan without permutations says there is no threshold rather than leaving a bare LOD to be read as significant.
8 dayscompute: R/qtl2 as a third scan, POST /compute/qtl2Pjotr
R/qtl sits between the existing two scans: Haley-Knott on marker genotypes, no kinship. GEMMA is an LMM but biallelic BIMBAM with mean-imputed genotypes. Neither can analyse a multiparent population, and neither models relatedness from the actual genotype data. R/qtl2 does both -- HMM genotype probabilities, then an LMM with leave-one-chromosome-out kinship -- and it is what names the third peak on DOL trait 10003 that R/qtl's scan cannot see (chr16, LOD 4.24, against GEMMA's -log10 p 7.15 there). * src/qtl2.rs converts this deployment's own .geno into a qtl2 cross under QTL2_CACHE_DIR: control.json plus geno/gmap/pmap/pheno, with the genotype-code map, the orientation and the paths written out explicitly. The shipped genotype/rqtl2/ control files cannot be used -- they point outside the data directory, declare no code map (every genotype reads as missing) and no orientation (individuals read as markers) -- and each of those produces a plausible flat scan instead of an error, which is why the conversion is ours. * Heterozygous calls are dropped on a riself panel, as R/qtl's convert2riself does. Left in, qtl2 reads the cross as multiparent and its scan1 takes a path the panel cannot satisfy ("Error in scan1_pg ... : No individuals in common"). * Autosomes only: X needs a sex covariate to be fitted properly and Y/MT degenerate the per-chromosome kinship. GEMMA's LOCO scan is restricted the same way. The note says how many markers were left out. * scripts/qtl2_scan.R caches genotype probabilities and kinship by a hash of the control file *and its directory*; two conversions of the same panel can have identical control files and identical file sizes, and reusing the first one's probabilities silently scans the wrong cross. It also takes the individual names from whichever shape the cross's pheno slot has -- a single-column pheno file leaves it a plain matrix, and the resulting empty phenotype reads as scan1's "No individuals in common". * Rows are marker, chr, pos, lod, so /plot/rqtl draws the reply unchanged -- post the same samples as /compute/rqtl. * tests/qtl2.rs: the fixture scans and its notes name the cross; BXD trait 10001 peaks on chr8 under both scans; refusals carry missing_genotype / invalid_kinship. cost: 2 concurrent scans, gated from RQTL_LARGE_PANEL individuals like R/qtl; a DO cross costs minutes of genotype probabilities, paid once per panel. QTL2_SCRIPT_PATH and QTL2_CACHE_DIR configure it.
10 dayscompute: gate GEMMA from 50 individuals, R/qtl from 1 000Pjotr
One threshold for both engines was the wrong shape. For R/qtl the interesting line is where a `scanone` stops being seconds (a few hundred individuals is seconds of one core), so 1 000 stays. For GEMMA it is much earlier, and not because of the runtime: - every GEMMA run goes through **one shared cache directory** (`GEMMA_CACHE_DIR`), where `gemma-wrapper` extracts per-chromosome kinship archives of hundreds of megabytes by content hash. Two runs at once are two writers over the same archives, and a half-extracted kinship another run reads surfaces as a GEMMA failure; - a real BXD WGS run is ~19 s of every core it can get; UM-HET3's 6 382 individuals write 207 MB of kinship per chromosome. So `GEMMA_LARGE_PANEL` is 50 -- the smallest real panel, which means every mapping request is gated in practice, and only a handful of individuals (a four-way cross's parents, a probe of the pipeline) runs free. `ravanan` uses the same number, since it writes into the same cache. The threshold moved onto the `Gate` (`large_panel()`, `is_large()`), so each endpoint carries its own; the policy, the slots and the 409 are unchanged. Tests, asked and answered: at 50 the three gemma tests that do real work (fixtures 198 individuals, BXD.4 with `Status` 93, ravanan 198) are gated and share one token, so on a cold cache two of them at once would collide into `409 compute_busy` -- a gate doing its job, failing a test that only wanted to map something. They now take a lock, and only they do; the validation tests stay parallel, and the rqtl suite is untouched (its gate still starts at 1 000). Verified: with `GEMMA_CACHE_DIR` pointed at an empty directory, `cargo test --test gemma` is green twice in a row, 18 s per run, i.e. really recomputing. * src/compute_gate.rs: `RQTL_LARGE_PANEL` (1 000), `GEMMA_LARGE_PANEL` (50), `Gate::large_panel`/`is_large`, `Gate::new(permits, large_panel)`; the test that pinned one number now pins both and that each endpoint is wired to its own. * tests/gemma.rs: the three real-job tests take the lock, and say why. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: the two thresholds and the cache-directory reason for the lower one. 277 tests pass; the same four as before do not.
10 daysapi: describe case attributes, and gate large computations onlyPjotr
Two things that came out of reading a real response rather than a test. **Say what a case attribute means.** `GET /groups/{group}/case-attributes` returned names and values, and the per-trait attribute endpoint returned values, so a caller had to guess that `SeqCvge` is a sequencing depth. MariaDB has had the answer all along in `CaseAttribute.Description`: SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016 Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct Epoch BXD family subgroups. Each number with common parents. ... Sex F= female, M= male Both endpoints now carry it -- a `descriptions` map on the group listing, a `description` field on the per-trait one -- and the covariate notes quote it, so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for 1 male of each strain in 2016): categorical ..." instead of leaving the name to be interpreted. It matters most for covariates: `SeqCvge` is a property of the *genotype* data, its values include compound ones (`40+31L`, two sequencing runs), and the server codes those as 20-odd levels rather than as a depth -- a covariate fitted for the wrong reason, which the description is the first hint against. **Gate large computations, not every computation.** The gate was applied to every job, so a 93-strain test request could be refused because somebody else's 6 000-individual run held the slot -- which is how the test suite started serialising itself behind a mutex, a workaround for the wrong problem. What the gate is for is *large* work: a BXD panel of ~200 finishes in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000 individuals): below that no slot is taken, nothing waits and nothing is refused; above it the old policy holds unchanged (1 slot for gemma, 1 for ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409 naming the run in the way). The size is not known when the request arrives -- the panel is resolved server-side -- so the decorator now makes a *light* claim and the handler promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the point where it knows, then hands it to `Gate::run` around the call that computes, so the slot still outlives a client disconnect. A light claim registers nothing and its drop removes nothing: only a held claim may touch the claimant map. * src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`, `Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved out of `claim_or_busy`), and `Gate::run` unchanged in spirit. * gnauth-macros: binds the light claim; the body keeps one gate call. * src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is known, and the compute call wrapped in `run`; error mapping moved inside the job so the future returns `ApiError`. * src/rqtl.rs: `Sample` is `Clone` (the job owns its copy). * tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are under the threshold); rqtl keeps its lock only for the machine's sake -- concurrent `Rscript` scans would compete for cores and scratch, not for a slot. Four new gate tests pin the threshold, that a small job never takes the slot, that one caller cannot hold two large jobs, and that another caller waits rather than fails. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: what the threshold is and why it is 1 000. Measured on :3001 after the change: two concurrent small-panel GEMMA requests from one token both answer 200 (one was 409 before), while two 6 382-individual requests answer 502/409 -- the second gated with the usual `compute_busy` naming the first. 277 tests pass; the same four as before do not.
10 dayscompute: gate ravanan at one job, and name the binary that failedPjotr
`POST /compute/gemma/ravanan` is now gated like the other two: one workflow at a time, one per caller, `409 compute_busy` naming the run and linking `/compute/gemma/status/{id}` (ravanan has no status route of its own). One slot because ravanan already fans its per-chromosome steps out -- over Slurm, when that backend is selected -- so a second request would only queue behind the same scratch directory. Its gate is separate from `/compute/gemma`'s: different runners, different cache directories, so one of each at once is the overlap `/compute/rqtl` is already allowed. Verified with a stub `CCWL_CMD` that sleeps: the holder's request held the slot for the full 5 s, the same caller's second request was refused at once, and another caller's waited for the slot and then ran. While testing that, `GemmaError::Io` turned out to print "failed to run gemma-wrapper" for *any* I/O failure -- a missing `CCWL_CMD` produced "failed to run gemma-wrapper: No such file or directory", which sent me looking at the wrong binary. Spawn failures are now their own variant naming the command, and plain file I/O says "I/O error: …". * src/compute_gate.rs (ravanan, permits): `Gate::new(1)`; `permits()` exposes the slot count, and a test pins all three endpoints' counts (gemma 1, ravanan 1, rqtl RQTL_MAX_CONCURRENT). * src/routes/gemma.rs: the decorator on compute_gemma_ravanan, with `status = "/compute/gemma/status"`. * src/gemma.rs (Spawn): names the command; the four run_streaming call sites use it; unit test for both messages. * README.md (/compute/gemma/ravanan), docs/www/ai-instruction.md (the slot list is now 1 / 1 / 8), docs/www/operator-checklist.md: concurrency.
10 dayscompute: gate R/qtl at eight concurrent scans, one per callerPjotr
`POST /compute/rqtl` now takes up to `RQTL_MAX_CONCURRENT` slots (default 8, read when the gate is built) with the same one-job-per-caller rule as GEMMA: a second submission from the same token while its scan is running or queued is refused `409 compute_busy`; a submission that finds every slot taken waits for one to come free. Eight because an R/qtl scan is a single-threaded Rscript on a 96-core box; not more, because a scan on a 3 000-individual cross is minutes of CPU and the same machine serves the database. Unlike GEMMA's single slot, concurrent scans share no cache (R/qtl results are uncached), so the only cost is the CPU they take. Two flaws only appeared once there was more than one slot: - the "what is running" record was a single `Option<Active>`, so a refusal quoted whichever job had most recently taken a slot -- robw's 409 named JoM's run id. It is now a map keyed by subject, and `a_refusal_quotes_the_callers_own_run_id` pins it; - `Busy::describe` took the endpoint identifier, so the message read "your rqtl job"; the decorator takes a separate `label` ("R/qtl") from the status path. The test suite hit the rule itself: six compute tests in tests/rqtl.rs share one token, and cargo runs them on several threads, so three were refused instead of run. They now serialise on a module mutex, with a comment saying why -- the same lesson clients need, that parallelism means more tokens, not more requests from one. * src/compute_gate.rs (rqtl()): eight slots from `RQTL_MAX_CONCURRENT`; `Gate::new(permits)` was already parameterised. `running` becomes a map keyed by subject, since one record cannot name N jobs. Module doc talks about slots rather than a single slot. Tests: eight subjects run and the ninth waits, a caller still may not take two even with slots free, and a refusal quotes the caller's own run id. * src/config.rs: RQTL_MAX_CONCURRENT, default 8, fixed at first use. * gnauth-macros/src/lib.rs: `label = "…"`, so prose and status path are independent. * src/routes/rqtl.rs: the decorator, and `_token` becomes `token`, which it needs. * tests/rqtl.rs: compute tests serialised. * README.md (/compute/rqtl), docs/www/ai-instruction.md (4a), docs/www/operator-checklist.md: the numbers per endpoint, and that more parallelism means more tokens.
10 dayscompute_gate: undo the claim if the wait for the slot is cancelledPjotr
`claim()` inserts the subject into `claimants` and *then* awaits the permit, and that await is cancellable: a client disconnect while queued dropped the handler's future with the entry still in the set, so that subject was refused `compute_busy` for the life of the process. The reservation guard removes the entry when the wait is cancelled, and hands ownership to the `Claim` once the permit arrives -- the slot itself was never leaked (no permit had been taken), only the subject's name. Found by asking what could race; the new test fails without the guard (verified by disabling it) and passes with it. * src/compute_gate.rs (Reservation): the entry's owner from insertion to permit, `hand_over()` at the hand-off. (Drop for Claim): field order is load-bearing -- the maps are cleaned before `_permit` is released, so the next waiter cannot see a stale entry or a stale `running` record; and take the locks poisoned-tolerant (`lock_recover`) rather than silently skipping the cleanup. (tests): `a_cancelled_waiter_does_not_wedge_its_subject`.
10 dayscompute: one gemma job at a time, per caller, via #[compute_gate]Pjotr
`POST /compute/gemma` now takes a single slot: one job runs at a time across all callers, a second submission from the same token is refused immediately with `409 compute_busy` (whatever it would have waited for, its own job is already in the way), and a submission from another token waits for the running job and then runs. Nothing else is gated -- ravanan and /compute/rqtl are untouched. The gate is one `tokio::sync::Semaphore(1)` (fair: waiters take the permit in arrival order) plus a `Mutex<HashSet<String>>` of subjects holding the slot or queued for it, in a per-process static. The claim is moved into the spawned job task, so a client disconnect -- which does not kill the child process -- cannot hand the slot to somebody else while the CPU is still busy. The endpoint carries none of that: `#[compute_gate(gemma, run_id = params.id.as_deref())]` expands to a claim and a run, and requires the handler to take `Ctx` and `AuthToken` (every compute endpoint is token-only, and the token's sub is the key), so an endpoint that is not token-gated cannot be decorated. The policy -- the 409's status, code, wording and next link -- stays in plain Rust (`Gate::claim_or_busy`), so the gate's own tests cover it; the attribute is decoration over two calls. * src/compute_gate.rs: new. Gate, Claim (released on drop, including a panic), Busy, claim/claim_or_busy/run, and four tests, two of them on a four-thread runtime (the earlier ones ran single-threaded and so never exercised the cross-thread hand-off; one now asserts at most one task inside the slot across eight tasks). * gnauth-macros/src/lib.rs: the `compute_gate` attribute, alongside the `gnauth` markers it shares `syn`/`quote` with; arguments documented (gate, endpoint, status, run_id) for the endpoints still to be gated. * src/routes/gemma.rs: the decorator replaces 57 lines of claim and spawn plumbing in compute_gemma. * docs/www/ai-instruction.md (4a, 5): one job at a time per caller, and that a `compute_busy` 409 means poll, not resubmit.