about summary refs log tree commit diff
path: root/src/covariates.rs
AgeCommit message (Collapse)Author
9 dayscovariates: refuse a mostly-numeric attribute, and teach the skill to use themPjotr
Fitting `SeqCvge` as a covariate came back as `502 gemma_failed` with a message about LU decomposition: its values are depths (`40`, `41`) with a few two-library entries (`36+39L`), so the attribute was coded as **twenty categories** and GEMMA died on the resulting dummies. The failure named GEMMA, which is the wrong place to look. `code_case_attribute` now refuses that shape at the covariate step, naming the values and what to send instead: 422 invalid_covariate: case attribute 'SeqCvge': the attribute mixes numbers with number-like text (14 of 19 values look numeric, but e.g. '36+30L' does not), so coding it as categories would fit a quantity as a factor -- and GEMMA fails on the result. Pass an explicit `map` giving each of those a number (there are 5 such value(s): ...), or send the numbers inline. The rule is deliberately narrow: an odd level with *no* digits (`1, 2, unknown`) is still a level and still coded, so nothing that worked stops working; only dirty numbers are caught. Also, the skill's covariate coverage -- which the review above found thin: * when a covariate is worth trying (metadata that could confound: sex, age, epoch, batch, sequencing depth), and the covariate step in the session outline; * read `.data.descriptions` before choosing one -- they are the only place that says `SeqCvge` is sequencing depth and `Epoch` a family subgroup -- and the endpoint table now says so; * read the covariate line in `notes` for the coding and the complete-case n, and report that n beside any LOD/p change; plus `Cyro`/`Cryo` being two levels in BXD; * the categorical-vs-numeric rule measured rather than asserted: `RRID` refused (82 distinct), `Status`/`Epoch` accepted, and the compound-value trap written up with the `map` workaround; * GEMMA's dummy columns, the column it drops when a level has no fitted individual, and that R/qtl's note has no such line; missing values dropping individuals; the 8-covariate cap and the `group` requirement. * src/covariates.rs: the refusal, with a unit test for both sides of it; * tests/gemma.rs: `SeqCvge` is a 422 naming `map`, not a GEMMA failure; * docs/www/SKILL.md: the covariate section, the session line, and the case-attributes rows in the endpoint table. 300 tests pass; the same four as before do not.
10 daysapi: describe case attributes, and gate large computations onlyPjotr
Two things that came out of reading a real response rather than a test. **Say what a case attribute means.** `GET /groups/{group}/case-attributes` returned names and values, and the per-trait attribute endpoint returned values, so a caller had to guess that `SeqCvge` is a sequencing depth. MariaDB has had the answer all along in `CaseAttribute.Description`: SeqCvge Depth of DNA sequencing for 1 male of each strain in 2016 Status Live= Available at JAX, Cryo=Cryopreserved only, Extinct Epoch BXD family subgroups. Each number with common parents. ... Sex F= female, M= male Both endpoints now carry it -- a `descriptions` map on the group listing, a `description` field on the per-trait one -- and the covariate notes quote it, so `/compute/gemma` answers "Covariate SeqCvge (Depth of DNA sequencing for 1 male of each strain in 2016): categorical ..." instead of leaving the name to be interpreted. It matters most for covariates: `SeqCvge` is a property of the *genotype* data, its values include compound ones (`40+31L`, two sequencing runs), and the server codes those as 20-odd levels rather than as a depth -- a covariate fitted for the wrong reason, which the description is the first hint against. **Gate large computations, not every computation.** The gate was applied to every job, so a 93-strain test request could be refused because somebody else's 6 000-individual run held the slot -- which is how the test suite started serialising itself behind a mutex, a workaround for the wrong problem. What the gate is for is *large* work: a BXD panel of ~200 finishes in seconds and two at once cost nothing, while UM-HET3's 6 382 write 207 MB of kinship per chromosome. It now starts at `LARGE_PANEL` (1 000 individuals): below that no slot is taken, nothing waits and nothing is refused; above it the old policy holds unchanged (1 slot for gemma, 1 for ravanan, `RQTL_MAX_CONCURRENT` for rqtl, one large job per caller, 409 naming the run in the way). The size is not known when the request arrives -- the panel is resolved server-side -- so the decorator now makes a *light* claim and the handler promotes it (`__gn4_claim.claim_slot_or_busy(individuals, rid)`) at the point where it knows, then hands it to `Gate::run` around the call that computes, so the slot still outlives a client disconnect. A light claim registers nothing and its drop removes nothing: only a held claim may touch the claimant map. * src/compute_gate.rs: `LARGE_PANEL`/`is_large`, `Gate::claim_light`, `Claim::claim_slot_or_busy`/`is_held`, `busy_error` (the 409 policy, moved out of `claim_or_busy`), and `Gate::run` unchanged in spirit. * gnauth-macros: binds the light claim; the body keeps one gate call. * src/routes/{gemma,rqtl}.rs: promotion after the panel/sample count is known, and the compute call wrapped in `run`; error mapping moved inside the job so the future returns `ApiError`. * src/rqtl.rs: `Sample` is `Clone` (the job owns its copy). * tests: the gemma suite loses its lock entirely (the fixtures and BXD.4 are under the threshold); rqtl keeps its lock only for the machine's sake -- concurrent `Rscript` scans would compete for cores and scratch, not for a slot. Four new gate tests pin the threshold, that a small job never takes the slot, that one caller cannot hold two large jobs, and that another caller waits rather than fails. * README.md, docs/www/ai-instruction.md, docs/www/operator-checklist.md: what the threshold is and why it is 1 000. Measured on :3001 after the change: two concurrent small-panel GEMMA requests from one token both answer 200 (one was 409 before), while two 6 382-individual requests answer 502/409 -- the second gated with the usual `compute_busy` naming the first. 277 tests pass; the same four as before do not.
10 dayscompute: say what each covariate note is actually aboutPjotr
The shared covariate note promised "5 dummy column(s) named Status.<level>" on both compute endpoints, but only one of them gets dummies: GEMMA is handed k-1 columns and R/qtl's wrapper is handed the single level column and expands it itself. On `/compute/rqtl` the sentence described something that never appears -- caught by reading a live response on production, where the note said "5 dummy column(s) named Status." one line above the Rscript command line that passes no such thing. The note is now engine-neutral -- `... 90 of 93 samples have a value, 34 with both the trait and the covariate; 6 levels` -- and GEMMA says what it was handed in its own note, which is the part a reader comparing two runs wants: `GEMMA fits 4 covariate column(s): Status.Live, Status.UTHSC, Status.UTHSC2, ...`. The unit test asserts the level count, and the end-to-end test asserts the columns note names `Status.` columns twice or more, so a covariate silently fitted as one ordinal number fails it. * src/covariates.rs (resolve), src/gemma.rs (compute), tests/gemma.rs, docs/www/ai-instruction.md (the example note).
10 dayscompute: fit covariates in GEMMA, and share them with R/qtlPjotr
`POST /compute/gemma` now takes the same `covariates` list `/compute/rqtl` does -- `{"case_attribute": "Status"}`, an explicit `map`, or per-sample numbers -- and fits it with `-c` on the association step only. That needed a covariate file of GEMMA's own, which is not R/qtl's: no header, no sample names, positional over the genotype panel, numbers only. GEMMA has no factors, so a categorical covariate is dummy-expanded here into k-1 columns named after their levels (`Status.Live`, the first level being the reference), where R/qtl's wrapper does that expansion for itself. The spec shape, the alignment onto the panel and the text coding are now one module, `src/covariates.rs`, called by both routes; `/compute/rqtl` behaves as before except that its note also reports the complete cases (`... 34 with both the trait and the covariate`), which README and the instructions had been promising all along. A missing covariate value is written `NA`, which GEMMA treats exactly as it treats a missing phenotype -- that individual leaves the fit (measured: one fewer "number of analyzed individuals"). Deliberately not GeneNetwork2's `-9`, which GEMMA fits as the value minus nine. Found on live data, not in the fixtures: a level that exists in the panel but that none of the *fitted* individuals carries becomes an all-zero dummy column, and GEMMA stops with `LUDecomp ... input domain error` rather than dropping the term the way R would. Columns that do not vary among the individuals with both a trait and a covariate are now left out, and the response names them (`Status.Cyro`, the `Cyro`/`Cryo` typo in BXD); a covariate left with no columns runs without one, with a note saying so. Covariates reach only the GWA step, never `-gk`, and gemma-wrapper hashes its file arguments, so a different covariate set is a different cache entry while the kinship already computed is still reused. Covariates need a `group`: individuals are aligned by name, which an explicit `genotype_file`/`snps_file` request does not carry (`422 covariate_unavailable`), and an attribute the group does not define lists the ones it has. * src/covariates.rs (new): `Spec`/`NamedValue` (the request shape, now `CovariateSpec`/`CovariateValue` in the OpenAPI document), `Level`, `Covariate`, `code_case_attribute`, `resolve`, `write_covarstruct_file` (moved out of src/rqtl.rs) and `write_gemma_covariates_file`. * src/gemma.rs (gwa_args, compute, compute_gwa): the optional covariate file, `-c` after `-a` and before `-lmm`, and the notes about dropped individuals and dropped columns. * src/routes/gemma.rs: the `covariates` field, resolved after the trait values so the note can count complete cases; `RqtlCovariate` replaced by the shared spec. * tests: six new -- the GEMMA file (numerical with `NA`; two and three levels; inline codes; a dropped degenerate column) and an end-to-end `compute_gemma_fits_a_case_attribute_as_a_covariate` on BXD/BXD.4 with `Status`, which also pins that `-c` is in the GWA command and not in the kinship one, and that the covariate changes the fit rather than reusing the plain run's cache. The gemma tests now take a lock: the gate answers a second job from the same caller with 409 instead of queueing it. * README.md (/compute/gemma covariates rewritten for GEMMA, /compute/rqtl points at the shared shape and explains `x` vs `NA`), docs/www/ ai-instruction.md (both endpoints; what differs in `notes`). Also in the instructions: `/version` reporting a `git` value you have not seen before now says explicitly to pull the repository and re-read `docs/www/ai-instruction.md` before the next call. This deployment is updated in place and a stale page does not announce itself -- the API accepts the request without the new option, or answers with fields the old text never mentions. 272 tests pass; the four that do not (benchmark, no_db, and two search parity/proxy replays) fail identically at the parent commit.