about summary refs log tree commit diff
path: root/src/baskets.rs
AgeCommit message (Collapse)Author
7 daysuploads: a genotype matrix that updates an existing populationPjotr
Tab-delimited, header names the individuals from column 4, then marker, chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own SQLite file -- nothing goes near MariaDB -- and written out on demand as a GN .geno carrying the base's @-metadata, so the R scripts read exactly what was uploaded. An upload updates a population, it does not invent one, and the checks say so: * the base version must exist (resolved through the same genotypes::resolve_named the rest of the API uses); * at most 50 individuals the base does not have, and at least one that it does -- no overlap is a different population, not an update; * at most 200 000 markers, and the refusal says the sysadmin has to organize a file that size, because this is a request path, not an ingest; * the values must be the same kind as the base's: letters, whole numbers or probabilities. A letters panel holding 0/1 would be read as the wrong thing silently, so it is refused naming both kinds. GET /uploads/genotypes lists them, /{name} its metadata (overlap, new individuals, kind, digest), /{name}/markers pages the marker table, and /{name}/geno writes the file and returns the path for `genotype_file` on the compute endpoints. DELETE removes one. scripts/example-genotype-upload.py turns any .geno into a valid update -- a subset of markers, a few individuals dropped and added -- and tests/genoupload.rs builds its own from the deployment's BXD.4 for the same reason: upload, page the markers, write the .geno and check the layout, delete, then break each rule (no overlap, numbers in a letters panel, commas) and check the refusal says which. Unit tests cover the three value kinds, missing cells, ragged rows, a non-numeric position, the 50-new-individual limit, and the .geno layout (Chr first, then the marker -- GN's order, not the upload's).
8 daysbaskets: clear orphan rows on open, so an affected file healsPjotr
The cascade fix cleans files written from now on, but a file written before it still carries rows whose basket was deleted without them -- the production file did, and the reused rowid made the next create fail with UNIQUE constraint failed. Two indexed deletes on open make such a file heal instead of failing forever, and a test simulates the old bug by deleting a basket row by hand and then creating another.
8 daysmatrices: upload new traits and covariates as one named tablePjotr
The other half of changing data: bringing new columns in. A matrix is a delimited table -- a header row, then one row per individual, the first column naming them -- stored in the uploader's own SQLite file and checked against an existing genotype version's panel before it is accepted. * POST /matrices {name, genotype_version, contents, kinds?}: parse the table (tab, comma or semicolon, guessed from the header), then check it against that version's individuals. The count must match and every name must be in the panel; 92 of 93, or a name the panel does not have, is a 422 that says which. A mapping that silently drops individuals is worse than an error. * kinds marks columns that are covariates; everything else is a trait. * Each column becomes a basket named <matrix>.<column> -- a dot, not a slash, because a basket name has to survive a URL path segment -- so a column maps on any engine with {"basket": "my-traits.bw"} and inherits the revision history and digest without a line of new mapping code. * GET /matrices lists them, GET /matrices/{name} their columns and basket names, GET /matrices/{name}/values the table back (parsed values, null where missing), DELETE removes the matrix and its columns. Two bugs this found in the basket store, both real: * SQLite has foreign keys OFF by default, so the ON DELETE CASCADE never fired: deleting a basket left its rows behind, rowids were reused, and the next create hit "UNIQUE constraint failed: basket_version.basket_id, revision". The pragma is now set, and deletion removes the dependants explicitly, because files written before it have orphans. * The first attempt named a column's basket "<matrix>/<column>", which the basket-name rule (rightly) refuses. The dot keeps both rules. scripts/example-matrix.py writes a table from a genotype file's own header with deterministic values, so the fixture passes the panel check by construction -- tests/matrices.rs builds its own for the same reason and walks the loop: upload, read back, map a column with R/qtl2, delete, plus the three refusals (too few individuals, a name not in the panel, an unknown version).
8 daysbaskets: every revision kept, files named for the user, all four enginesPjotr
Three things the first cut was missing, and both were asked for. * An automatic version history. Revision 1 is the column as first stored and each PATCH adds one -- with its own digest and Unix timestamp -- and earlier revisions are kept. GET /baskets/{name}/revisions is the history, ?revision=N reads an old column, and a mapping can pin one: {"basket": "wbc-edit@2"}. That is the point of a digest: a result quoted yesterday can still be reproduced after the column has moved on. * Files named for the user, not a hash of the subject: BASKET_DIR/<user>.sqlite (0600, WAL). Tokens rotate, subjects do not, so a new token finds the same baskets -- and an operator can see whose baskets a file holds. A subject that is not filename-safe is sanitised and disambiguated with a short digest, and a pre-naming hash file is renamed on first use rather than orphaned. * /compute/rqtl and /compute/gemma and /compute/fitqtl take `basket` too, so all four engines share one meaning: a basket resolves to the same (name, value) pairs a client would have posted, and every reply records the revision and digest it ran. The schema upgrade is a real one, not a reset: an older file's single column becomes revision 1 of the history, cells copied across with the digest and timestamp they already had. Tested by building the old schema by hand and opening it (an_older_file_is_upgraded_with_its_column_intact), which is also how I found that the migration was recreating a table the base schema had already made. Verified on all four engines, head and pinned: rqtl 3811 rows, gemma 3809, qtl2 3730, fitqtl 3 markers -- each reply carrying "Basket '...' revision N (digest ...)".
8 daysbaskets: a user's own column of values, in their own SQLite filePjotr
The next phase is changing data, and the constraint is that MariaDB is never written to. A basket is the storage half of that: take a trait's values, edit some of them (or build a covariate column that does not exist), keep the column under a name, and map that instead. * POST /baskets stores a complete column; GET /baskets lists yours; GET /baskets/{name} its metadata; GET /baskets/{name}/values the cells; PATCH /baskets/{name} replaces the column with a new revision; DELETE removes it. * POST /compute/qtl2 takes "basket": "<name>" in place of `samples`, and both at once is a 422 rather than a silent choice. The mapping code below it is untouched -- a basket resolves to the same (name, value) pairs a client would have posted. * One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600, WAL, busy_timeout. `GET /baskets` only ever lists the caller's own; the file name is a hash, so a directory listing does not name users. * The full column is stored, not a diff -- the caller has the original beside it, and a mapping needs one complete column. * Text with a parsed copy beside it: raw (what was typed), value (the number or NULL), status (ok/missing/bad). "NA" and 0 stay different claims, and a column with something that is neither a number nor a missing marker (NA, x, -, ., ?) is refused with the samples named -- silently dropping it would change a trait without saying so. * A revision and a digest: an edit bumps the revision and changes the digest, and a mapping reply records both ("Basket 'wbc-edit' revision 2 (digest 9559a70e...)"), so two runs on one name cannot look alike. A result cache must key on the digest, not the name. The loop, end to end on BXD trait 10001: 34 values posted as a basket, one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2 -- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the revision and digest. tests/baskets.rs walks that loop (create, list, values, patch, map by name, delete) and the refusals: a non-number named in the error, a name that cannot become a path, an unknown basket, and basket-plus-samples. Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest changing on edit, per-user isolation, and name validation. rqtl/gemma take the same 10-line change next -- their `samples` shapes differ, so they are a separate commit rather than a rushed one.