| Age | Commit message (Collapse) | Author |
|
Tab-delimited, header names the individuals from column 4, then marker,
chr, pos (Mb). Named $USER-GENONAME, stored as text in the owner's own
SQLite file -- nothing goes near MariaDB -- and written out on demand as
a GN .geno carrying the base's @-metadata, so the R scripts read exactly
what was uploaded.
An upload updates a population, it does not invent one, and the checks
say so:
* the base version must exist (resolved through the same
genotypes::resolve_named the rest of the API uses);
* at most 50 individuals the base does not have, and at least one that
it does -- no overlap is a different population, not an update;
* at most 200 000 markers, and the refusal says the sysadmin has to
organize a file that size, because this is a request path, not an
ingest;
* the values must be the same kind as the base's: letters, whole
numbers or probabilities. A letters panel holding 0/1 would be read
as the wrong thing silently, so it is refused naming both kinds.
GET /uploads/genotypes lists them, /{name} its metadata (overlap, new
individuals, kind, digest), /{name}/markers pages the marker table, and
/{name}/geno writes the file and returns the path for `genotype_file` on
the compute endpoints. DELETE removes one.
scripts/example-genotype-upload.py turns any .geno into a valid update --
a subset of markers, a few individuals dropped and added -- and
tests/genoupload.rs builds its own from the deployment's BXD.4 for the
same reason: upload, page the markers, write the .geno and check the
layout, delete, then break each rule (no overlap, numbers in a letters
panel, commas) and check the refusal says which.
Unit tests cover the three value kinds, missing cells, ragged rows, a
non-numeric position, the 50-new-individual limit, and the .geno layout
(Chr first, then the marker -- GN's order, not the upload's).
|
|
The cascade fix cleans files written from now on, but a file written
before it still carries rows whose basket was deleted without them --
the production file did, and the reused rowid made the next create fail
with UNIQUE constraint failed. Two indexed deletes on open make such a
file heal instead of failing forever, and a test simulates the old bug
by deleting a basket row by hand and then creating another.
|
|
The other half of changing data: bringing new columns in. A matrix is a
delimited table -- a header row, then one row per individual, the first
column naming them -- stored in the uploader's own SQLite file and checked
against an existing genotype version's panel before it is accepted.
* POST /matrices {name, genotype_version, contents, kinds?}: parse the
table (tab, comma or semicolon, guessed from the header), then check
it against that version's individuals. The count must match and every
name must be in the panel; 92 of 93, or a name the panel does not
have, is a 422 that says which. A mapping that silently drops
individuals is worse than an error.
* kinds marks columns that are covariates; everything else is a trait.
* Each column becomes a basket named <matrix>.<column> -- a dot, not a
slash, because a basket name has to survive a URL path segment -- so
a column maps on any engine with {"basket": "my-traits.bw"} and
inherits the revision history and digest without a line of new
mapping code.
* GET /matrices lists them, GET /matrices/{name} their columns and
basket names, GET /matrices/{name}/values the table back (parsed
values, null where missing), DELETE removes the matrix and its
columns.
Two bugs this found in the basket store, both real:
* SQLite has foreign keys OFF by default, so the ON DELETE CASCADE
never fired: deleting a basket left its rows behind, rowids were
reused, and the next create hit "UNIQUE constraint failed:
basket_version.basket_id, revision". The pragma is now set, and
deletion removes the dependants explicitly, because files written
before it have orphans.
* The first attempt named a column's basket "<matrix>/<column>", which
the basket-name rule (rightly) refuses. The dot keeps both rules.
scripts/example-matrix.py writes a table from a genotype file's own header
with deterministic values, so the fixture passes the panel check by
construction -- tests/matrices.rs builds its own for the same reason and
walks the loop: upload, read back, map a column with R/qtl2, delete, plus
the three refusals (too few individuals, a name not in the panel, an
unknown version).
|
|
Three things the first cut was missing, and both were asked for.
* An automatic version history. Revision 1 is the column as first
stored and each PATCH adds one -- with its own digest and Unix
timestamp -- and earlier revisions are kept. GET
/baskets/{name}/revisions is the history, ?revision=N reads an old
column, and a mapping can pin one: {"basket": "wbc-edit@2"}. That is
the point of a digest: a result quoted yesterday can still be
reproduced after the column has moved on.
* Files named for the user, not a hash of the subject:
BASKET_DIR/<user>.sqlite (0600, WAL). Tokens rotate, subjects do not,
so a new token finds the same baskets -- and an operator can see
whose baskets a file holds. A subject that is not filename-safe is
sanitised and disambiguated with a short digest, and a pre-naming
hash file is renamed on first use rather than orphaned.
* /compute/rqtl and /compute/gemma and /compute/fitqtl take `basket`
too, so all four engines share one meaning: a basket resolves to the
same (name, value) pairs a client would have posted, and every reply
records the revision and digest it ran.
The schema upgrade is a real one, not a reset: an older file's single
column becomes revision 1 of the history, cells copied across with the
digest and timestamp they already had. Tested by building the old schema
by hand and opening it (an_older_file_is_upgraded_with_its_column_intact),
which is also how I found that the migration was recreating a table the
base schema had already made.
Verified on all four engines, head and pinned: rqtl 3811 rows, gemma
3809, qtl2 3730, fitqtl 3 markers -- each reply carrying "Basket '...'
revision N (digest ...)".
|
|
The next phase is changing data, and the constraint is that MariaDB is
never written to. A basket is the storage half of that: take a trait's
values, edit some of them (or build a covariate column that does not
exist), keep the column under a name, and map that instead.
* POST /baskets stores a complete column; GET /baskets lists yours;
GET /baskets/{name} its metadata; GET /baskets/{name}/values the
cells; PATCH /baskets/{name} replaces the column with a new revision;
DELETE removes it.
* POST /compute/qtl2 takes "basket": "<name>" in place of `samples`,
and both at once is a 422 rather than a silent choice. The mapping
code below it is untouched -- a basket resolves to the same
(name, value) pairs a client would have posted.
* One SQLite file per user: BASKET_DIR/<sha256(token sub)>.sqlite, 0600,
WAL, busy_timeout. `GET /baskets` only ever lists the caller's own;
the file name is a hash, so a directory listing does not name users.
* The full column is stored, not a diff -- the caller has the original
beside it, and a mapping needs one complete column.
* Text with a parsed copy beside it: raw (what was typed), value (the
number or NULL), status (ok/missing/bad). "NA" and 0 stay different
claims, and a column with something that is neither a number nor a
missing marker (NA, x, -, ., ?) is refused with the samples named --
silently dropping it would change a trait without saying so.
* A revision and a digest: an edit bumps the revision and changes the
digest, and a mapping reply records both ("Basket 'wbc-edit'
revision 2 (digest 9559a70e...)"), so two runs on one name cannot
look alike. A result cache must key on the digest, not the name.
The loop, end to end on BXD trait 10001: 34 values posted as a basket,
one bumped by 5 and one marked NA, revision 2 mapped with /compute/qtl2
-- 3730 rows, peak chr8 112.9 Mb, and the reply's note naming the
revision and digest.
tests/baskets.rs walks that loop (create, list, values, patch, map by
name, delete) and the refusals: a non-number named in the error, a name
that cannot become a path, an unknown basket, and basket-plus-samples.
Unit tests cover parsing (NA/x/-/. vs NaN/inf vs "twelve"), the digest
changing on edit, per-user isolation, and name validation.
rqtl/gemma take the same 10-line change next -- their `samples` shapes
differ, so they are a separate commit rather than a rushed one.
|