about summary refs log tree commit diff
path: root/src/normalize.rs
AgeCommit message (Collapse)Author
8 daysapi: normalize trait values, and winsorise the tails, before mappingPjotr
GeneNetwork2's trait page offers a set of normalisations, and its mapping page warns about outliers without ever fixing them. This adds both, under one spelling that `/compute/rqtl`, `/compute/gemma` and `/groups/{g}/traits/{id}/values` all share: normalize none | log2 | log10 | sqrt | negate | zscore | robust_zscore | rank (GN2's names `invert` and `qnorm` accepted) winsor_quantile clip each tail at a fraction, e.g. 0.05 winsor_fences clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule They run in a fixed, reported order -- transform, then winsorise, then standardise -- so clipping acts on the scale being analysed and the result lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The note gives the steps *in the order they ran* (`winsorized at Tukey fences, then zscore`), the count clipped in each tail, and how many values sit outside Tukey fences -- the warning GN2 gives before mapping and this API gave nowhere. **Checked against genenetwork.org itself.** GN2's public trait page embeds its own reference values (`js_data.qnorm_values` / `zscore_values`, the `data-qnorm`/`data-zscore` attributes), so there is an oracle for the two transforms whose definition matters. On BXD trait 10010 (28 samples): `rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's three-decimal rounding. Getting there found a real disagreement: GN2 uses `scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use the **population** sd; ours used the sample sd and came out `sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2. Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based inverse-normal (Blom) transform, so it is called `rank` here; and its `sqrt` silently adds 1 when a value is in [0,1], which shifts every effect size -- a negative value is refused instead. * src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom scores over an Acklam inverse-normal, `Params` for query/body use, `to_json` for suggestions, and eleven unit tests -- the quantile table, ties sharing an average rank, missing values never imputed, GN2's own `find_outliers` example, and the note's wording. * src/routes/{rqtl,gemma}.rs: the three fields, applied **after** alignment (over the samples that enter the fit) and before anything reads the values, with the note; the R/qtl permutation suggestion and GEMMA's R/qtl suggestion carry the pipeline. * src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values endpoint, and the mapping bodies in `actions` carry the same fields -- a caller who fetches z-scores must map those numbers, not the raw ones. * tests/groups.rs: values are mean 0 / population sd 1, the note says so, every `/compute/*` suggestion in `actions` carries it, and an impossible pipeline is a 422 `invalid_normalization`. * README.md, docs/www/SKILL.md: what each name means, the order, the population-sd parity, and that effects come out on the transformed scale. 298 tests pass; the same four as before do not.