about summary refs log tree commit diff
path: root/test/python/server_test_case.py
diff options
context:
space:
mode:
authorPjotr <wrk@bosa.lan>2026-11-09 13:50:31 +0000
committerPjotr <wrk@bosa.lan>2026-11-09 13:54:11 +0000
commit195ee5d285da98bc911813d8e578f4f17e9ba7ba (patch)
treee1e33771e55105fbea4a69143d22526563ab070c /test/python/server_test_case.py
parent216d1491435cce156f89a01684eb5c07e07db31e (diff)
downloadgenenetwork4-195ee5d285da98bc911813d8e578f4f17e9ba7ba.tar.gz
api: normalize trait values, and winsorise the tails, before mapping
GeneNetwork2's trait page offers a set of normalisations, and its mapping
page warns about outliers without ever fixing them. This adds both, under
one spelling that `/compute/rqtl`, `/compute/gemma` and
`/groups/{g}/traits/{id}/values` all share:

  normalize        none | log2 | log10 | sqrt | negate | zscore |
                   robust_zscore | rank          (GN2's names `invert` and
                                                 `qnorm` accepted)
  winsor_quantile  clip each tail at a fraction, e.g. 0.05
  winsor_fences    clip at Q1-k*IQR / Q3+k*IQR, GN2's outlier rule

They run in a fixed, reported order -- transform, then winsorise, then
standardise -- so clipping acts on the scale being analysed and the result
lands exactly on mean 0 / sd 1 rather than on the sd of clipped data. The
note gives the steps *in the order they ran* (`winsorized at Tukey fences,
then zscore`), the count clipped in each tail, and how many values sit
outside Tukey fences -- the warning GN2 gives before mapping and this API
gave nowhere.

**Checked against genenetwork.org itself.** GN2's public trait page embeds
its own reference values (`js_data.qnorm_values` / `zscore_values`, the
`data-qnorm`/`data-zscore` attributes), so there is an oracle for the two
transforms whose definition matters. On BXD trait 10010 (28 samples):
`rank` matches to 0.0005 and `zscore` to 0.0005 -- i.e. GN2's
three-decimal rounding. Getting there found a real disagreement: GN2 uses
`scipy.stats.mstats.zscore`, whose default is `ddof=0`, so its z-scores use
the **population** sd; ours used the sample sd and came out
`sqrt((n-1)/n)` low (1.8% at n = 28). We now match GN2.

Two GN2 quirks deliberately not reproduced: its "Quantile" is a rank-based
inverse-normal (Blom) transform, so it is called `rank` here; and its
`sqrt` silently adds 1 when a value is in [0,1], which shifts every effect
size -- a negative value is refused instead.

* src/normalize.rs (new): the transforms, percentile/Tukey fences, Blom
  scores over an Acklam inverse-normal, `Params` for query/body use,
  `to_json` for suggestions, and eleven unit tests -- the quantile table,
  ties sharing an average rank, missing values never imputed, GN2's own
  `find_outliers` example, and the note's wording.
* src/routes/{rqtl,gemma}.rs: the three fields, applied **after**
  alignment (over the samples that enter the fit) and before anything reads
  the values, with the note; the R/qtl permutation suggestion and GEMMA's
  R/qtl suggestion carry the pipeline.
* src/routes/groups.rs, src/suggest.rs: `?normalize=` on the values
  endpoint, and the mapping bodies in `actions` carry the same fields -- a
  caller who fetches z-scores must map those numbers, not the raw ones.
* tests/groups.rs: values are mean 0 / population sd 1, the note says so,
  every `/compute/*` suggestion in `actions` carries it, and an impossible
  pipeline is a 422 `invalid_normalization`.
* README.md, docs/www/SKILL.md: what each name means, the order, the
  population-sd parity, and that effects come out on the transformed scale.

298 tests pass; the same four as before do not.
Diffstat (limited to 'test/python/server_test_case.py')
0 files changed, 0 insertions, 0 deletions