about summary refs log tree commit diff
path: root/src/gemma_io.cpp
diff options
context:
space:
mode:
authorPjotr Prins <pjotr.public01@thebird.nl>2026-11-17 11:25:45 +0000
committerPjotr Prins <pjotr.public01@thebird.nl>2026-11-17 11:25:45 +0000
commitcebca3b5cc041731df65bb5008fd3e7d5202c299 (patch)
tree845fe569565f033b2a57b763b1817cb2e0c3355e /src/gemma_io.cpp
parent314dbfd4b834cd7ad30e808587cd1ec3dc30feec (diff)
downloadpangemma-cebca3b5cc041731df65bb5008fd3e7d5202c299.tar.gz
param: count analyzed individuals properly, which revives the -r2 filter
ProcessInclusionIndicators summed ni_test with a range-for over the values
but used each value as an index:

    for (auto &i : indicator_idv) {
      ni_test += indicator_idv[i];
    }

so ni_test came out as the number of elements times indicator_idv[0] or
indicator_idv[1] -- 198 on the BXD fixture, where 67 individuals are
analyzed. Everything sized from it there was sized wrong: the covariate
matrix W was allocated with 198 rows while genotypes were written only at
the 67 analyzed positions, leaving 131 entries of uninitialised memory.

That memory is what killed the -r2 covariate filter. The filter computes
v_x = genotype . genotype over the whole vector, so a single garbage entry
(a value of 1.26e248 in the run that was traced) makes v_x infinite and the
ratio v_w/v_x zero, and no SNP is ever dropped. Measured on the BXD
fixture with two covariates: GEMMA 0.98.6 drops three SNPs and reports
7317; pangemma reported 7320 at every -r2 setting, including -r2 0.0001,
which should have dropped nearly everything. Those three SNPs, at lines
615-617 of the genotype file, were the entire remaining difference between
the two builds' association output, and they are not duplicates: one of
them is unique among all 7320 markers.

With the count fixed:

  -r2 0.9999 (default)   pangemma 7317   gemma 7317
  -r2 0.5                pangemma 3029   gemma 3029

and the BXD association output is identical to GEMMA's for all 7317 rows,
cell for cell, allele columns included. The GRM is unchanged (max |diff| 0
against GEMMA) and the lmdb LOCO path is unchanged (chr 1: 636 identical
rows).
Diffstat (limited to 'src/gemma_io.cpp')
0 files changed, 0 insertions, 0 deletions