diff options
| author | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-11-18 11:30:59 +0000 |
|---|---|---|
| committer | Pjotr Prins <pjotr.public01@thebird.nl> | 2026-11-18 11:30:59 +0000 |
| commit | f026a7b808fd2d61e64d4d53c540fd66fffcff81 (patch) | |
| tree | 0b5ca159775cba3a6d7a847d1c03cc6e22a061f7 /src/mathfunc.cpp | |
| parent | cebca3b5cc041731df65bb5008fd3e7d5202c299 (diff) | |
| download | pangemma-f026a7b808fd2d61e64d4d53c540fd66fffcff81.tar.gz | |
mdb: apply the marker filters, count missing, and give K the same marker set
An lmdb scan and a BIMBAM scan of the same data were not answering the same
question. Measured on HLC (358454 markers, 23 chromosomes, gemma-wrapper's
LOCO recipe), before this commit:
-maf 0.05 gemma 20246 markers on chr1, pangemma 27940 (every marker in the
store): mdb_calc_gwa computed maf and put it in MarkerInfo but
nothing ever compared it with maf_level, and -miss was ignored too
af/n_miss nan / 0 for every marker carrying a missing call, because
compute_maf was called with n_miss hard-wired to 0 ("count NAs:
FIXME") and summed NaNs into the frequency
Gb stores 255 is the missing sentinel; read as 255/127.0 it became 2.0079, a
valid-looking genotype that silently poisoned af and the test
K mdb_calc_kin had no missing handling at all (its mask marked every
individual, so the imputation branch never fired) and applied no
filters, so HLC's K was built from 330514 markers where GEMMA uses
273349 -- GEMMA's K reader skips whatever its genotype reader
rejected (indicator_snp[i] == 0) -- and every p-value moved
log line mdb_calc_kin never reported the count, so "number of analyzed
SNPs/var" printed an uninitialised ns_test (140048413575759)
All four readers of gemma_io.cpp apply the same five filters in a fixed order;
the lmdb path now does the same, in the same order, with GEMMA's own allele
bins: -miss, -maf (maf_level == -1 disables), monomorphic (every observed
genotype equal), -hwe, and -r2 against the covariates (missing imputed with
maf*2 as the readers do, W'W inverted once per scan). Missing calls arrive as
NaN in both formats, compute_maf counts what it sees, and the K path takes the
same filtered marker set and imputes the observed mean exactly as
gemma_io.cpp's ReadFile_geno does for K.
Verified, HLC LOCO on the slurm cluster (23 tasks, 1 CPU each, all 23 RUNNING
together), pangemma lmdb (Gb, missing kept) against gemma 0.98.6 on BIMBAM:
same row count on all 23 chromosomes (260251 rows), same top SNP on 23/23 (was
12/23), and no differing numeric cell in beta, se, l_mle or p_lrt (was 260251
rows differing with l_mle pinned at the 1e5 bound). The BIMBAM route is
unchanged: 23/23 and 0 differing cells, as before. Cost of the filtering:
job time for the lmdb route fell from 179 s to 141 s for the 23 chromosomes
(gemma BIMBAM: 958 s), because 30 percent of the markers are no longer
analysed.
On BXD.8 (20453 markers, 198 individuals, 1460 missing calls) the Gf store now
reproduces gemma's output column for column -- af and n_miss included -- with
the only differences being 33 `ps` values, where GEMMA's annotation reader
truncates R's "1.76e+08" to 1 and the store has the rounded position.
Diffstat (limited to 'src/mathfunc.cpp')
| -rw-r--r-- | src/mathfunc.cpp | 23 |
1 files changed, 18 insertions, 5 deletions
diff --git a/src/mathfunc.cpp b/src/mathfunc.cpp index c82872e..ad29982 100644 --- a/src/mathfunc.cpp +++ b/src/mathfunc.cpp @@ -667,14 +667,27 @@ std::tuple<size_t, size_t, size_t> compute_ratio(size_t ni_total, const gsl_vect } double compute_maf(size_t ni_total, size_t ni_test, size_t n_miss, const double *gs, const vector<int> &indicator) { - double maf = 0.0; - + // Missing genotypes arrive here as NaN (Gf stores) or as the 255 sentinel that + // the Gb decoder turns into NaN, so they must be left out of the sum; the + // denominator is the number of *observed* genotypes, which is what GEMMA uses: + // maf = sum(geno) / (2 * (ni_test - n_miss)) + // It used to add NaNs into the sum, so af came out NaN for any marker with a + // missing call, and it trusted the caller's n_miss, which the LMDB path passed + // as 0 (see mdb_calc_gwa). + double total = 0.0; + size_t observed = 0; for (size_t i = 0; i < ni_total; ++i) { // read genotypes if (indicator[i]) { double geno = gs[i]; - maf += geno; // 0..2 range + if (!std::isfinite(geno)) + continue; + total += geno; // 0..2 range + observed++; } } - maf /= 2.0 * (double)(ni_test - n_miss); // Assumption is that geno value is between 0 and 2. FIXME - return maf; + (void)ni_test; + (void)n_miss; // kept for the signature: `observed' is the same number, counted here + if (observed == 0) + return 0.0; + return total / (2.0 * (double)observed); } |
