From f026a7b808fd2d61e64d4d53c540fd66fffcff81 Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Wed, 18 Nov 2026 11:30:59 +0000 Subject: mdb: apply the marker filters, count missing, and give K the same marker set An lmdb scan and a BIMBAM scan of the same data were not answering the same question. Measured on HLC (358454 markers, 23 chromosomes, gemma-wrapper's LOCO recipe), before this commit: -maf 0.05 gemma 20246 markers on chr1, pangemma 27940 (every marker in the store): mdb_calc_gwa computed maf and put it in MarkerInfo but nothing ever compared it with maf_level, and -miss was ignored too af/n_miss nan / 0 for every marker carrying a missing call, because compute_maf was called with n_miss hard-wired to 0 ("count NAs: FIXME") and summed NaNs into the frequency Gb stores 255 is the missing sentinel; read as 255/127.0 it became 2.0079, a valid-looking genotype that silently poisoned af and the test K mdb_calc_kin had no missing handling at all (its mask marked every individual, so the imputation branch never fired) and applied no filters, so HLC's K was built from 330514 markers where GEMMA uses 273349 -- GEMMA's K reader skips whatever its genotype reader rejected (indicator_snp[i] == 0) -- and every p-value moved log line mdb_calc_kin never reported the count, so "number of analyzed SNPs/var" printed an uninitialised ns_test (140048413575759) All four readers of gemma_io.cpp apply the same five filters in a fixed order; the lmdb path now does the same, in the same order, with GEMMA's own allele bins: -miss, -maf (maf_level == -1 disables), monomorphic (every observed genotype equal), -hwe, and -r2 against the covariates (missing imputed with maf*2 as the readers do, W'W inverted once per scan). Missing calls arrive as NaN in both formats, compute_maf counts what it sees, and the K path takes the same filtered marker set and imputes the observed mean exactly as gemma_io.cpp's ReadFile_geno does for K. Verified, HLC LOCO on the slurm cluster (23 tasks, 1 CPU each, all 23 RUNNING together), pangemma lmdb (Gb, missing kept) against gemma 0.98.6 on BIMBAM: same row count on all 23 chromosomes (260251 rows), same top SNP on 23/23 (was 12/23), and no differing numeric cell in beta, se, l_mle or p_lrt (was 260251 rows differing with l_mle pinned at the 1e5 bound). The BIMBAM route is unchanged: 23/23 and 0 differing cells, as before. Cost of the filtering: job time for the lmdb route fell from 179 s to 141 s for the 23 chromosomes (gemma BIMBAM: 958 s), because 30 percent of the markers are no longer analysed. On BXD.8 (20453 markers, 198 individuals, 1460 missing calls) the Gf store now reproduces gemma's output column for column -- af and n_miss included -- with the only differences being 33 `ps` values, where GEMMA's annotation reader truncates R's "1.76e+08" to 1 and the store has the rounded position. --- src/mathfunc.cpp | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) (limited to 'src/mathfunc.cpp') diff --git a/src/mathfunc.cpp b/src/mathfunc.cpp index c82872e..ad29982 100644 --- a/src/mathfunc.cpp +++ b/src/mathfunc.cpp @@ -667,14 +667,27 @@ std::tuple compute_ratio(size_t ni_total, const gsl_vect } double compute_maf(size_t ni_total, size_t ni_test, size_t n_miss, const double *gs, const vector &indicator) { - double maf = 0.0; - + // Missing genotypes arrive here as NaN (Gf stores) or as the 255 sentinel that + // the Gb decoder turns into NaN, so they must be left out of the sum; the + // denominator is the number of *observed* genotypes, which is what GEMMA uses: + // maf = sum(geno) / (2 * (ni_test - n_miss)) + // It used to add NaNs into the sum, so af came out NaN for any marker with a + // missing call, and it trusted the caller's n_miss, which the LMDB path passed + // as 0 (see mdb_calc_gwa). + double total = 0.0; + size_t observed = 0; for (size_t i = 0; i < ni_total; ++i) { // read genotypes if (indicator[i]) { double geno = gs[i]; - maf += geno; // 0..2 range + if (!std::isfinite(geno)) + continue; + total += geno; // 0..2 range + observed++; } } - maf /= 2.0 * (double)(ni_test - n_miss); // Assumption is that geno value is between 0 and 2. FIXME - return maf; + (void)ni_test; + (void)n_miss; // kept for the signature: `observed' is the same number, counted here + if (observed == 0) + return 0.0; + return total / (2.0 * (double)observed); } -- cgit 1.4.1