From f026a7b808fd2d61e64d4d53c540fd66fffcff81 Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Wed, 18 Nov 2026 11:30:59 +0000 Subject: mdb: apply the marker filters, count missing, and give K the same marker set An lmdb scan and a BIMBAM scan of the same data were not answering the same question. Measured on HLC (358454 markers, 23 chromosomes, gemma-wrapper's LOCO recipe), before this commit: -maf 0.05 gemma 20246 markers on chr1, pangemma 27940 (every marker in the store): mdb_calc_gwa computed maf and put it in MarkerInfo but nothing ever compared it with maf_level, and -miss was ignored too af/n_miss nan / 0 for every marker carrying a missing call, because compute_maf was called with n_miss hard-wired to 0 ("count NAs: FIXME") and summed NaNs into the frequency Gb stores 255 is the missing sentinel; read as 255/127.0 it became 2.0079, a valid-looking genotype that silently poisoned af and the test K mdb_calc_kin had no missing handling at all (its mask marked every individual, so the imputation branch never fired) and applied no filters, so HLC's K was built from 330514 markers where GEMMA uses 273349 -- GEMMA's K reader skips whatever its genotype reader rejected (indicator_snp[i] == 0) -- and every p-value moved log line mdb_calc_kin never reported the count, so "number of analyzed SNPs/var" printed an uninitialised ns_test (140048413575759) All four readers of gemma_io.cpp apply the same five filters in a fixed order; the lmdb path now does the same, in the same order, with GEMMA's own allele bins: -miss, -maf (maf_level == -1 disables), monomorphic (every observed genotype equal), -hwe, and -r2 against the covariates (missing imputed with maf*2 as the readers do, W'W inverted once per scan). Missing calls arrive as NaN in both formats, compute_maf counts what it sees, and the K path takes the same filtered marker set and imputes the observed mean exactly as gemma_io.cpp's ReadFile_geno does for K. Verified, HLC LOCO on the slurm cluster (23 tasks, 1 CPU each, all 23 RUNNING together), pangemma lmdb (Gb, missing kept) against gemma 0.98.6 on BIMBAM: same row count on all 23 chromosomes (260251 rows), same top SNP on 23/23 (was 12/23), and no differing numeric cell in beta, se, l_mle or p_lrt (was 260251 rows differing with l_mle pinned at the 1e5 bound). The BIMBAM route is unchanged: 23/23 and 0 differing cells, as before. Cost of the filtering: job time for the lmdb route fell from 179 s to 141 s for the 23 chromosomes (gemma BIMBAM: 958 s), because 30 percent of the markers are no longer analysed. On BXD.8 (20453 markers, 198 individuals, 1460 missing calls) the Gf store now reproduces gemma's output column for column -- af and n_miss included -- with the only differences being 33 `ps` values, where GEMMA's annotation reader truncates R's "1.76e+08" to 1 and the store has the rounded position. --- src/param.cpp | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) (limited to 'src/param.cpp') diff --git a/src/param.cpp b/src/param.cpp index 7e34830..b2098ce 100644 --- a/src/param.cpp +++ b/src/param.cpp @@ -1309,7 +1309,14 @@ void PARAM::ReadBIMBAMGenotypes(gsl_matrix *UtX, gsl_matrix *K, const bool calc_ } gsl_matrix *PARAM::MdbCalcKin() { - return mdb_calc_kin(file_geno, a_mode - 20, loco); + size_t n_used = 0; + gsl_matrix *K = mdb_calc_kin(file_geno, a_mode - 20, loco, n_used, + maf_level, miss_level); + // Feed the log: ns_test was never set on this path, so "number of analyzed + // SNPs/var" printed whatever the PARAM object happened to hold. + ns_total = n_used; + ns_test = n_used; + return K; } void PARAM::CalcKin(gsl_matrix *matrix_kin) { -- cgit 1.4.1