1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
|
# What is missing from genenetwork4, ranked by effort
A survey of `genenetwork1`, `genenetwork2` and `genenetwork3` (all cloned
side by side) for functionality that genenetwork4 does not have, limited to
what would fit a JSON API rather than a rendered page. Ranked by the effort
it would take to migrate, cheapest first, because that -- not which repo it
came from -- is the real cost driver.
Date: 2026-11-15. Row counts and table locations were measured read-only
against the production `db_webqtl`; the GN1/GN2/GN3 columns were
established by reading the code in each clone.
## How to read the columns
yes the feature exists in that repo, found by reading its code
part only partly there, or a stub, or only the write/UI half
no no implementation found in that repo
`GN4` is not a column: every row in these tables is a gap in
genenetwork4, and the "why" column says what the gap consists of.
Effort is S (hours to a day), M (days), L (a week or two), XL (a new
concept -- write paths, permissions, a service of its own).
## Tier S -- hours to a day
The data is already in the database genenetwork4 reads, and the shape
already exists in the API.
+--------------------------------------+------+-----+------+--------+-----------------------------------------------------------+
| feature | GN1 | GN2 | GN3 | effort | why |
+--------------------------------------+------+-----+------+--------+-----------------------------------------------------------+
| correlation matrix | yes | yes | no | S-M | decisions recorded; N^2 calls to the live correlate_named |
| limit_to / ids_only / names_only | no | yes | no | S | optional query params on existing listings |
| tissue correlation | yes | yes | yes | S-M | TissueProbeSetData is in the DB; the maths already exists |
| literature correlation | yes | yes | yes | S-M | LCorrRamin3 is a lookup table, no statistics involved |
| whole-dataset sample matrix export | yes | yes | yes | S-M | one SQL sweep; the only choice is streaming/CSV vs JSON |
| genotype file exports (bimbam/rqtl2) | part | yes | no | S | BIMBAM is already written internally for GEMMA |
| menu / dropdown tree | yes | yes | yes | S | SQL only; the gap is the nested shape, not the data |
| probeset annotation enrichment | yes | yes | yes | S-M | GeneList and db_GeneOntology are present |
| mapping-result store | yes | no | part | S-M | runs live in /tmp caches today; durable results are new |
+--------------------------------------+------+-----+------+--------+-----------------------------------------------------------+
## Tier M -- days
New data inside MariaDB, or new maths, but no new service or storage.
+-----------------------------------+------+-----+------+--------+-------------------------------------------------------------+
| feature | GN1 | GN2 | GN3 | effort | why |
+-----------------------------------+------+-----+------+--------+-------------------------------------------------------------+
| partial correlations | part | yes | yes | M | needs a genotype/covariate matrix plus a job shape |
| clustered heatmap | yes | yes | yes | M | clustering is cheap; plotting infrastructure already exists |
| SNP / variant browser | yes | yes | no | M-L | SnpAll is 79.5 million rows; window queries first, UI later |
| region / interval queries | yes | no | part | M | locations exist everywhere, a range endpoint does not |
| GeneRIF / GeneWiki read | yes | yes | yes | M | small tables, but a gene-centric resource the model lacks |
| compare traits / corr. annotation | yes | yes | no | M | two-trait scatter plus stats, assembled from what exists |
| QTLminer / qtlreaper re-analysis | yes | yes | yes | M-L | external R/Python, job shape like /compute/rqtl |
| metadata about / history | no | yes | yes | M-L | a second data source: Virtuoso + the gn-docs git tree |
+-----------------------------------+------+-----+------+--------+-------------------------------------------------------------+
## Tier L and XL -- weeks, or a new concept
Write paths, permissions, visualisation services, or a different storage
model. None of these is a data-read problem.
+------------------------------------+-----+-----+------+--------+------------------------------------------------------------+
| feature | GN1 | GN2 | GN3 | effort | why |
+------------------------------------+-----+-----+------+--------+------------------------------------------------------------+
| case-attribute write / moderation | no | no | yes | L | genenetwork4 is read-only; needs scopes and an audit trail |
| WGCNA | no | yes | yes | L | R package, heavy compute, result storage |
| network graph | yes | yes | no | L | visualisation; overlaps the BNW structure work |
| collections / data sharing | yes | yes | no | L-XL | baskets exist, a sharing and permission model does not |
| submit / update trait (data entry) | yes | yes | part | XL | write + moderation + validation + auth, all new |
| genome graph / JBrowse | yes | no | no | XL | a browser service, not an API resource |
| LLM / GN-QA | no | no | yes | XL | external LLM dependency, unrelated to the data model |
| LMDB sample-data backend | no | no | yes | XL | a storage migration question, not an endpoint |
+------------------------------------+-----+-----+------+--------+------------------------------------------------------------+
## What genenetwork4 already covers
So the tables above are not misread as a general backlog: species; groups
with strains, datasets, traits, genotypes, case attributes, parents-and-F1s
and phewas; datasets with probesets; traits with values, per-trait case
attributes and peak genotypes; genotype files with markers, frequencies,
classes and locations; uploads; probesets; genes with traits and aliases;
publications leading to traits; search; mapping methods; tokens; baskets;
matrices; plots; and the compute family -- GEMMA (direct and via ravanan),
R/qtl, R/qtl2, fitqtl, BulkLMM, BNW, and the sample-correlation row.
## Evidence for the cost estimates
ProbeSetData 5 314 694 464 rows trait x individual x value
PublishData 3 288 638 rows phenotype values
LCorrRamin3 127 368 780 rows the only stored correlations
SnpAll 79 534 772 rows db_webqtl
IndelAll 142 894 rows db_webqtl
GeneList 133 471 rows db_webqtl (87 688 in db_webqtl_s)
GeneRIF / GeneRIFXRef 8 425 / 9 387 rows db_webqtl
TissueProbeSetData 90 562 DataIds, 26-32 values each (avg 29.0)
The pattern that matters: every *value* feature is a query-design problem
because of the first four rows, while every *metadata* feature is small --
which is why the S tier is almost entirely metadata.
## Recommended order
1. correlation matrix decisions are already recorded, and the
row endpoint it builds on is live
2. tissue + literature corr. completes the three kinds GN2 offers on
the trait page
3. pagination / dropdown tree unblocks any client that has to list
4. sample matrix + genotypes the "get my data out" gap
5. partial correlations the last GN2 correlation still missing
## Repo locations, for whoever picks a row up
GN1 web/webqtl/{correlation,correlationMatrix,intervalMapping,
markerRegression,pairScan,snpBrowser,heatmap,geneWiki,
collection,dataSharing,qtlminer,submitTrait,updateTrait}
GN2 gn2/wqflask/{api,correlation,correlation_matrix,heatmap,wgcna,
network_graph}; templates/show_trait_calculate_correlations.html
GN3 gn3/api/{correlation,heatmaps,wgcna,search,metadata,menu,
sampledata,case_attributes,data_entry,rqtl,rqtl2,gemma}.py
|