about summary refs log tree commit diff
path: root/docs/migration.md
blob: b3f0d64f8230f0818d1e1206de7c9d8600fa2f9b (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
# What is missing from genenetwork4, ranked by effort

A survey of `genenetwork1`, `genenetwork2` and `genenetwork3` (all cloned
side by side) for functionality that genenetwork4 does not have, limited to
what would fit a JSON API rather than a rendered page. Ranked by the effort
it would take to migrate, cheapest first, because that -- not which repo it
came from -- is the real cost driver.

Date: 2026-11-15. Row counts and table locations were measured read-only
against the production `db_webqtl`; the GN1/GN2/GN3 columns were
established by reading the code in each clone.

## How to read the columns

    yes   the feature exists in that repo, found by reading its code
    part  only partly there, or a stub, or only the write/UI half
    no    no implementation found in that repo

`GN4` is not a column: every row in these tables is a gap in
genenetwork4, and the "why" column says what the gap consists of.
Effort is S (hours to a day), M (days), L (a week or two), XL (a new
concept -- write paths, permissions, a service of its own).

## Tier S -- hours to a day

The data is already in the database genenetwork4 reads, and the shape
already exists in the API.

    +--------------------------------------+------+-----+------+--------+-----------------------------------------------------------+
    | feature                              | GN1  | GN2 | GN3  | effort | why                                                       |
    +--------------------------------------+------+-----+------+--------+-----------------------------------------------------------+
    | correlation matrix                   | yes  | yes | no   | S-M    | decisions recorded; N^2 calls to the live correlate_named |
    | limit_to / ids_only / names_only     | no   | yes | no   | S      | optional query params on existing listings                |
    | tissue correlation                   | yes  | yes | yes  | S-M    | TissueProbeSetData is in the DB; the maths already exists |
    | literature correlation               | yes  | yes | yes  | S-M    | LCorrRamin3 is a lookup table, no statistics involved     |
    | whole-dataset sample matrix export   | yes  | yes | yes  | S-M    | one SQL sweep; the only choice is streaming/CSV vs JSON   |
    | genotype file exports (bimbam/rqtl2) | part | yes | no   | S      | BIMBAM is already written internally for GEMMA            |
    | menu / dropdown tree                 | yes  | yes | yes  | S      | SQL only; the gap is the nested shape, not the data       |
    | probeset annotation enrichment       | yes  | yes | yes  | S-M    | GeneList and db_GeneOntology are present                  |
    | mapping-result store                 | yes  | no  | part | S-M    | runs live in /tmp caches today; durable results are new   |
    +--------------------------------------+------+-----+------+--------+-----------------------------------------------------------+

## Tier M -- days

New data inside MariaDB, or new maths, but no new service or storage.

    +-----------------------------------+------+-----+------+--------+-------------------------------------------------------------+
    | feature                           | GN1  | GN2 | GN3  | effort | why                                                         |
    +-----------------------------------+------+-----+------+--------+-------------------------------------------------------------+
    | partial correlations              | part | yes | yes  | M      | needs a genotype/covariate matrix plus a job shape          |
    | clustered heatmap                 | yes  | yes | yes  | M      | clustering is cheap; plotting infrastructure already exists |
    | SNP / variant browser             | yes  | yes | no   | M-L    | SnpAll is 79.5 million rows; window queries first, UI later |
    | region / interval queries         | yes  | no  | part | M      | locations exist everywhere, a range endpoint does not       |
    | GeneRIF / GeneWiki read           | yes  | yes | yes  | M      | small tables, but a gene-centric resource the model lacks   |
    | compare traits / corr. annotation | yes  | yes | no   | M      | two-trait scatter plus stats, assembled from what exists    |
    | QTLminer / qtlreaper re-analysis  | yes  | yes | yes  | M-L    | external R/Python, job shape like /compute/rqtl             |
    | metadata about / history          | no   | yes | yes  | M-L    | a second data source: Virtuoso + the gn-docs git tree       |
    +-----------------------------------+------+-----+------+--------+-------------------------------------------------------------+

## Tier L and XL -- weeks, or a new concept

Write paths, permissions, visualisation services, or a different storage
model. None of these is a data-read problem.

    +------------------------------------+-----+-----+------+--------+------------------------------------------------------------+
    | feature                            | GN1 | GN2 | GN3  | effort | why                                                        |
    +------------------------------------+-----+-----+------+--------+------------------------------------------------------------+
    | case-attribute write / moderation  | no  | no  | yes  | L      | genenetwork4 is read-only; needs scopes and an audit trail |
    | WGCNA                              | no  | yes | yes  | L      | R package, heavy compute, result storage                   |
    | network graph                      | yes | yes | no   | L      | visualisation; overlaps the BNW structure work             |
    | collections / data sharing         | yes | yes | no   | L-XL   | baskets exist, a sharing and permission model does not     |
    | submit / update trait (data entry) | yes | yes | part | XL     | write + moderation + validation + auth, all new            |
    | genome graph / JBrowse             | yes | no  | no   | XL     | a browser service, not an API resource                     |
    | LLM / GN-QA                        | no  | no  | yes  | XL     | external LLM dependency, unrelated to the data model       |
    | LMDB sample-data backend           | no  | no  | yes  | XL     | a storage migration question, not an endpoint              |
    +------------------------------------+-----+-----+------+--------+------------------------------------------------------------+

## What genenetwork4 already covers

So the tables above are not misread as a general backlog: species; groups
with strains, datasets, traits, genotypes, case attributes, parents-and-F1s
and phewas; datasets with probesets; traits with values, per-trait case
attributes and peak genotypes; genotype files with markers, frequencies,
classes and locations; uploads; probesets; genes with traits and aliases;
publications leading to traits; search; mapping methods; tokens; baskets;
matrices; plots; and the compute family -- GEMMA (direct and via ravanan),
R/qtl, R/qtl2, fitqtl, BulkLMM, BNW, and the sample-correlation row.

## Evidence for the cost estimates

    ProbeSetData            5 314 694 464 rows   trait x individual x value
    PublishData                 3 288 638 rows   phenotype values
    LCorrRamin3               127 368 780 rows   the only stored correlations
    SnpAll                     79 534 772 rows   db_webqtl
    IndelAll                      142 894 rows   db_webqtl
    GeneList                      133 471 rows   db_webqtl (87 688 in db_webqtl_s)
    GeneRIF / GeneRIFXRef     8 425 / 9 387 rows   db_webqtl
    TissueProbeSetData     90 562 DataIds, 26-32 values each (avg 29.0)

The pattern that matters: every *value* feature is a query-design problem
because of the first four rows, while every *metadata* feature is small --
which is why the S tier is almost entirely metadata.

## Recommended order

    1.  correlation matrix          decisions are already recorded, and the
                                    row endpoint it builds on is live
    2.  tissue + literature corr.   completes the three kinds GN2 offers on
                                    the trait page
    3.  pagination / dropdown tree  unblocks any client that has to list
    4.  sample matrix + genotypes   the "get my data out" gap
    5.  partial correlations        the last GN2 correlation still missing

## Repo locations, for whoever picks a row up

    GN1   web/webqtl/{correlation,correlationMatrix,intervalMapping,
          markerRegression,pairScan,snpBrowser,heatmap,geneWiki,
          collection,dataSharing,qtlminer,submitTrait,updateTrait}
    GN2   gn2/wqflask/{api,correlation,correlation_matrix,heatmap,wgcna,
          network_graph}; templates/show_trait_calculate_correlations.html
    GN3   gn3/api/{correlation,heatmaps,wgcna,search,metadata,menu,
          sampledata,case_attributes,data_entry,rqtl,rqtl2,gemma}.py