about summary refs log tree commit diff
path: root/docs
diff options
context:
space:
mode:
authorPjotr Prins <pjotr.public01@thebird.nl>2026-10-03 13:32:53 +0000
committerPjotr <wrk@bosa.lan>2026-10-03 13:32:53 +0000
commit5bcab4cbcf7bea03223161dd29d81d93f71975ce (patch)
treef2895db3b83b77391a2a7d4c5428c5b8d5dcd4bb /docs
parent04347014fd571ff99b4343c1ba26822c2de990bd (diff)
downloadgenenetwork4-5bcab4cbcf7bea03223161dd29d81d93f71975ce.tar.gz
doc: Add docs/genenetwork_datatypes.md, Species section.
* docs/genenetwork_datatypes.md: New file.  Species table field
semantics (sourced from factory/docs/gn-gemtext-threads/topics/
gn-uploader/adding-species.gmi and this repo's own
docs/mariadb_db-webqtl.md schema dump), its relationships to
InbredSet, Strain, Chr_Length and GeneIDXRef, its role in GN's
authorization and RDF classification scheme, known quirks
(mouse-centric legacy genotype columns, undocumented FamilyOrderId/
OrderId, no UI for adding a species), and a survey of existing
Species-related implementations across GeneNetwork2, GeneNetwork3,
gn-guile and this repo.
Diffstat (limited to 'docs')
-rw-r--r--docs/genenetwork_datatypes.md99
1 files changed, 99 insertions, 0 deletions
diff --git a/docs/genenetwork_datatypes.md b/docs/genenetwork_datatypes.md
new file mode 100644
index 0000000..f666f50
--- /dev/null
+++ b/docs/genenetwork_datatypes.md
@@ -0,0 +1,99 @@
+# GeneNetwork data types
+
+Notes on the core data types GeneNetwork is built around, compiled from
+`factory/docs/gn-gemtext-threads`, `factory/code/genenetwork2`,
+`factory/code/genenetwork3`, `factory/code/gn-guile`, and this repo's own
+`docs/mariadb_db-webqtl.md` schema dump.
+
+## Species
+
+The `Species` table (`db_webqtl`; full `CREATE TABLE` in
+`docs/mariadb_db-webqtl.md`) is the root of GeneNetwork's data hierarchy:
+every `InbredSet` (a group/population, e.g. BXD), every `Strain`, and
+every chromosome length record (`Chr_Length`) is scoped to a species via
+a `SpeciesId` foreign key.
+
+### Fields
+
+Field semantics, from `topics/gn-uploader/adding-species.gmi` (written
+while manually adding C. elegans to the database) and cross-checked
+against the live schema:
+
+| Field           | Meaning |
+|-----------------|---------|
+| `Id`            | Internal GeneNetwork species identifier (auto-increment primary key). |
+| `SpeciesId`     | Set equal to `Id` after insert (see "Known quirks" below); other tables' `SpeciesId` foreign keys reference this, not `Id` directly, though in practice the two are always equal. |
+| `SpeciesName`   | A layman's name, e.g. "Mouse", "Roundworm", "Barley". |
+| `Name`          | Mostly the same as `SpeciesName`, lowercased, e.g. "mouse". This is the short name used in URLs and API lookups. |
+| `MenuName`      | Display name shown in the UI, typically combining `SpeciesName` and `FullName`/assembly, e.g. "Mouse (Mus musculus, mm10)". |
+| `FullName`      | The species' binomial (scientific) name, e.g. "Mus musculus", "Caenorhabditis elegans". |
+| `Family`        | A loose grouping across species, e.g. "Vertebrates", "Nematodes". |
+| `FamilyOrderId` | Sort order within `Family` for menu display. Exact derivation undocumented (`adding-species.gmi` flags this with a TODO). |
+| `TaxonomyId`    | The NCBI Taxonomy ID (https://www.ncbi.nlm.nih.gov/), e.g. 10090 for mouse, 6239 for C. elegans. |
+| `OrderId`       | Sort order for display. Exact derivation undocumented, same TODO as `FamilyOrderId`. |
+
+### Relationships
+
+- `InbredSet.SpeciesId` -> `Species` -- every group/population (e.g.
+  BXD) belongs to exactly one species.
+- `Strain.SpeciesId` -> `Species` -- every strain belongs to exactly one
+  species (`genenetwork2/gn2/base/species.py`'s `TheSpecies`/
+  `Chromosomes` classes join on this).
+- `Chr_Length.SpeciesId` -> `Species` -- chromosome names and lengths
+  are per species (used by `gn3.db.species.get_chromosome` and GN2's
+  `Chromosomes.chromosomes`, both selecting from `Chr_Length` joined to
+  either `Species` directly by name, or indirectly via `InbredSet` for
+  cases where only a group name is known).
+- `GeneIDXRef` cross-references gene IDs between species. GeneNetwork's
+  internal gene identifiers are mouse-centric by convention: GN3's
+  `gn3.db.species.translate_to_mouse_gene_id` translates a rat or human
+  gene ID to its mouse equivalent via this table (returns the ID
+  unchanged if the species is already "mouse").
+- `InbredSet.SpeciesId` also scopes authorization: GN3's
+  `gn3/api/case_attributes.py` resolves a species ID from `InbredSet` to
+  build the resource ID used to check a user's access with `gn-auth`
+  (https://github.com/genenetwork/gn-auth), so species is part of GN's
+  permission hierarchy, not just a data-organization concept.
+- In the RDF/SPARQL metadata graph (`topics/next-gen-databases/gn-classification-scheme.gmi`),
+  species is one of the three levels of GN's resource classification
+  scheme (`gnc:DatasetType`, `gnc:Set`, `gn:Species`), alongside
+  `InbredSet` ("Set") and `DatasetType`. Every member of the `Species`
+  table has a corresponding `gn:Species`-classified RDF resource;
+  `gn-guile`'s `gn/data/species.scm` builds this metadata from SPARQL,
+  separately from the MariaDB `Species` table.
+
+### Known quirks
+
+- GeneNetwork's genotype tables were originally built mouse-only and
+  still carry that legacy: `topics/gn-uploader/genotypes-assemblies-markers-and-genenetwork.gmi`
+  notes the `Geno` table has assembly-specific, mouse-named columns
+  (`Mb_mm8`, `Chr_mm8`) that don't generalize to other species' genome
+  assemblies.
+- There is no UI for adding a new species; it's done by hand with
+  `INSERT INTO Species(...)` followed by `UPDATE Species SET
+  SpeciesId=Id WHERE ...` (see `adding-species.gmi` for the full
+  worked example, adding C. elegans).
+- `FamilyOrderId` and `OrderId` remain undocumented upstream (flagged
+  TODO in `adding-species.gmi`); their exact intended derivation isn't
+  recorded anywhere found in these repos.
+
+### Existing implementations
+
+- **GeneNetwork2** (`gn2/wqflask/api/router.py`): `GET
+  /api/v_pre1/species` (all species, columns `Id`/`Name`/`FullName`/
+  `TaxonomyId`) and `GET /api/v_pre1/species/<species_name>` (one
+  species, matched against `Name`, `FullName` or `SpeciesName`, same
+  4-column subset). `gn2/base/species.py`'s `TheSpecies` class also
+  exposes a species' chromosomes via `Chr_Length`.
+- **GeneNetwork3**: `gn3/db/species.py` has `get_all_species`,
+  `get_chromosome`, `species_name` (species for an `InbredSet` group)
+  and `translate_to_mouse_gene_id`; none of these are exposed as a
+  `Species`-table REST endpoint. `gn3/api/metadata.py`'s `GET /species`
+  and `GET /species/<name>` are SPARQL-based RDF metadata lookups
+  instead, a different data source from the MariaDB `Species` table.
+- **gn-guile** (`gn/data/species.scm`): SPARQL/RDF metadata only, same
+  category as GN3's `/species` above, not backed by `db_webqtl`.
+- **GeneNetwork4** (this repo): `GET /species` (all species, mirrors
+  GN3's `get_all_species` query/ordering) and `GET /species/{species}`
+  (one species, every column, same match logic as GN2's
+  `get_species_info` -- see `README.md`).