Location of small database

2 files changed, 35 insertions, 33 deletions

diff --git a/topics/database/mariadb-database-architecture.gmi b/topics/database/mariadb-database-architecture.gmi
index 5c9b0c5..0454d71 100644
--- a/topics/database/mariadb-database-architecture.gmi
+++ b/topics/database/mariadb-database-architecture.gmi
@@ -28,6 +28,12 @@ Naming convention-wise there is a confusing use of id and data-id in particular.
 The default install comes with a smaller database which includes a
 number of the BXDs and the Human liver dataset (GSE9588).
 
+It can be downloaded from:
+
+=> https://files.genenetwork.org/database/
+
+Try the latest one first.
+
 # GeneNetwork database
 
 Estimated table sizes with metadata comment for the important tables
@@ -536,8 +542,8 @@ select * from ProbeSetSE limit 5;
 
 For the other tables, you may check the GN2/doc/database.org document (the starting point for this document).
 
-# Contributions regarding data upload to the GeneNetwork webserver 
-* Ideas shared by the GeneNetwork team to facilitate the process of uploading data to production 
+# Contributions regarding data upload to the GeneNetwork webserver
+* Ideas shared by the GeneNetwork team to facilitate the process of uploading data to production
 
 ## Quality check and integrity of the data to be uploaded to gn2
 
@@ -556,7 +562,7 @@ For the other tables, you may check the GN2/doc/database.org document (the start
 * Unique identifiers solve the hurdles that come with having duplicate genes. So, the QA tools in place should ensure the uploaded dataset adheres to the requirements mentioned
 * However, newer RNA-seq data sets generated by sequencing do not usually have an official vendor identifier. The identifier is usually based on the NCBI mRNA model (NM_XXXXXX) that was used to evaluate an expression and on the sequence that is involved, usually the start and stop nucleotide positions based on a specific genome assembly or just a suffix to make sure it is unique. In this case, you are looking at mRNA assays for a single transcript, but different parts of the transcript that have different genome coordinates. We now typically use ENSEMBL identifiers.
 * The mouse version of the sonic hedgehog gene as an example: `ENSMUST00000002708` or `ENSMUSG00000002633` sources should be fine. The important thing is to know the provenance of the ID—who is in charge of that ID type?
-* When a mRNA assay is super precise (one exon only or a part of the 5' UTR), then we should use exon identifiers from ENSEMBL probably. 
+* When a mRNA assay is super precise (one exon only or a part of the 5' UTR), then we should use exon identifiers from ENSEMBL probably.
 * Ideally, we should enter the sequence's first and last 100 nt in GeneNetwork for verification and  alignment. We did this religiously for arrays, but have started to get lazy now. The sequence is the ultimate identifier
 * For methylation arrays and CpG assays, we can use this format `cg14050475` as seen in MBD UTHSC Ben's data
 * For metabolites like isoleucine—the ID we have been using is the mass-to-charge (MZ) ratio such as `130.0874220_MZ`
@@ -579,16 +585,16 @@ abcb10_q9ji39_t312
 
 ## BXD individuals
 
-* Basically groups (represented by the InbredSet tables) are primarily defined by their list of samples/strains (represented by the Strain tables). When we create a new group, it's because we have data with a distinct set of samples/strains from any existing groups. 
-* So when we receive data for BXD individuals, as far as the database is concerned they are a completely separate group (since the list of samples is new/distinct from any other existing groups). We can choose to also enter it as part of the "generic" BXD group (by converting it to strain means/SEs using the strain of each individual, assuming it's provided like in the files Arthur was showing us). 
+* Basically groups (represented by the InbredSet tables) are primarily defined by their list of samples/strains (represented by the Strain tables). When we create a new group, it's because we have data with a distinct set of samples/strains from any existing groups.
+* So when we receive data for BXD individuals, as far as the database is concerned they are a completely separate group (since the list of samples is new/distinct from any other existing groups). We can choose to also enter it as part of the "generic" BXD group (by converting it to strain means/SEs using the strain of each individual, assuming it's provided like in the files Arthur was showing us).
 * This same logic could apply to other groups as well - we could choose to make one group the "strain mean" group for another set of groups that contain sample data for individuals. But the database doesn't reflect the relationship between these groups*
 * As far as the database is concerned, there is no distinction between strain means and individual sample data - they're all rows in the ProbeSetData/PublishData tables. The only difference is that strain mean data will probably also have an SE value in the ProbeSetSE/PublishSE tables and/or an N (number of individuals per strain) value in the NStrain table
 * As for what this means for the uploader - I think it depends on whether Rob/Arthur/etc wants to give users the ability to simultaneously upload both strain mean and individual data. For example, if someone uploads some BXD individuals' data, do we want the uploader to both create a new group for this (or add to an existing BXD individuals group) and calculate the strain means/SE and enter it into the "main" BXD group? My personal feeling is that it's probably best to postpone that for later and only upload the data with the specific set of samples indicated in the file since it would insert some extra complexity to the uploading process that could always be added later (since the user would need to select "the group the strains are from" as a separate option)
 * The relationship is sorta captured in the CaseAttribute and CaseAttributeXRefNew tables (which contain sample metadata), but only in the form of the metadata that is sometimes displayed as extra columns in the trait page table - this data isn't used in any queries/analyses currently (outside of some JS filters run on the table itself) and isn't that important as part of the uploading process (or at least can be postponed)
 
-## Individual Datasets and Derivatives datasets in gn2 
-* Individual dataset reflects the actual data provided or submitted by the investigator (user). Derivative datasets include the processed information from the individual dataset, as in the case of the average datasets. 
-* An example of an individual dataset would look something like; (MBD dataset) 
+## Individual Datasets and Derivatives datasets in gn2
+* Individual dataset reflects the actual data provided or submitted by the investigator (user). Derivative datasets include the processed information from the individual dataset, as in the case of the average datasets.
+* An example of an individual dataset would look something like; (MBD dataset)
 ```
 #+begin_example
 sample, strain, Sex, Age,…
@@ -600,13 +606,13 @@ FEB0005,BXD16,F,14,…
 ⋮
 #+end_example
 ```
-* The strain column above has repetitive values. Each value has a one-to-many relationship with values on sample column. From this dataset, there can be several derivatives. For example; 
-- Sex-based categories 
-- Average data (3 sample values averaged to one strain value) 
-- Standard error table computed for the averages 
+* The strain column above has repetitive values. Each value has a one-to-many relationship with values on sample column. From this dataset, there can be several derivatives. For example;
+- Sex-based categories
+- Average data (3 sample values averaged to one strain value)
+- Standard error table computed for the averages
 
-## Saving data to database 
-* Strain table schema 
+## Saving data to database
+* Strain table schema
 ```
 #+begin_src sql
   MariaDB [db_webqtl]> DESC Strain;
@@ -639,21 +645,21 @@ FEB0005,BXD16,F,14,…
   5 rows in set (0.00 sec)
 #+end_src
 ```
-* Where the =InbredSetId= comes from the =InbredSet= table and the =StrainId= comes from the =Strain= table. The *individual data* would be linked to an inbredset group that is for individuals 
+* Where the =InbredSetId= comes from the =InbredSet= table and the =StrainId= comes from the =Strain= table. The *individual data* would be linked to an inbredset group that is for individuals
 * For the *average data*, the only value to save would be the =strain= field, which would be saved as =Name= in the =Strain= table and linked to an InbredSet group that is for averages
 *Question 01*: How do we distinguish the inbredset groups?
 *Answer*: The =Family= field is useful for this.
 
 *Question 02*: If you have more derived "datasets", e.g. males-only, females-only, under-10-years, 10-to-25-years, etc. How would the =Strains= table handle all those differences?
 
-## Metadata 
+## Metadata
 * The data we looked at had =gene id= and =gene symbol= fields. These fields were used to fetch the *Ensembl ID* and *descriptions* from [[https://www.ncbi.nlm.nih.gov/][NCBI]] and the [[https://useast.ensembl.org/][Ensembl Genome Browser]]
 
-## Files for mapping 
+## Files for mapping
 * Files used for mapping need to be in =bimbam= or =.geno= formats. We would need to do conversions to at least one of these formats where necessary
 
-## Annotation files 
-* Consider the following schema of DB tables 
+## Annotation files
+* Consider the following schema of DB tables
 #+begin_src sql
   MariaDB [db_webqtl]> DESC InbredSet;
   +-----------------+----------------------+------+-----+---------+----------------+
@@ -718,10 +724,10 @@ FEB0005,BXD16,F,14,…
 - The =used_for_mapping= field should be set to ~Y~ unless otherwise informed
 - The =PedigreeStatus= field is unknown to us for now: set to ~NULL~
 
-* Annotation file format 
+* Annotation file format
 The important fields are:
 - =ChipId=: The platform that the data was collected from/with
-Consider the following table; 
+Consider the following table;
 #+begin_src sql
     MariaDB [db_webqtl]> DESC GeneChip;
     +---------------+----------------------+------+-----+---------+----------------+
@@ -744,7 +750,7 @@ Consider the following table;
  - =Probe_set_Blat_Mb_start=/=Probe_set_Blat_Mb_end=: In Byron's and Beni's data, these correspond to the =geneStart= and =geneEnd= fields respectively. These are the positions, in megabasepairs, that the gene begins and ends at, respectively.
  - =Mb=: This is the =geneStart=/=Probe_set_Blat_Mb_start= value divided by *1000000*. (*Note to self*: Maybe the Probe_set_Blat_Mb_* fields above might not be in megabase pairs — please confirm)
  - =Strand_Probe= and =Strand_Gene=: These fields' values are simply ~+~ or ~-~. If these values are missing, you can [[https://ftp.ncbi.nih.gov/gene/README][retrieve them from NCBI]], specifically from the =orientation= field of seemingly any text file with the field
- - =Chr=: This is the chromosome on which the gene is found 
+ - =Chr=: This is the chromosome on which the gene is found
 
 * The final annotation file will have (at minimum) the following fields (or their
 analogs):
@@ -765,8 +771,8 @@ analogs):
 *  =.geno= Files
 - The =.geno= files have sample names, not the strain/symbol. The =Locus= field in the =.geno= file corresponds to the **marker**. =.geno= files are used with =QTLReaper=
 - The sample names in the ~.geno~ files *MUST* be in the same order as the
-strains/symbols for that species. For example; 
-Data format is as follows; 
+strains/symbols for that species. For example;
+Data format is as follows;
 ```
 #+begin_example
 SampleName,Strain,…
@@ -779,7 +785,7 @@ BJCWI0005,BXD50,…
 ⋮
 #+end_example
 ```
-and the order of strains is as follows; 
+and the order of strains is as follows;
 ```
 #+begin_example
 …,BXD33,…,BXD40,…,BXD50,…
@@ -806,9 +812,9 @@ The order of samples that belong to the same strain is irrelevant - they share t
  - Treatment
  - Sex (Really? Isn't sex an expression of genes?)
  - batch
- - Case ID, etc 
+ - Case ID, etc
 
-* Summary steps to load data to the database 
+* Summary steps to load data to the database
 - [x] Create *InbredSet* group (think population)
 - [x] Load the strains/samples data
 - [x] Load the sample cross-reference data to link the samples to their
@@ -821,8 +827,4 @@ The order of samples that belong to the same strain is irrelevant - they share t
 - [x] Load the *Log2* data (ProbeSetData and ProbeSetXRef tables)
 - [x] Compute means (an SQL query was used — this could be pre-computed in code
   and entered along with the data)
-- [x] Run QTLReaper 
-
-
-
-
+- [x] Run QTLReaper
diff --git a/topics/deploy/installation.gmi b/topics/deploy/installation.gmi
index 757d848..d6baa79 100644
--- a/topics/deploy/installation.gmi
+++ b/topics/deploy/installation.gmi
@@ -319,7 +319,7 @@ Currently we have two databases for deployment,
 from BXD mice and 'db_webqtl_plant' which contains all plant related
 material.
 
-Download one database from
+Download a recent database from
 
 => https://files.genenetwork.org/database/