diff options
author | Munyoki Kilyungi | 2024-08-28 15:05:14 +0300 |
---|---|---|
committer | Munyoki Kilyungi | 2024-08-28 15:05:14 +0300 |
commit | d47cf4ca6f800b914b4d68942baf7830a558609f (patch) | |
tree | c8e5c1729e924847bcdea81da0c97830f3eeef52 | |
parent | 65240db0391704f6502b98ce95f3e45c835896f6 (diff) | |
download | gn-gemtext-d47cf4ca6f800b914b4d68942baf7830a558609f.tar.gz |
Clean up white spaces in topic.
Signed-off-by: Munyoki Kilyungi <me@bonfacemunyoki.com>
-rw-r--r-- | topics/database/mariadb-database-architecture.gmi | 40 |
1 files changed, 15 insertions, 25 deletions
diff --git a/topics/database/mariadb-database-architecture.gmi b/topics/database/mariadb-database-architecture.gmi index a9a8269..b200dbd 100644 --- a/topics/database/mariadb-database-architecture.gmi +++ b/topics/database/mariadb-database-architecture.gmi @@ -536,33 +536,33 @@ select * from ProbeSetSE limit 5; For the other tables, you may check the GN2/doc/database.org document (that was the starting point for this document). -## Additional Information +## Additional Information -* This includes all contributions from the gn2 team on making sure the process of uploading data to gn2 production is a tremendous success, enhancing the process of uploading using Fred's upload system, and most importantly, making sure that the dataset uploaded is valid and can be tested and used by investigators while in gn2 webserver. +* This includes all contributions from the gn2 team on making sure the process of uploading data to gn2 production is a tremendous success, enhancing the process of uploading using Fred's upload system, and most importantly, making sure that the dataset uploaded is valid and can be tested and used by investigators while in gn2 webserver. -### Arthur's contributions +### Arthur's contributions -#### On the quality check and integrity of the data to be uploaded to gn2 +#### On the quality check and integrity of the data to be uploaded to gn2 * A note to add (from Arthur): Some datasets have the following identifiers: ProbeSet IDs {chr_3020701, chr_3020851, etc}. This is not an acceptable way to name the probeset IDs. So, the data provider needs to understand what format is needed for gn2 to accept the ProbeSet IDs in their dataset -* Also, for the annotation file, among other important columns, it is crucial that there are descriptions, aliases, and location columns. And the formatting should be exactly as found in the public repositories such as NCBI, Ensembl, etc. For instance, for description: `X-linked Kx blood group related 4`, and Aliases: ` XRG4; Gm210; mKIAA1889` as in +* Also, for the annotation file, among other important columns, it is crucial that there are descriptions, aliases, and location columns. And the formatting should be exactly as found in the public repositories such as NCBI, Ensembl, etc. For instance, for description: `X-linked Kx blood group related 4`, and Aliases: ` XRG4; Gm210; mKIAA1889` as in => https://www.ncbi.nlm.nih.gov/gene/497097 #### On the valid ProbeSetIDs * He suggested that the official ProbeSetIDs would be the one from the vendor. This would also constitute the platform used to generate data {Novogene-specific platform}, for instance; `NovaSeqPE150` for the MBD UTHSC mice seq dataset -* NB; in this case, if the vendor does not provide the official names as expected, we can use the platform + the numbering order of the file to generate probeset IDs. For instance; `NseqPE150_000001 to NseqPE150_432694` for samples 1 to 432694 +* NB; in this case, if the vendor does not provide the official names as expected, we can use the platform + the numbering order of the file to generate probeset IDs. For instance; `NseqPE150_000001 to NseqPE150_432694` for samples 1 to 432694 -### Rob's contributions +### Rob's contributions -#### On the importance of having unique identifiers within a platform +#### On the importance of having unique identifiers within a platform -* Unique identifiers solve the hurdles that come with having duplicate genes. So, the QA tools in place should ensure the uploaded dataset adheres to the requirements mentioned -* However, newer RNA-seq data sets generated by sequencing do not usually have an official vendor identifier. The identifier is usually based on the NCBI mRNA model (NM_XXXXXX) that was used to evaluate an expression and on the sequence that is involved, usually the start and stop nucleotide positions based on a specific genome assembly or just a suffix to make sure it is unique. In this case, you are looking at mRNA assays for a single transcript, but different parts of the transcript that have different genome coordinates. We now typically use ENSEMBL identifiers. Here is the mouse version of the sonic hedgehog gene as an example: `ENSMUST00000002708` or `ENSMUSG00000002633` sources should be fine. The important thing is to know the provenance of the ID—who is in charge of that ID type? +* Unique identifiers solve the hurdles that come with having duplicate genes. So, the QA tools in place should ensure the uploaded dataset adheres to the requirements mentioned +* However, newer RNA-seq data sets generated by sequencing do not usually have an official vendor identifier. The identifier is usually based on the NCBI mRNA model (NM_XXXXXX) that was used to evaluate an expression and on the sequence that is involved, usually the start and stop nucleotide positions based on a specific genome assembly or just a suffix to make sure it is unique. In this case, you are looking at mRNA assays for a single transcript, but different parts of the transcript that have different genome coordinates. We now typically use ENSEMBL identifiers. Here is the mouse version of the sonic hedgehog gene as an example: `ENSMUST00000002708` or `ENSMUSG00000002633` sources should be fine. The important thing is to know the provenance of the ID—who is in charge of that ID type? * When a mRNA assay is super precise (one exon only or a part of the 5' UTR), then we should use exon identifiers from ENSEMBL probably. Ideally, we should enter the first and last 100 nt of the sequence in GeneNetwork for verification and alignment. We did this religiously for arrays, but have started to get lazy now. The sequence is the ultimate identifier -* For methylation arrays and CpG assays, we can use this format `cg14050475` as seen in MBD UTHSC Ben's data -* For metabolites like isoleucine—the ID we have been using is the mass-to-charge (MZ) ratio such as `130.0874220_MZ` -* For protein and peptide identifiers we have used the official Protein ID followed by an underscore character and then some or all of the sequence. This is then followed by another underscore and a number. Evan to confirm, but the suffix number is the charge state if I remember correctly +* For methylation arrays and CpG assays, we can use this format `cg14050475` as seen in MBD UTHSC Ben's data +* For metabolites like isoleucine—the ID we have been using is the mass-to-charge (MZ) ratio such as `130.0874220_MZ` +* For protein and peptide identifiers we have used the official Protein ID followed by an underscore character and then some or all of the sequence. This is then followed by another underscore and a number. Evan to confirm, but the suffix number is the charge state if I remember correctly ``` Q9JHJ3_LLHTADVCQLEVALVGASPR_3 A2A8E1_TIVEFECR_2 @@ -577,22 +577,12 @@ abcb10_q9ji39_t312 * The above is just the gene symbol then the protein ID and not so sure what t311 and t312 mean * Ideally these IDs are explained to some extent when they embed some information -### Zach's contributions +### Zach's contributions -#### Regarding the BXD individuals as far as GeneNetwork is concerned +#### Regarding the BXD individuals as far as GeneNetwork is concerned * Basically groups (represented by the InbredSet tables) are primarily defined by their list of samples/strains (represented by the Strain tables). When we create a new group, it's because we have data with a distinct set of samples/strains from any existing groups. So when we receive data for BXD individuals, as far as the database is concerned they are a completely separate group (since the list of samples is new/distinct from any other existing groups). We can choose to also enter it as part of the "generic" BXD group (by converting it to strain means/SEs using the strain of each individual, assuming it's provided like in the files Arthur was showing us). This same logic could apply to other groups as well - we could choose to make one group the "strain mean" group for another set of groups that contain sample data for individuals. But the database doesn't reflect the relationship between these groups* * As far as the database is concerned, there is no distinction between strain means and individual sample data - they're all rows in the ProbeSetData/PublishData tables. The only difference is that strain mean data will probably also have an SE value in the ProbeSetSE/PublishSE tables and/or an N (number of individuals per strain) value in the NStrain table * As for what this means for the uploader - I think it depends on whether Rob/Arthur/etc wants to give users the ability to simultaneously upload both strain mean and individual data. For example, if someone uploads some BXD individuals' data, do we want the uploader to both create a new group for this (or add to an existing BXD individuals group) and calculate the strain means/SE and enter it into the "main" BXD group? My personal feeling is that it's probably best to postpone that for later and only upload the data with the specific set of samples indicated in the file since it would insert some extra complexity to the uploading process that could always be added later (since the user would need to select "the group the strains are from" as a separate option) * In general Fred and I are probably the best people to ask about database stuff, while Arthur (and maybe also Fred?) is the best reference for the actual files that are submitted (which I'm not that familiar with). Arthur is familiar with the database structure, but not as familiar with the way the code interacts with it * The relationship is sorta captured in the CaseAttribute and CaseAttributeXRefNew tables (which contain sample metadata), but only in the form of the metadata that is sometimes displayed as extra columns in the trait page table - this data isn't used in any queries/analyses currently (outside of some JS filters run on the table itself) and isn't that important as part of the uploading process (or at least can be postponed) - - - - - - - - - - |