diff options
Diffstat (limited to 'topics')
| -rw-r--r-- | topics/arvados/covid19.gmi | 7 | ||||
| -rw-r--r-- | topics/cli/weechat.gmi | 66 | ||||
| -rw-r--r-- | topics/data-uploads/editing-data.gmi | 10 | ||||
| -rw-r--r-- | topics/documentation/gn-documentation.gmi | 21 | ||||
| -rw-r--r-- | topics/gemini/cheatsheet.gmi | 5 | ||||
| -rw-r--r-- | topics/gemini/ideas_and_references.gmi | 3 | ||||
| -rw-r--r-- | topics/pudb/pytest-pudb.gmi | 7 | ||||
| -rw-r--r-- | topics/pudb/using-pudb.gmi | 21 | ||||
| -rw-r--r-- | topics/quality-control/qc-checks.gmi | 47 | ||||
| -rw-r--r-- | topics/quality-control/shapiro-wilk-test.gmi | 33 | ||||
| -rw-r--r-- | topics/quality-control/ui-design.gmi | 41 | ||||
| -rw-r--r-- | topics/systems/machine-room.gmi | 7 | ||||
| -rw-r--r-- | topics/systems/sheepdog.gmi | 1 | ||||
| -rw-r--r-- | topics/testing/genenetwork-checklist.gmi | 59 |
14 files changed, 325 insertions, 3 deletions
diff --git a/topics/arvados/covid19.gmi b/topics/arvados/covid19.gmi new file mode 100644 index 0000000..be96605 --- /dev/null +++ b/topics/arvados/covid19.gmi @@ -0,0 +1,7 @@ +# COVID-19 + +We are setting up a Cloud HPC for COVID-19 using Arvados as the underlying infrastructure. + +## Tasks + +=> https://github.com/yatisht/usher Fast phylogeny diff --git a/topics/cli/weechat.gmi b/topics/cli/weechat.gmi new file mode 100644 index 0000000..b6fc062 --- /dev/null +++ b/topics/cli/weechat.gmi @@ -0,0 +1,66 @@ +# Weechat + +Weechat is a very versatile terminal chat that can handle IRC, Matrix and Slack. + +## Weechat install + slack + +Arun has packaged weechat for slack and matrix in GNU Guix. Current install using a guix environment: + +``` +env GUIX_PACKAGE_PATH=~/guix-bioinformatics/ ~/opt/guix-latest/bin/guix environment --ad-hoc coreutils weechat weechat-matrix weechat-wee-slack nss-certs openssl python +echo $GUIX_ENVIRONMENT +/gnu/store/i4a3ynfjvq8r8cch14gkn2fwfs77wagj-profile/ +weechat +/python load /gnu/store/i4a3ynfjvq8r8cch14gkn2fwfs77wagj-profile/share/weechat/python/wee_slack.py +/slack register +``` + +So, capture the shell GUIX_ENVIRONMENT and use that to launch the wee-slack plugin. Next register a token through te Slack web interface. + +If you go through the motions you get a token and + +``` +/slack register 2654233192084.2682932906197.fd6d6e55c0038f-etc-052b4fa578ec497710b4b8770f1-etc +/python reload slack +``` + +More instructions for https://github.com/wee-slack/wee-slack. + +## Weechat IRC + +We have a GeneNetwork channel + +``` +/set irc.server_default.nicks pjotrp +/server add libera irc.libera.chat/6697 -ssl -autoconnect +/connect libera +/join #guix +/join #guix-risc-v +/join #guix-hpc +/join #genenetwork +``` + +## Weechat matrix + +The matrix plugin works similar to above Slack plugin. + +``` +/script load matrix.py +/matrix server add fosdem chat.fosdem.org +/set matrix.server.matrix_org.username pjotrp-linux +/set matrix.server.matrix_org.password **** +/set matrix.server.matrix_org.autoconnect on +/save +``` + +Only the matrix version is not so polished. For example, it is hard to find the buffers because they are named by a hash. In a buffer you can rename with + +``` +/buffer set short_name myname +``` + +Also, you need to enable every channel explicitly to exchange keys + +``` +/olm verify @user:matrix.org * +``` diff --git a/topics/data-uploads/editing-data.gmi b/topics/data-uploads/editing-data.gmi index 1bc1bf8..01144a0 100644 --- a/topics/data-uploads/editing-data.gmi +++ b/topics/data-uploads/editing-data.gmi @@ -1,3 +1,7 @@ +## Tags + +* assigned: bonfacem + ### Introduction At the moment, you can edit metadata related to a published phenotype and a probeset. When an edit is done, the diff data is stored in a table, `metadata_audit` in json format that looks something like: @@ -60,3 +64,9 @@ After editing traits, you could see changes in a diff format at the very top of That needs to be improved. - For Probeset data, atm some tables aren't yet updated(unfortunately those tables aren't used anywhere). I need to figure that out. + +#### Tue 23 Nov 2021 + +- Fixed excel issue described here: + +=> https://github.com/genenetwork/genenetwork2/compare/testing...BonfaceKilz:bug/fix-excel-adding-new-line?expand=1
\ No newline at end of file diff --git a/topics/documentation/gn-documentation.gmi b/topics/documentation/gn-documentation.gmi new file mode 100644 index 0000000..653a1a4 --- /dev/null +++ b/topics/documentation/gn-documentation.gmi @@ -0,0 +1,21 @@ +# GeneNetwork documentation + +GeneNetwork requires an online 'wiki' and a place for documentation. Both of these can be maintained in markdown and/or gemini gemtext. We are using a repo: + +=> https://github.com/genenetwork/gn-docs + +What we need to do is render documents from markdown (as is done with the facilities page) + +=> https://github.com/genenetwork/gn-docs/blob/master/general/help/facilities.md + +the markdown is rendered on github, which makes it easy to edit for most users, so it should be the default. + +## Tags + +* assigned: pjotrp + +## Tasks + +=> storyboards + +## Info diff --git a/topics/gemini/cheatsheet.gmi b/topics/gemini/cheatsheet.gmi new file mode 100644 index 0000000..b2158ea --- /dev/null +++ b/topics/gemini/cheatsheet.gmi @@ -0,0 +1,5 @@ +# gemtext cheatsheet + +=> gemini://gemini.circumlunar.space/docs/cheatsheet.gmi + +=> https://gemini.circumlunar.space/docs/cheatsheet.gmi proxy diff --git a/topics/gemini/ideas_and_references.gmi b/topics/gemini/ideas_and_references.gmi new file mode 100644 index 0000000..dfb026d --- /dev/null +++ b/topics/gemini/ideas_and_references.gmi @@ -0,0 +1,3 @@ +Ideas and References + +=> gemini://drewdevault.com/2021/11/15/Reframing-gemini.gmi Reframing the philosophy of Gemini diff --git a/topics/pudb/pytest-pudb.gmi b/topics/pudb/pytest-pudb.gmi new file mode 100644 index 0000000..5a979c0 --- /dev/null +++ b/topics/pudb/pytest-pudb.gmi @@ -0,0 +1,7 @@ +# Using pudb with pytest + +``` +pytest --pdbcls pudb.debugger:Debugger --pdb --capture=no +``` + +=> https://documen.tician.de/pudb/starting.html?highlight=pytest#usage-with-pytest diff --git a/topics/pudb/using-pudb.gmi b/topics/pudb/using-pudb.gmi new file mode 100644 index 0000000..877a700 --- /dev/null +++ b/topics/pudb/using-pudb.gmi @@ -0,0 +1,21 @@ +# Using pudb + +## Setting Breakpoints + +You can add the following line anywhere in your code to start a pudb breakpoint: + +``` +import pudb; pudb.set_trace() +``` + +You can also press `b` in pudb to add a breakpoint with a keyboard shortcut. + +## pdb and pudb API similarity + +The standard pdb docs should work for pudb with one small exception. + +> At the programming language level, PuDB displays the same interface +> as Python’s built-in pdb module. Just replace pdb with pudb. (One +> exception: run is called runstatement.) + +=> https://docs.python.org/3/library/pdb.html?highlight=pdb#module-pdb diff --git a/topics/quality-control/qc-checks.gmi b/topics/quality-control/qc-checks.gmi new file mode 100644 index 0000000..0d04625 --- /dev/null +++ b/topics/quality-control/qc-checks.gmi @@ -0,0 +1,47 @@ +# Quality Control Checks + +1. ProbeSetId (Affymetrix format): + +We favour using Illumina, Affimetrix, and other platform formats. + +Custom formats require a new annotation file to be created. + +We usually use Ensemble ID or Gene IDs. + +1.1 Ensemble transcript IDs usually have duplicates that need to be pruned. + +ENSMBL1234 + +## Example Gene Symbol to ProbeSetId + +AFFX-BkGr-GC03_st -> TCO500002136.mm.2 + +2. Inbred Strain names should prefer long form: + +B6 -> C57BL/6 +D2 -> DBA/2 + +3. Probeset IDs that don't have any values should be pruned: + +For example an Affymetrix data set might have ~28,000 entries and the data set that +is allowed into the GeneNetwork will be 22,000 entries. + +4. The standard error between male and female mice has to be computed. + +5. SE values have to be computed to 6 or greater decimal places. + +6. The average between male and female mice has to be computed to 3 decimal places. + +7. Datasets/studies having the same ProbeSetID should be grouped together. + +8. There should be no trailing spaces in data cells. + +9. Entries should have the same capitalization style. + +10. Assesing Phenotypes for normality with Shapiro-Wilk Test. + +11. Check for annotations file. + +12. Check for CRLF. + +13. Check for UTF-8 encoding. diff --git a/topics/quality-control/shapiro-wilk-test.gmi b/topics/quality-control/shapiro-wilk-test.gmi new file mode 100644 index 0000000..b1bdbfd --- /dev/null +++ b/topics/quality-control/shapiro-wilk-test.gmi @@ -0,0 +1,33 @@ +# Shapiro Wilk Test + +This document contains more info about QC step 11. + +https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html +https://vedexcel.com/how-to-perform-a-shapiro-wilk-test-in-python/ + +``` +Here is a simple QC procedure we may want to consider that was used by +Megan and Camron in a recent paper that I have attached. + + +Phenotypes were assessed for normality using the Shapiro–Wilk Test. Because +some of the data residuals devi- ated significantly from normality, we used +the orderNorm function to perform Ordered Quantile normalization43 on all +phenotypes. + +-- +Rob +``` + +``` +QTL analysis was performed in F2 mice using the R package R/qtl +(RRID:SCR_009085) as previously described.11,29,42 Quality checking +of genotypes and QTL analysis were performed in R (https://www.r- +project.org/) using R/bestNormalize (https://github.com/petersonR/ +bestNormalize) and R/qtl.42 Phenotypes were assessed for normality +using the Shapiro–Wilk Test. Because some of the data residuals devi- +ated significantly from normality, we used the orderNorm function to +perform Ordered Quantile normalization43 on all phenotypes. QTL +-- +A quantitative trait variant in Gabra2 underlies increased methamphetamine stimulant sensitivity +``` diff --git a/topics/quality-control/ui-design.gmi b/topics/quality-control/ui-design.gmi new file mode 100644 index 0000000..84f7748 --- /dev/null +++ b/topics/quality-control/ui-design.gmi @@ -0,0 +1,41 @@ +# UI Design + +1. Input/Receive Data in UI (drag and drop/upload submit form) + +2. Select Mouse + +"What type of Group are you using?" + +> (AKXD, BXH, Mouse Diversity Panel, BXD) + +3. "What is your platform?" + +> (Aff, Ilumina, ...) + +If Affymetrix (Aff) is selected then there should be various options +like Clarion S. + +If the platform you chose is not available: + + Tell PI that they should solicit for their platform to be added to the list. + + They can contact us via email. + +4. Allow excel file upload? + +## More Example UI Interactions and Checks + +"If your dataset does not comply with GN then you can try uploading your +dataset so that we can inspect it." + +"Your dataset has two erroneous entries: Gene Accession Gene." + +"The last two columns have the wrong format for the strain name." + +"Here's our format of how your dataset should look like." + +> ProbeSetID Strains ... + +"Inbred Set ID 1 is the same as BXD" + +> These are the strains: ... diff --git a/topics/systems/machine-room.gmi b/topics/systems/machine-room.gmi index bece333..d75883a 100644 --- a/topics/systems/machine-room.gmi +++ b/topics/systems/machine-room.gmi @@ -6,9 +6,10 @@ ## Tasks -* [ ] Make tux02e visible from outside -* [ ] Network switch 10Gbs -* [ ] Add disks to tux01 - Sep 28 To Brock +* [X] Make tux02e visible from outside +* [ ] Network switch 10Gbs - add hosts +* [ ] Add disks to tux01 and tux02 - need to reboot +* [ ] Set up E-mail relay for tux01 and tux02 smtp.uthsc.edu, port 25 => tux02-production.gmi setup new production machine => decommission-machines.gmi Decommission machines diff --git a/topics/systems/sheepdog.gmi b/topics/systems/sheepdog.gmi index 478adf1..137a5dd 100644 --- a/topics/systems/sheepdog.gmi +++ b/topics/systems/sheepdog.gmi @@ -13,6 +13,7 @@ that need resolving at some point. ## Tasks * [X] add locking functionality for tags - added borg with-lock (test) +* [X] chgrp functionality in sheepdog_borg * [ ] check whether rsync dir exists, repo valid and/or no lock before proceeding * [ ] send digest E-mails * [ ] smart state E-mails on services going down diff --git a/topics/testing/genenetwork-checklist.gmi b/topics/testing/genenetwork-checklist.gmi new file mode 100644 index 0000000..ba19733 --- /dev/null +++ b/topics/testing/genenetwork-checklist.gmi @@ -0,0 +1,59 @@ +# GN Checklist + +This is the checklist to run through when testing GN2. + +## Tags + +* assigned: pjotrp + +## Tasks + +This is a check list for testing whether GN2 is completely deployed + +* [ ] visit main page +=> https://luna.genenetwork.org/ +* [ ] simple search (e.g. shh in hippocampus mRNA) +=> https://luna.genenetwork.org/search?species=mouse&group=BXD&type=Hippocampus+mRNA&dataset=HC_M2_0606_P&search_terms_or=shh&search_terms_and=&FormID=searchResult +* [ ] try your own search term +* [ ] wild card seach (e.g. sh* in hippocampus mRNA) +=> https://luna.genenetwork.org/search?species=mouse&group=BXD&type=Hippocampus+mRNA&dataset=HC_M2_0606_P&search_terms_or=sh*&search_terms_and=&FormID=searchResult +* [ ] try your own wild-card search +* [ ] visit a trait page +=> https://luna.genenetwork.org/show_trait?trait_id=1427571_at&dataset=HC_M2_0606_P + + +will continue... + + +curl -X POST -b 'anon_user_v1=4d2ce6d3-0d4c-4349-b491-a4e179fdc163:d5fdc040c737e2a775d2' https://luna.genenetwork.org/run_mapping + +https://luna.genenetwork.org/run_mapping +User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0 +Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 +Accept-Language: en-US,en;q=0.5 +Accept-Encoding: gzip, deflate, br +Content-Type: application/x-www-form-urlencoded +Content-Length: 12345 +Origin: https://luna.genenetwork.org +Connection: keep-alive +Referer: https://luna.genenetwork.org/loading +Cookie: anon_user_v1=4d2ce6d3-0d4c-4349-b491-a4e179fdc163:d5fdc040c737e2a775d2; 404 Not Found: The requested URL=Wild-Type-Mouse.gif +Upgrade-Insecure-Requests: 1 +Sec-GPC: 1 + +https://luna.genenetwork.org/run_mapping +POST /run_mapping HTTP/1.1 +Host: luna.genenetwork.org +User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0 +Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 +Accept-Language: en-US,en;q=0.5 +Accept-Encoding: gzip, deflate, br +Referer: https://luna.genenetwork.org/loading +Content-Type: application/x-www-form-urlencoded +Content-Length: 12345 +Origin: https://luna.genenetwork.org +Connection: keep-alive +Cookie: anon_user_v1=4d2ce6d3-0d4c-4349-b491-a4e179fdc163:d5fdc040c737e2a775d2; 404 Not Found: The requested URL=Wild-Type-Mouse.gif +Upgrade-Insecure-Requests: 1 +Sec-GPC: 1 +Cache-Control: max-age=0 |