Merge branch 'main' of github.com:genenetwork/gn-gemtext-threads

14 files changed, 325 insertions, 3 deletions

diff --git a/topics/arvados/covid19.gmi b/topics/arvados/covid19.gmi
new file mode 100644
index 0000000..be96605
--- /dev/null
+++ b/topics/arvados/covid19.gmi
@@ -0,0 +1,7 @@
+# COVID-19
+
+We are setting up a Cloud HPC for COVID-19 using Arvados as the underlying infrastructure.
+
+## Tasks
+
+=> https://github.com/yatisht/usher Fast phylogeny
diff --git a/topics/cli/weechat.gmi b/topics/cli/weechat.gmi
new file mode 100644
index 0000000..b6fc062
--- /dev/null
+++ b/topics/cli/weechat.gmi
@@ -0,0 +1,66 @@
+# Weechat
+
+Weechat is a very versatile terminal chat that can handle IRC, Matrix and Slack.
+
+## Weechat install + slack
+
+Arun has packaged weechat for slack and matrix in GNU Guix. Current install using a guix environment:
+
+```
+env GUIX_PACKAGE_PATH=~/guix-bioinformatics/ ~/opt/guix-latest/bin/guix environment --ad-hoc coreutils weechat weechat-matrix weechat-wee-slack nss-certs openssl python
+echo $GUIX_ENVIRONMENT
+/gnu/store/i4a3ynfjvq8r8cch14gkn2fwfs77wagj-profile/
+weechat
+/python load /gnu/store/i4a3ynfjvq8r8cch14gkn2fwfs77wagj-profile/share/weechat/python/wee_slack.py
+/slack register
+```
+
+So, capture the shell GUIX_ENVIRONMENT and use that to launch the wee-slack plugin. Next register a token through te Slack web interface.
+
+If you go through the motions you get a token and
+
+```
+/slack register 2654233192084.2682932906197.fd6d6e55c0038f-etc-052b4fa578ec497710b4b8770f1-etc
+/python reload slack
+```
+
+More instructions for https://github.com/wee-slack/wee-slack.
+
+## Weechat IRC
+
+We have a GeneNetwork channel
+
+```
+/set irc.server_default.nicks pjotrp
+/server add libera irc.libera.chat/6697 -ssl -autoconnect
+/connect libera
+/join #guix
+/join #guix-risc-v
+/join #guix-hpc
+/join #genenetwork
+```
+
+## Weechat matrix
+
+The matrix plugin works similar to above Slack plugin.
+
+```
+/script load matrix.py
+/matrix server add fosdem chat.fosdem.org
+/set matrix.server.matrix_org.username pjotrp-linux
+/set matrix.server.matrix_org.password ****
+/set matrix.server.matrix_org.autoconnect on
+/save
+```
+
+Only the matrix version is not so polished. For example, it is hard to find the buffers because they are named by a hash. In a buffer you can rename with
+
+```
+/buffer set short_name myname
+```
+
+Also, you need to enable every channel explicitly to exchange keys
+
+```
+/olm verify @user:matrix.org *
+```
diff --git a/topics/data-uploads/editing-data.gmi b/topics/data-uploads/editing-data.gmi
index 1bc1bf8..01144a0 100644
--- a/topics/data-uploads/editing-data.gmi
+++ b/topics/data-uploads/editing-data.gmi
@@ -1,3 +1,7 @@
+## Tags
+
+* assigned: bonfacem
+
 ### Introduction
 
 At the moment, you can edit metadata related to a published phenotype and a probeset. When an edit is done, the diff data is stored in a table, `metadata_audit` in json format that looks something like:
@@ -60,3 +64,9 @@ After editing traits, you could see changes in a diff format at the very top of
 That needs to be improved.
 
 - For Probeset data, atm some tables aren't yet updated(unfortunately those tables aren't used anywhere). I need to figure that out.
+
+#### Tue 23 Nov 2021
+
+- Fixed excel issue described here:
+
+=> https://github.com/genenetwork/genenetwork2/compare/testing...BonfaceKilz:bug/fix-excel-adding-new-line?expand=1
\ No newline at end of file
diff --git a/topics/documentation/gn-documentation.gmi b/topics/documentation/gn-documentation.gmi
new file mode 100644
index 0000000..653a1a4
--- /dev/null
+++ b/topics/documentation/gn-documentation.gmi
@@ -0,0 +1,21 @@
+# GeneNetwork documentation
+
+GeneNetwork requires an online 'wiki' and a place for documentation. Both of these can be maintained in markdown and/or gemini gemtext. We are using a repo:
+
+=> https://github.com/genenetwork/gn-docs
+
+What we need to do is render documents from markdown (as is done with the facilities page)
+
+=> https://github.com/genenetwork/gn-docs/blob/master/general/help/facilities.md
+
+the markdown is rendered on github, which makes it easy to edit for most users, so it should be the default.
+
+## Tags
+
+* assigned: pjotrp
+
+## Tasks
+
+=> storyboards
+
+## Info
diff --git a/topics/gemini/cheatsheet.gmi b/topics/gemini/cheatsheet.gmi
new file mode 100644
index 0000000..b2158ea
--- /dev/null
+++ b/topics/gemini/cheatsheet.gmi
@@ -0,0 +1,5 @@
+# gemtext cheatsheet
+
+=> gemini://gemini.circumlunar.space/docs/cheatsheet.gmi
+
+=> https://gemini.circumlunar.space/docs/cheatsheet.gmi proxy
diff --git a/topics/gemini/ideas_and_references.gmi b/topics/gemini/ideas_and_references.gmi
new file mode 100644
index 0000000..dfb026d
--- /dev/null
+++ b/topics/gemini/ideas_and_references.gmi
@@ -0,0 +1,3 @@
+Ideas and References
+
+=> gemini://drewdevault.com/2021/11/15/Reframing-gemini.gmi Reframing the philosophy of Gemini
diff --git a/topics/pudb/pytest-pudb.gmi b/topics/pudb/pytest-pudb.gmi
new file mode 100644
index 0000000..5a979c0
--- /dev/null
+++ b/topics/pudb/pytest-pudb.gmi
@@ -0,0 +1,7 @@
+# Using pudb with pytest
+
+```
+pytest --pdbcls pudb.debugger:Debugger --pdb --capture=no
+```
+
+=> https://documen.tician.de/pudb/starting.html?highlight=pytest#usage-with-pytest 
diff --git a/topics/pudb/using-pudb.gmi b/topics/pudb/using-pudb.gmi
new file mode 100644
index 0000000..877a700
--- /dev/null
+++ b/topics/pudb/using-pudb.gmi
@@ -0,0 +1,21 @@
+# Using pudb
+
+## Setting Breakpoints
+
+You can add the following line anywhere in your code to start a pudb breakpoint:
+
+```
+import pudb; pudb.set_trace()
+```
+
+You can also press `b` in pudb to add a breakpoint with a keyboard shortcut.
+
+## pdb and pudb API similarity
+
+The standard pdb docs should work for pudb with one small exception.
+
+> At the programming language level, PuDB displays the same interface
+> as Python’s built-in pdb module. Just replace pdb with pudb. (One
+> exception: run is called runstatement.)
+
+=> https://docs.python.org/3/library/pdb.html?highlight=pdb#module-pdb
diff --git a/topics/quality-control/qc-checks.gmi b/topics/quality-control/qc-checks.gmi
new file mode 100644
index 0000000..0d04625
--- /dev/null
+++ b/topics/quality-control/qc-checks.gmi
@@ -0,0 +1,47 @@
+# Quality Control Checks
+
+1. ProbeSetId (Affymetrix format):
+
+We favour using Illumina, Affimetrix, and other platform formats.
+
+Custom formats require a new annotation file to be created.
+
+We usually use Ensemble ID or Gene IDs.
+
+1.1 Ensemble transcript IDs usually have duplicates that need to be pruned.
+
+ENSMBL1234
+
+## Example Gene Symbol to ProbeSetId
+
+AFFX-BkGr-GC03_st -> TCO500002136.mm.2
+
+2. Inbred Strain names should prefer long form:
+
+B6 -> C57BL/6
+D2 -> DBA/2
+
+3. Probeset IDs that don't have any values should be pruned:
+
+For example an Affymetrix data set might have ~28,000 entries and the data set that
+is allowed into the GeneNetwork will be 22,000 entries.
+
+4. The standard error between male and female mice has to be computed.
+
+5. SE values have to be computed to 6 or greater decimal places.
+
+6. The average between male and female mice has to be computed to 3 decimal places.
+
+7. Datasets/studies having the same ProbeSetID should be grouped together.
+
+8. There should be no trailing spaces in data cells.
+
+9. Entries should have the same capitalization style.
+
+10. Assesing Phenotypes for normality with Shapiro-Wilk Test.
+
+11. Check for annotations file.
+
+12. Check for CRLF.
+
+13. Check for UTF-8 encoding.
diff --git a/topics/quality-control/shapiro-wilk-test.gmi b/topics/quality-control/shapiro-wilk-test.gmi
new file mode 100644
index 0000000..b1bdbfd
--- /dev/null
+++ b/topics/quality-control/shapiro-wilk-test.gmi
@@ -0,0 +1,33 @@
+# Shapiro Wilk Test
+
+This document contains more info about QC step 11.
+
+https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html
+https://vedexcel.com/how-to-perform-a-shapiro-wilk-test-in-python/
+
+```
+Here is a simple QC procedure we may want to consider that was used by
+Megan and Camron in a recent paper  that I have attached.
+
+
+Phenotypes were assessed for normality using the Shapiro–Wilk Test. Because
+some of the data residuals devi- ated significantly from normality, we used
+the orderNorm function to perform Ordered Quantile normalization43 on all
+phenotypes.
+
+--
+Rob
+```
+
+```
+QTL analysis was performed in F2 mice using the R package R/qtl
+(RRID:SCR_009085) as previously described.11,29,42 Quality checking
+of genotypes and QTL analysis were performed in R (https://www.r-
+project.org/) using R/bestNormalize (https://github.com/petersonR/
+bestNormalize) and R/qtl.42 Phenotypes were assessed for normality
+using the Shapiro–Wilk Test. Because some of the data residuals devi-
+ated significantly from normality, we used the orderNorm function to
+perform Ordered Quantile normalization43 on all phenotypes. QTL
+--
+A quantitative trait variant in Gabra2 underlies increased methamphetamine stimulant sensitivity
+```
diff --git a/topics/quality-control/ui-design.gmi b/topics/quality-control/ui-design.gmi
new file mode 100644
index 0000000..84f7748
--- /dev/null
+++ b/topics/quality-control/ui-design.gmi
@@ -0,0 +1,41 @@
+# UI Design
+
+1. Input/Receive Data in UI (drag and drop/upload submit form)
+
+2. Select Mouse
+
+"What type of Group are you using?" 
+
+> (AKXD, BXH, Mouse Diversity Panel, BXD)
+
+3. "What is your platform?"
+
+> (Aff, Ilumina, ...)
+
+If Affymetrix (Aff) is selected then there should be various options
+like Clarion S.
+
+If the platform you chose is not available:
+
+  Tell PI that they should solicit for their platform to be added to the list.
+  
+  They can contact us via email.
+
+4. Allow excel file upload?
+
+## More Example UI Interactions and Checks
+
+"If your dataset does not comply with GN then you can try uploading your
+dataset so that we can inspect it."
+
+"Your dataset has two erroneous entries: Gene Accession Gene."
+
+"The last two columns have the wrong format for the strain name."
+
+"Here's our format of how your dataset should look like."
+
+> ProbeSetID Strains ...
+
+"Inbred Set ID 1 is the same as BXD"
+
+> These are the strains: ...
diff --git a/topics/systems/machine-room.gmi b/topics/systems/machine-room.gmi
index bece333..d75883a 100644
--- a/topics/systems/machine-room.gmi
+++ b/topics/systems/machine-room.gmi
@@ -6,9 +6,10 @@
 
 ## Tasks
 
-* [ ] Make tux02e visible from outside
-* [ ] Network switch 10Gbs
-* [ ] Add disks to tux01 - Sep 28 To Brock
+* [X] Make tux02e visible from outside
+* [ ] Network switch 10Gbs - add hosts
+* [ ] Add disks to tux01 and tux02 - need to reboot
+* [ ] Set up E-mail relay for tux01 and tux02 smtp.uthsc.edu, port 25
 
 => tux02-production.gmi setup new production machine
 => decommission-machines.gmi Decommission machines
diff --git a/topics/systems/sheepdog.gmi b/topics/systems/sheepdog.gmi
index 478adf1..137a5dd 100644
--- a/topics/systems/sheepdog.gmi
+++ b/topics/systems/sheepdog.gmi
@@ -13,6 +13,7 @@ that need resolving at some point.
 ## Tasks
 
 * [X] add locking functionality for tags - added borg with-lock (test)
+* [X] chgrp functionality in sheepdog_borg
 * [ ] check whether rsync dir exists, repo valid and/or no lock before proceeding
 * [ ] send digest E-mails
 * [ ] smart state E-mails on services going down
diff --git a/topics/testing/genenetwork-checklist.gmi b/topics/testing/genenetwork-checklist.gmi
new file mode 100644
index 0000000..ba19733
--- /dev/null
+++ b/topics/testing/genenetwork-checklist.gmi
@@ -0,0 +1,59 @@
+# GN Checklist
+
+This is the checklist to run through when testing GN2.
+
+## Tags
+
+* assigned: pjotrp
+
+## Tasks
+
+This is a check list for testing whether GN2 is completely deployed
+
+* [ ] visit main page
+=> https://luna.genenetwork.org/
+* [ ] simple search (e.g. shh in hippocampus mRNA)
+=> https://luna.genenetwork.org/search?species=mouse&group=BXD&type=Hippocampus+mRNA&dataset=HC_M2_0606_P&search_terms_or=shh&search_terms_and=&FormID=searchResult
+* [ ] try your own search term
+* [ ] wild card seach (e.g. sh* in hippocampus mRNA)
+=> https://luna.genenetwork.org/search?species=mouse&group=BXD&type=Hippocampus+mRNA&dataset=HC_M2_0606_P&search_terms_or=sh*&search_terms_and=&FormID=searchResult
+* [ ] try your own wild-card search
+* [ ] visit a trait page
+=> https://luna.genenetwork.org/show_trait?trait_id=1427571_at&dataset=HC_M2_0606_P
+
+
+will continue...
+
+
+curl -X POST -b 'anon_user_v1=4d2ce6d3-0d4c-4349-b491-a4e179fdc163:d5fdc040c737e2a775d2' https://luna.genenetwork.org/run_mapping
+
+https://luna.genenetwork.org/run_mapping
+User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
+Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
+Accept-Language: en-US,en;q=0.5
+Accept-Encoding: gzip, deflate, br
+Content-Type: application/x-www-form-urlencoded
+Content-Length: 12345
+Origin: https://luna.genenetwork.org
+Connection: keep-alive
+Referer: https://luna.genenetwork.org/loading
+Cookie: anon_user_v1=4d2ce6d3-0d4c-4349-b491-a4e179fdc163:d5fdc040c737e2a775d2; 404 Not Found: The requested URL=Wild-Type-Mouse.gif
+Upgrade-Insecure-Requests: 1
+Sec-GPC: 1
+
+https://luna.genenetwork.org/run_mapping
+POST /run_mapping HTTP/1.1
+Host: luna.genenetwork.org
+User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
+Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
+Accept-Language: en-US,en;q=0.5
+Accept-Encoding: gzip, deflate, br
+Referer: https://luna.genenetwork.org/loading
+Content-Type: application/x-www-form-urlencoded
+Content-Length: 12345
+Origin: https://luna.genenetwork.org
+Connection: keep-alive
+Cookie: anon_user_v1=4d2ce6d3-0d4c-4349-b491-a4e179fdc163:d5fdc040c737e2a775d2; 404 Not Found: The requested URL=Wild-Type-Mouse.gif
+Upgrade-Insecure-Requests: 1
+Sec-GPC: 1
+Cache-Control: max-age=0