diff options
41 files changed, 917 insertions, 12 deletions
diff --git a/issues/authorisation.gmi b/issues/authorisation.gmi index 044962d..0adac56 100644 --- a/issues/authorisation.gmi +++ b/issues/authorisation.gmi @@ -2,13 +2,28 @@ Look at Chris' authorisation code: -=> https://github.com/chfi/gn-proxy +=> https://github.com/genenetwork/gn-proxy Atm genenetwork2's handling of authorisation is a bit clunky; Since @Bonface is working on the data data up-loader, and it requires authorisation, this is a good time to clean up authorisation. +# Tags + +* assigned: bonfacem, pjotrp, zachs + +# TODO + +- [x] Make user able to delete/ add extra value + +- [x] Fetch complete list of samples from database and genotype file + instead of only fetching that list from the database. Look at trait + page for reference. + +- [ ] Extend idea of csv generation to probeset data. Get data from + the genotype file and work out how to update the values from the + database. # Notes @@ -111,4 +126,4 @@ On the UI(quote from Zach): to the show trait page. Key point is that I changed the URLs and moved them to their own module. Particularly for published datasets. -- New PR: https://github.com/genenetwork/genenetwork2/pull/614
\ No newline at end of file +- New PR: https://github.com/genenetwork/genenetwork2/pull/614 diff --git a/issues/binderlite/add-configuration-to-set-server-name.gmi b/issues/binderlite/add-configuration-to-set-server-name.gmi new file mode 100644 index 0000000..d54997c --- /dev/null +++ b/issues/binderlite/add-configuration-to-set-server-name.gmi @@ -0,0 +1,11 @@ +# Add configuration to set server name + +* assigned: jgart + +* feature + +Let's use environment variables as configuration + +https://flask.palletsprojects.com/en/2.0.x/cli/?highlight=environment%20variables + +https://docs.python.org/3/library/os.html?highlight=os%20environ#os.environ diff --git a/issues/binderlite/add-spinner-for-submit-form.gmi b/issues/binderlite/add-spinner-for-submit-form.gmi index 0800fa8..de19c55 100644 --- a/issues/binderlite/add-spinner-for-submit-form.gmi +++ b/issues/binderlite/add-spinner-for-submit-form.gmi @@ -13,3 +13,15 @@ a container to be built? A spinning guix gnu logo could be cool. https://www.w3schools.com/howto/howto_css_loader.asp https://spin.js.org/https://spin.js.org/ + + +## Solution + +We're using ansi2html now. + +See the following issue: + +=> ./ansi2html_terminal_user_feedback.gmi + +* closed + diff --git a/issues/binderlite/ansi2html_terminal_user_feedback.gmi b/issues/binderlite/ansi2html_terminal_user_feedback.gmi new file mode 100644 index 0000000..969448a --- /dev/null +++ b/issues/binderlite/ansi2html_terminal_user_feedback.gmi @@ -0,0 +1,36 @@ +# ansi2html terminal user feedback + +* ongoing +* feature-request +* assigned: jgart + +https://builds.sr.ht/~sircmpwn/job/610711 + +https://builds.sr.ht/~sircmpwn/job/611028 + +This issue proposes an alternative way to generate terminal output to the browser +from a process that does not dependent on js libraries but still looks great. + +Drew Devault uses this method in builds.sr.ht, the continuous integration +system/app that is part of the SourceHut suite. + +builds.sr.ht generates terminal output using jinja templates and a relatively +small python library called ansi2html. It is already packaged by guix upstream +as python-ansi2html (It could use an update). + +https://github.com/pycontribs/ansi2html +https://pypi.org/project/ansi2html/ + +Here are two examples of it in action at builds.sr.ht: + +https://builds.sr.ht/~sircmpwn/job/610711 + +https://builds.sr.ht/~sircmpwn/job/611028 + +And, the relevant places in builds.sr.ht code base: + +https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/master/item/buildsrht/templates/job.html#L117 + +https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/master/item/buildsrht/blueprints/jobs.py#L333 + +I've added a diagram showing how it will integrate with binderlite. diff --git a/issues/binderlite/binderlite_ansi2html_terminal_diagram.png b/issues/binderlite/binderlite_ansi2html_terminal_diagram.png Binary files differnew file mode 100644 index 0000000..a4f16b4 --- /dev/null +++ b/issues/binderlite/binderlite_ansi2html_terminal_diagram.png diff --git a/issues/binderlite/error-regarding-static-assets-after-forwarding.gmi b/issues/binderlite/error-regarding-static-assets-after-forwarding.gmi index 27e7bd1..bd39df9 100644 --- a/issues/binderlite/error-regarding-static-assets-after-forwarding.gmi +++ b/issues/binderlite/error-regarding-static-assets-after-forwarding.gmi @@ -50,3 +50,5 @@ http://localhost/49761/?token=f6269bbf48ee5c9887732f5bc76d32f37e1f25483195ae88 The next step is to get jupyter to accept the generated token on the first request without needing to input it manually first. + +* closed diff --git a/issues/binderlite/integrate-styling-with-gn2.gmi b/issues/binderlite/integrate-styling-with-gn2.gmi new file mode 100644 index 0000000..dc158fa --- /dev/null +++ b/issues/binderlite/integrate-styling-with-gn2.gmi @@ -0,0 +1,11 @@ +# Integrate styling of app with GN2 + +* ongoing +* assigned: jgart + +Currently using antiweb for styling with modifications. + +=> https://git.sr.ht/~sircmpwn/antiweb antiweb + +Copying nav bar and fonts from GN2 will make the app look more integrated with +GN2 UI. diff --git a/issues/binderlite/open-files-directly-if-full-path-is-given.gmi b/issues/binderlite/open-files-directly-if-full-path-is-given.gmi new file mode 100644 index 0000000..203e87a --- /dev/null +++ b/issues/binderlite/open-files-directly-if-full-path-is-given.gmi @@ -0,0 +1,18 @@ +# Open notebook directly if full path to notebook file is given + +* feature-request +* assigned: jgart + +We'd like to be able launch a notebook directly if the user gives the fullpath to the notebook file. + +Should we allow urls like this? + +https://github.com/jgarte/guile-notebook-genenetwork-api/blob/master/genenetwork-api.ipynb + +The above is a valid github url that also points to the notebook file. + +Or only this? + +https://github.com/jgarte/guile-notebook-genenetwork-api/genenetwork-api.ipynb + +* closed diff --git a/issues/binderlite/return-canned-text-if-notebook-description-returns-none.gmi b/issues/binderlite/return-canned-text-if-notebook-description-returns-none.gmi new file mode 100644 index 0000000..49f5d3f --- /dev/null +++ b/issues/binderlite/return-canned-text-if-notebook-description-returns-none.gmi @@ -0,0 +1,25 @@ +# Return canned text if notebook description returns none + +* bug? +* assigned: jgart + +Notebooks that do not have a description will use python's `None` value as the +project's description. + +Should we return some other text instead? + +We could do something like the following idea: + +``` +if notebook.description == None: + return "Project does not have a description." +``` + +Or we could do: + +``` +if notebook.description == None: + return "Could not retrieve description for project." +``` + +WDYT? diff --git a/issues/database-not-responding.gmi b/issues/database-not-responding.gmi index c8e752b..e66c939 100644 --- a/issues/database-not-responding.gmi +++ b/issues/database-not-responding.gmi @@ -1,4 +1,136 @@ -# Mariadb table locked +# Hanging database + +Mariadb occassionally stops responding. + +## Tasks + +* assigned: pjotrp, zsloan +* bug + +# Info + +## Mariadb is 'hanging' + +In the last 12 hours GN2 monitoring shows the website is responding intermittendly. A quick check shows the database is blocking. Rather than simply restarting the database - which is known to sort the issue - the timing is that the US is sleeping so I can do some checking. Let's take a look. + +Mariadb is at 4x CPU + +``` +PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND + 40514 mysql 20 0 37.3g 1.8g 18268 S 394.4 0.7 4855:33 mysqld +``` + +The ps table shows a backup is ongoing + +``` +root 57559 0.0 0.0 2388 756 ? Ss 03:00 0:00 /bin/sh -c /bin/su mysql -c /export/backup/scripts/tux01/backup_mariadb.sh >> ~/cron.log 2>&1 +mysql 57588 0.2 0.0 200112 27292 ? Sl 03:00 0:25 mariabackup --backup --target-dir=/home/backup/tux01_mariadb_new/latest/ --user=webqtlout --password=x xxxxxxx +``` + +Tales in use: + +``` +MariaDB [db_webqtl]> show open tables where in_use > 1; ++-----------+----------------+--------+-------------+ +| Database | Table | In_use | Name_locked | ++-----------+----------------+--------+-------------+ +| db_webqtl | InfoFiles | 5 | 0 | +| db_webqtl | GeneRIF_BASIC | 3 | 0 | +| db_webqtl | ProbeFreeze | 4 | 0 | +| db_webqtl | ProbeSet | 4 | 0 | +| db_webqtl | ProbeSetXRef | 4 | 0 | +| db_webqtl | Species | 4 | 0 | +| db_webqtl | Geno | 4 | 0 | +| db_webqtl | Chr_Length | 2 | 0 | +| db_webqtl | Tissue | 4 | 0 | +| db_webqtl | ProbeSetFreeze | 4 | 0 | +| db_webqtl | InbredSet | 4 | 0 | ++-----------+----------------+--------+-------------+ +11 rows in set (0.001 sec) +``` + +In the error log we seeing a lot of + +``` +2021-12-21 6:17:57 256179 [Warning] Aborted connection 256179 to db: 'db_webqtl' user: 'webqtlout' host: '128.169.5.59' (Got timeout reading communication packets) +``` + +``` +SHOW FULL PROCESSLIST; +458 rows in set (0.003 sec) +``` + +with entries + +``` +| 256363 | webqtlout | 128.169.5.59:59120 | db_webqtl | Query | 15 | Waiting for table flush | SELECT Id, Name, FullName, ShortName, DataScale FROM ProbeSetFreeze WHERE public > 0 AND (Name = "CB_M_0305_R" OR FullName = "CB_M_0305_R" OR ShortName = "CB_M_0305_R") +``` + +waiting for tables to flush! + +at the top of the process list we find + +``` +Id User Host db Command Time State Info Progress 1 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000 2 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 3 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 5 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000 227365 webqtlout 127.0.0.1:33950 db_webqtl Sleep 13015 NULL 0.000 245634 webqtlout 127.0.0.1:38098 db_webqtl Sleep 23180 NULL 0.000 +``` + +This is quite informative: + +=> https://programmer.group/analysis-of-mysql-process-in-waiting-for-table-flush.html + +it suggests that the backup can be the root of the problem. + +And then + +=> https://www.thegeekdiary.com/troubleshooting-mysql-query-hung-waiting-for-table-flush/ + +suggests + +* Wait for the long-running queries which are blocking the FLUSH TABLE to complete; +* Identify the long-running queries and kill them; +* Restart the server + +and that is somewhat amusing. + +Stripping out all reqular queries we get: + +``` +grep localhost test.out |grep -vi probesetfreeze|grep -vi species +255559 webqtlout localhost NULL Query 13092 Waiting for table flush FLUSH NO_WRITE_TO_BINLOG TABLES0.000 +256351 webqtlout localhost db_webqtl Field List 1588 Waiting for table flush NULL 0.000 +256383 webqtlout localhost db_webqtl Query 0 Init SHOW FULL PROCESSLIST 0.000 +``` + +and it appears everyone is waiting for id 255559. Let's kill that. + +``` +kill 255559; +``` + +and inspect + +``` +mysql -u webqtlout -pwebqtlout db_webqtl -A -e "show processlist;"|less +``` + +it started processing again. To speed up recovery back to: + +``` +systemctl restart mysql +``` + +of course that stopped the running backup. But processing is back in business. + +My first conclusion is that this problem was triggered by the backup procedure. Interestingly, it happens irregularly. We also have seen this issue before this backup procedure was instated, so I figure it has to do with Mariadb. + +The version on production is from 2017 - we should update that soon: + +Server version: 10.3.27-MariaDB-0+deb10u1-log Debian 10 + +we have been running a more recent version of mariadb on luna. Still, that is unlikely to fix this issue because I think it really has to do with myisam and locking of large tables. Switching to innodb does away with global locks and is the default on mariadb (there are less and less people using myisam). + + +## Mariadb table locked Arthur reports: MariaDB is not responding Saturday, July 24 2021 at 10:48 pm. I tried to enter data to the table ProbeSetXRef.pValue and when normally takes few seconds, now is more than 10 minutes without completion/responding. diff --git a/issues/deploy.gmi b/issues/deploy.gmi new file mode 100644 index 0000000..94d73ec --- /dev/null +++ b/issues/deploy.gmi @@ -0,0 +1,26 @@ +# Handle Configuration Files + +How we handle our configuration is a mess; and this makes deployment a +hassle. In genenetwork2, our configs live in different places, and it +takes some fiddling tinged with a bit of domain (gn2) knowledge to +work out what setting live where. With the introduction of blueprints +in our code-base, this duplication became a necessary evil because +most of the settings QC checks happen in a "utility/tools.py" which +unfortunately imports the "app" object. This import leads to circular +imports whenever we use blueprints. Unfortunately, a huge chunk of +the code base uses "utility/tools.py" to get configuration +settings; and removing this would make gn2 as is disfunctional. + +# Tags + +* assigned: bonfacem + +# Notes + +Here's an example of how to use json as our settings: + +=> https://stackoverflow.com/questions/15122312/how-to-import-from-config-file-in-flask + +Here's Flask's official documentation on handling configuration: + +=> https://flask.palletsprojects.com/en/1.1.x/config/ diff --git a/issues/fix-broken-builds-in-gn2-and-gn3.gmi b/issues/fix-broken-builds-in-gn2-and-gn3.gmi index cf9506f..3c0e6aa 100644 --- a/issues/fix-broken-builds-in-gn2-and-gn3.gmi +++ b/issues/fix-broken-builds-in-gn2-and-gn3.gmi @@ -5,7 +5,55 @@ gn3 and gn2. ## Tasks -* assigned: bonfacekilz +* assigned: bonfacekilz, efraim -* [ ] Fix pylint/ mypy issues in gn3 -* [ ] Update docker image in gn2 +### Notes + +At the moment, we are getting the following build failures in gn2, + +``` +Compiling '/gnu/store/bm0909ha5nv5chgp3bk5fw0my6mrsabr-genenetwork2-3.11-guix-fe345c9/lib/python3.9/site-packages/wqflask/wqflask/wgcna/wgcna_analysis.py'... +Compiling '/gnu/store/bm0909ha5nv5chgp3bk5fw0my6mrsabr-genenetwork2-3.11-guix-fe345c9/lib/python3.9/site-packages/wqflask/wsgi.py'... +error: in phase 'install': uncaught exception: +%exception #<&invoke-error program: "python" arguments: ("-m" "compileall" "--invalidation-mode=unchecked-hash" "/gnu/store/bm0909ha5nv5chgp3bk5fw0my6mrsabr-genenetwork2-3.11-guix-fe345c9") exit-status: 1 term-signal: #f stop-signal: #f> +phase `install' failed after 0.9 seconds +command "python" "-m" "compileall" "--invalidation-mode=unchecked-hash" "/gnu/store/bm0909ha5nv5chgp3bk5fw0my6mrsabr-genenetwork2-3.11-guix-fe345c9" failed with status 1 +note: keeping build directory `/tmp/guix-build-genenetwork2-3.11-guix-fe345c9.drv-0' +builder for `/gnu/store/l5hb8bfz3di3zga93963arlka03lijn8-genenetwork2-3.11-guix-fe345c9.drv' failed with exit code 1 +build of /gnu/store/l5hb8bfz3di3zga93963arlka03lijn8-genenetwork2-3.11-guix-fe345c9.drv failed +View build log at '/var/log/guix/drvs/l5/hb8bfz3di3zga93963arlka03lijn8-genenetwork2-3.11-guix-fe345c9.drv.bz2'. +guix build: error: build of `/gnu/store/l5hb8bfz3di3zga93963arlka03lijn8-genenetwork2-3.11-guix-fe345c9.drv' failed + +``` + +Some missing bits from the build log: + +``` +Compiling '/gnu/store/n2w3bdka4zfl3jw0iqk49km1hgl8i7g5-genenetwork2-3.11-guix-fe345c9/lib/python3.9/site-packages/wqflask/utility/redis_tools.py'... +Compiling '/gnu/store/n2w3bdka4zfl3jw0iqk49km1hgl8i7g5-genenetwork2-3.11-guix-fe345c9/lib/python3.9/site-packages/wqflask/utility/startup_config.py'... +Compiling '/gnu/store/n2w3bdka4zfl3jw0iqk49km1hgl8i7g5-genenetwork2-3.11-guix-fe345c9/lib/python3.9/site-packages/wqflask/utility/svg.py'... +*** File "/gnu/store/n2w3bdka4zfl3jw0iqk49km1hgl8i7g5-genenetwork2-3.11-guix-fe345c9/lib/python3.9/site-packages/wqflask/utility/svg.py", line 113 + True = 1 + ^ + SyntaxError: cannot assign to True + +Compiling '/gnu/store/n2w3bdka4zfl3jw0iqk49km1hgl8i7g5-genenetwork2-3.11-guix-fe345c9/lib/python3.9/site-packages/wqflask/utility/temp_data.py'... +Compiling '/gnu/store/n2w3bdka4zfl3jw0iqk49km1hgl8i7g5-genenetwork2-3.11-guix-fe345c9/lib/python3.9/site-packages/wqflask/utility/tools.py'... + +``` + +This starts happening (not yet comfirmed) when python is bumped up +from 3.8 to 3.9 + +In addition, there are some dependencies of genenetwork2 which FTBFS, due to incorrect versions of inputs. + +``` + starting phase `sanity-check'¬ + validating 'pingouin' /gnu/store/dpy5ln4zjgv53l8jpm1qalwn45cagxfm-python-pingouin-0.5.0/lib/python3.9/site-packages¬ + ...checking requirements: ERROR: pingouin==0.5.0 (statsmodels 0.11.1 (/gnu/store/ab8k9bwmaml2bplq4bd9xkg08z7bn8fr-python-statsmodels-0.11.1/lib/python3.9/site-packages), Requirement.parse('statsmodels>=0.12.0'), {'pingouin'})¬ + error: in phase 'sanity-check': uncaught exception:¬ + %exception #<&invoke-error program: "python" arguments: ("/gnu/store/nwwr89v2vyg1hs48i49m083vhczsgh3m-sanity-check.py" "/gnu/store/dpy5ln4zjgv53l8jpm1qalwn45cagxfm-python-pingouin-0.5.0/lib/python3.9/site-packages") exit-status: 1 term-signal: #f stop-signal: #f>▸¬ + phase `sanity-check' failed after 0.1 seconds¬ +``` + +This was fixed by reverting the update to python-pingouin, dropping it back to 0.3.12. diff --git a/issues/gemma/gemma-wrapper-has-incomplete-files.gmi b/issues/gemma/gemma-wrapper-has-incomplete-files.gmi new file mode 100644 index 0000000..d530fb4 --- /dev/null +++ b/issues/gemma/gemma-wrapper-has-incomplete-files.gmi @@ -0,0 +1,42 @@ +# gemma-wrapper has incomplete files + +Gemma wrapper caches files - but it can happen a cached file is incomplete and never updated again. The problem appears when GNU parallel is invoked and hits an error. The task here is to make gemma-wrapper transactional. + +## Tags + +* assigned: pjotrp, zachs + +## Tasks + +* [X] parse parallel job log for failed tasks and remove the output files. +* [X] create a (global) lock file for gemma-wrapper + +## Info + +GNU parallel can fail, but does not tell how individual processes did. Need to check if it can return a thread (number). If not we have the option of checking the GEMMA status file and/or see if the output file is complete (by counting number of lines). + +The 'obvious' fix would be to create an error handler in GEMMA itself that would clean up output files on error exit. E.g. using + +=> https://www.cplusplus.com/reference/exception/set_terminate/ + +The problem is that it is NOT a catch all. If there is a hardware fault or a problem in a library, such as openblas, there is no guarantee that the terminate handler will be called. Another complication is that a terminate handler needs to be aware of the files being output - i.e., we need to carry the state down somehow. I think we can probably address these issues as much is handled in the GEMMA PARAM class, but it is not worth the effort (I'll take care of it in a GEMMA rewrite). + +It turns out that GNU parallel can keep track of jobs in a job log - and even rerun the ones missing using the `--joblog` and `--resume` switches. The last we don't need because we are using a cache. But we can use the log file to remove any incomplete output files! To me this is the obvious solution because 'parallel' is monitoring outside the GEMMA process and is a hardened piece of software. On failure it simply designates runs that way and we can clean up any (partly) produced files followed by a safe rerun. The lock routine below ascertains no processes are creating the same output at the same time. + +## Delete files on failure + +Implemented in + +=> https://github.com/genetics-statistics/gemma-wrapper/commit/624ed0d805f29ab682cffbe46bc104dffd0d713c + +## Dealing with locks + +There is another parallel issue (pun intended) where gemma-wrapper is invoked twice for the same job. This is quite possible when people get impatient waiting for a first job to finish. + +One solution is to write a lock file using the inputs as a hash. The lock file can contain a PID and we can check if that is still alive. I should do the same for sheepdog locks(!) + +=> https://github.com/genetics-statistics/gemma-wrapper/commit/e7e516ec5a6ffc5b398302fa204685a40e76e171 Added locking support + +I added the same code to replace sheepdog locks. Sheepdog is running every minute on our machines so it is a great test case. + +=> https://github.com/pjotrp/deploy/commit/4790b81ee897c8244280169edb3cac751eb0a9b3 sheepdog locking diff --git a/issues/genenetwork/genewiki.gmi b/issues/genenetwork/genewiki.gmi new file mode 100644 index 0000000..e7096dc --- /dev/null +++ b/issues/genenetwork/genewiki.gmi @@ -0,0 +1,24 @@ +# Genewiki conversion + +GeneNetwork1 contains a genewiki: + +=> http://gn1.genenetwork.org/webqtl/main.py?FormID=geneWiki&symbol=BRCA2 + +We want to migrate it to markdown documents that can be fetched from + +=> https://github.com/genenetwork/gn-docs/genes + +So the steps are to (1) migrate the existing genewiki data in the database to named markdown documents in that repository and (2) create a rendered page that is found through + +=> https://genenetwork.org/doc/genes/BRCA2 + +with an edit button, similar to + +=> http://genenetwork.org/facilities/ + +## Tags + +* assigned: pjotrp, zsloan +* enhancement + +## Tasks diff --git a/issues/genenetwork/global-search.gmi b/issues/genenetwork/global-search.gmi new file mode 100644 index 0000000..01d9c0f --- /dev/null +++ b/issues/genenetwork/global-search.gmi @@ -0,0 +1,12 @@ +# Global search problems + +Global search is the top bar of GN2 + +## Tags + +* assigned: pjotrp, zsloan + +## Tasks + +* [ ] BRCA2 does not render results in table +* [ ] 'Brca2' with quotes gives a SQL error diff --git a/issues/genenetwork/http-https.gmi b/issues/genenetwork/http-https.gmi new file mode 100644 index 0000000..80306bf --- /dev/null +++ b/issues/genenetwork/http-https.gmi @@ -0,0 +1,15 @@ +# Arthur mentioned last meet that GeneNetwork2 does not redirect http urls to https. + +He'd like http to redirect to https. + +http://genenetwork.org/ -> https://genenetwork.org/ + +# nginx snippet + +``` +if ($scheme = http) { + return 301 https://$server_name$request_uri; +} +``` + +* assigned: pjotr, arthur, jgart diff --git a/issues/genenetwork1/gn1-symbols.gmi b/issues/genenetwork1/gn1-symbols.gmi new file mode 100644 index 0000000..ac4cf71 --- /dev/null +++ b/issues/genenetwork1/gn1-symbols.gmi @@ -0,0 +1,23 @@ +# GN1 symbols + +# Tags + +* assigned: pjotrp, arthurc + +# Tasks + +* [ ] fix font issue +* [ ] fix linked webqtl images + +# Info + +The fonts differ on + +=> http://gn1-lily.genenetwork.org/webqtl/main.py?FormID=sharinginfo&GN_AccessionId=2 +=> http://gn1.genenetwork.org/webqtl/main.py?FormID=sharinginfo&GN_AccessionId=2 + +There is a deeper problem. When you click on one of the symbols it wants to show, for example: + +=> http://www.webqtl.org/array_images/S092UM-1F2.png + +This fails both for GN1 and Lily. Do we still want to show those images, or simply disable? If we want to show them, where are they? diff --git a/issues/gnbug/add-convenience-templates.gmi b/issues/gnbug/add-convenience-templates.gmi new file mode 100644 index 0000000..41eb975 --- /dev/null +++ b/issues/gnbug/add-convenience-templates.gmi @@ -0,0 +1,29 @@ +# Add convenienve templates + +* feature-request + +WDYT of adding a subcommand called `report` to generate a template for a bug report? + +I was thinking of an interface like the following: + +``` +gnbug report feature-request "New feature: IPFS Downloads" +``` + +Or expressed as generic command parameters: + +``` +command subcommand tag(s) title +``` + +Which creates a file in issues/binderlite directory with a template like the +following: + + +``` +# New feature: IPFS Downloads + +* feature-request +``` + +Might be bloat but might be nice and convenient also. WDYT? diff --git a/issues/gnbug/add-linter.gmi b/issues/gnbug/add-linter.gmi index 7992753..1a8a301 100644 --- a/issues/gnbug/add-linter.gmi +++ b/issues/gnbug/add-linter.gmi @@ -14,3 +14,11 @@ support. I am not sure. Pjotr is of the philosophy that we should keep the text as unstructured and unrestrictive as possible, and I agree with this philosophy. Linters are likely to annoy users more than help them. They might discourage casual use of the issue tracker. Especially with tags, we don't really want to restrict the user from coming up with new tags. WDYT? Am I missing some scenario where the linter could be essential? + +Nope, it was just something I thought could be convenient if we wanted to restrict +the format for tags, etc... + +If we'd like to "keep the text as unstructured and unrestrictive as possible" I'm +fine with that too. + +I was just worried about using "invalid" tags. Now, I know it's ok :) diff --git a/issues/per-service-profile-for-shepherd-services.gmi b/issues/per-service-profile-for-shepherd-services.gmi new file mode 100644 index 0000000..b22448b --- /dev/null +++ b/issues/per-service-profile-for-shepherd-services.gmi @@ -0,0 +1,33 @@ +# Each shepherd service needs its own guix profile. + +Currently there are a bunch of shepherd services (on Penguin2, as user shepherd) who are using the global shepherd guix profile and loading guix-bioinformatics at service start time to start their services. This is bad because: + +* we aren't using per-service guix-bioinformatics worktrees, so the branch changes between service refreshes +* service start times are high because we have to rebuild the service (due to above) +* services which are restarted are in an "untested" state until they work + +## Services which need to be migrated: + +* covid19-pubseq (uses shared (broken!) guix profile, uses shared config directory)(currently running by hand by Pjotr) +* hrdp-project (uses shared (broken!) guix profile) (upstream source is gone) +* ratspub (uses own guix profile, uses shared guix-bioinformatics)(deprecated(?)) +* pluto (uses own guix profile, uses shared config directory)(currently disabled) + +## Services which need to be monitored: + +* genenetwork1 (uses own guix profile, uses shared config directory) +* genome_browser (uses own guix profile, uses shared config directory) +* mcron (uses shared guix profile, probably OK) +* virtuoso (uses shared guix profile, probably OK) + +## Services which already have their own profile: + +* bnw +* cronjob-gitea +* cronjob-pubmed (genecup) +* genecup +* gitea +* ipfs +* power +* rn6app +* singlecell diff --git a/issues/quality-control/enumerate-all-qc-checks.gmi b/issues/quality-control/enumerate-all-qc-checks.gmi new file mode 100644 index 0000000..2f72f24 --- /dev/null +++ b/issues/quality-control/enumerate-all-qc-checks.gmi @@ -0,0 +1,7 @@ +# Enumerating all QC checks + +We need to enumerate all QC checks and platforms that we want to test against. + +* assigned: jgart, arthur + +* in progress diff --git a/issues/quality-control/gene-symbols.gmi b/issues/quality-control/gene-symbols.gmi new file mode 100644 index 0000000..63265ea --- /dev/null +++ b/issues/quality-control/gene-symbols.gmi @@ -0,0 +1,13 @@ +# Gene Symbol Mappings + +How can/should I lookup *all* the known mappings between different representations of symbol names? + +For example, B6 maps to C57BL/6J and D2 maps to DBA/2J. + +I'll need that mapping information for every strain, symbol etc... that has an alias in order to make the conversions. + +Ideally, I can have those mappings in some machine-readable format like csv, json, etc... + +* assigned: jgart, arthur + +* in progress diff --git a/issues/redis-data.gmi b/issues/redis-data.gmi new file mode 100644 index 0000000..2246c96 --- /dev/null +++ b/issues/redis-data.gmi @@ -0,0 +1,6 @@ +# Document redis data + +ATM how we store data in redis is undocumented. Without this, it's +hard for contributors to work out how to modify some things. For +example, PJ would want to add himself to the editors group. Doing +this was non-trivial for him.
\ No newline at end of file diff --git a/issues/remove-elastic-search.gmi b/issues/remove-elastic-search.gmi new file mode 100644 index 0000000..af475d6 --- /dev/null +++ b/issues/remove-elastic-search.gmi @@ -0,0 +1,15 @@ +# Remove Elastic Search + +Elastic Search is no longer used, yet we have dead code referencing it +in gn2. + + +## Tasks + +* assigned: bonfacekilz +* keywords: chore, done + +## Notes + +While at it, "@depracated" was removed since it added extra noise to +the logs.
\ No newline at end of file diff --git a/tasks/dana.gmi b/tasks/dana.gmi index af053b7..51f33b6 100644 --- a/tasks/dana.gmi +++ b/tasks/dana.gmi @@ -6,9 +6,16 @@ ## Tasks -* [X] Tux02 https firewalling * [ ] Octopus 10Gbs switch + cables -* [ ] Mount 4TB SSDs for Tux01 and Tux02 + - a test shows Octopus08 is at 10Gbs on the second interface + - need to hook up new cables +* [ ] Add gateway on 10Gbs switch => ../topics/systems/decommission-machines.gmi +Completed: + +* [X] Tux02 https firewalling +* [X] Mount 4TB SSDs for Tux01 and Tux02 + + ## Info diff --git a/tasks/pjotrp.gmi b/tasks/pjotrp.gmi index 5b5178e..f6554fd 100644 --- a/tasks/pjotrp.gmi +++ b/tasks/pjotrp.gmi @@ -7,7 +7,8 @@ ## Tasks * [ ] Rabbit https and GN2 https -* [X] Allow @bonfacem to run redis on P2 +* [ ] fix GN1 images linking to http://www.webqtl.org/array_images/S238-1F1-U74Av2.png +=> http://gn1.genenetwork.org/webqtl/main.py?FormID=sharinginfo&GN_AccessionId=2 => ../topics/systems/tux02-production.gmi Set up stable GN service * [ ] Pluto tool with Zach => ../issues/genenetwork1/gn1-production-system-issues.gmi diff --git a/tasks/zsloan.gmi b/tasks/zsloan.gmi new file mode 100644 index 0000000..7e60240 --- /dev/null +++ b/tasks/zsloan.gmi @@ -0,0 +1,12 @@ +# Tasks for Zach + +## Tags + +* assigned: zsloan + +## Tasks + +* [ ] Fix global search +=> ../issues/genenetwork/global-search.gmi fix global search +* [ ] GN1 migrate genewiki pages +=> ../issues/genenetwork/genewiki.gmi diff --git a/topics/arvados/covid19.gmi b/topics/arvados/covid19.gmi new file mode 100644 index 0000000..be96605 --- /dev/null +++ b/topics/arvados/covid19.gmi @@ -0,0 +1,7 @@ +# COVID-19 + +We are setting up a Cloud HPC for COVID-19 using Arvados as the underlying infrastructure. + +## Tasks + +=> https://github.com/yatisht/usher Fast phylogeny diff --git a/topics/cli/weechat.gmi b/topics/cli/weechat.gmi new file mode 100644 index 0000000..b6fc062 --- /dev/null +++ b/topics/cli/weechat.gmi @@ -0,0 +1,66 @@ +# Weechat + +Weechat is a very versatile terminal chat that can handle IRC, Matrix and Slack. + +## Weechat install + slack + +Arun has packaged weechat for slack and matrix in GNU Guix. Current install using a guix environment: + +``` +env GUIX_PACKAGE_PATH=~/guix-bioinformatics/ ~/opt/guix-latest/bin/guix environment --ad-hoc coreutils weechat weechat-matrix weechat-wee-slack nss-certs openssl python +echo $GUIX_ENVIRONMENT +/gnu/store/i4a3ynfjvq8r8cch14gkn2fwfs77wagj-profile/ +weechat +/python load /gnu/store/i4a3ynfjvq8r8cch14gkn2fwfs77wagj-profile/share/weechat/python/wee_slack.py +/slack register +``` + +So, capture the shell GUIX_ENVIRONMENT and use that to launch the wee-slack plugin. Next register a token through te Slack web interface. + +If you go through the motions you get a token and + +``` +/slack register 2654233192084.2682932906197.fd6d6e55c0038f-etc-052b4fa578ec497710b4b8770f1-etc +/python reload slack +``` + +More instructions for https://github.com/wee-slack/wee-slack. + +## Weechat IRC + +We have a GeneNetwork channel + +``` +/set irc.server_default.nicks pjotrp +/server add libera irc.libera.chat/6697 -ssl -autoconnect +/connect libera +/join #guix +/join #guix-risc-v +/join #guix-hpc +/join #genenetwork +``` + +## Weechat matrix + +The matrix plugin works similar to above Slack plugin. + +``` +/script load matrix.py +/matrix server add fosdem chat.fosdem.org +/set matrix.server.matrix_org.username pjotrp-linux +/set matrix.server.matrix_org.password **** +/set matrix.server.matrix_org.autoconnect on +/save +``` + +Only the matrix version is not so polished. For example, it is hard to find the buffers because they are named by a hash. In a buffer you can rename with + +``` +/buffer set short_name myname +``` + +Also, you need to enable every channel explicitly to exchange keys + +``` +/olm verify @user:matrix.org * +``` diff --git a/topics/data-uploads/editing-data.gmi b/topics/data-uploads/editing-data.gmi index 1bc1bf8..01144a0 100644 --- a/topics/data-uploads/editing-data.gmi +++ b/topics/data-uploads/editing-data.gmi @@ -1,3 +1,7 @@ +## Tags + +* assigned: bonfacem + ### Introduction At the moment, you can edit metadata related to a published phenotype and a probeset. When an edit is done, the diff data is stored in a table, `metadata_audit` in json format that looks something like: @@ -60,3 +64,9 @@ After editing traits, you could see changes in a diff format at the very top of That needs to be improved. - For Probeset data, atm some tables aren't yet updated(unfortunately those tables aren't used anywhere). I need to figure that out. + +#### Tue 23 Nov 2021 + +- Fixed excel issue described here: + +=> https://github.com/genenetwork/genenetwork2/compare/testing...BonfaceKilz:bug/fix-excel-adding-new-line?expand=1
\ No newline at end of file diff --git a/topics/documentation/gn-documentation.gmi b/topics/documentation/gn-documentation.gmi new file mode 100644 index 0000000..653a1a4 --- /dev/null +++ b/topics/documentation/gn-documentation.gmi @@ -0,0 +1,21 @@ +# GeneNetwork documentation + +GeneNetwork requires an online 'wiki' and a place for documentation. Both of these can be maintained in markdown and/or gemini gemtext. We are using a repo: + +=> https://github.com/genenetwork/gn-docs + +What we need to do is render documents from markdown (as is done with the facilities page) + +=> https://github.com/genenetwork/gn-docs/blob/master/general/help/facilities.md + +the markdown is rendered on github, which makes it easy to edit for most users, so it should be the default. + +## Tags + +* assigned: pjotrp + +## Tasks + +=> storyboards + +## Info diff --git a/topics/gemini/cheatsheet.gmi b/topics/gemini/cheatsheet.gmi new file mode 100644 index 0000000..b2158ea --- /dev/null +++ b/topics/gemini/cheatsheet.gmi @@ -0,0 +1,5 @@ +# gemtext cheatsheet + +=> gemini://gemini.circumlunar.space/docs/cheatsheet.gmi + +=> https://gemini.circumlunar.space/docs/cheatsheet.gmi proxy diff --git a/topics/gemini/ideas_and_references.gmi b/topics/gemini/ideas_and_references.gmi new file mode 100644 index 0000000..dfb026d --- /dev/null +++ b/topics/gemini/ideas_and_references.gmi @@ -0,0 +1,3 @@ +Ideas and References + +=> gemini://drewdevault.com/2021/11/15/Reframing-gemini.gmi Reframing the philosophy of Gemini diff --git a/topics/pudb/pytest-pudb.gmi b/topics/pudb/pytest-pudb.gmi new file mode 100644 index 0000000..5a979c0 --- /dev/null +++ b/topics/pudb/pytest-pudb.gmi @@ -0,0 +1,7 @@ +# Using pudb with pytest + +``` +pytest --pdbcls pudb.debugger:Debugger --pdb --capture=no +``` + +=> https://documen.tician.de/pudb/starting.html?highlight=pytest#usage-with-pytest diff --git a/topics/pudb/using-pudb.gmi b/topics/pudb/using-pudb.gmi new file mode 100644 index 0000000..877a700 --- /dev/null +++ b/topics/pudb/using-pudb.gmi @@ -0,0 +1,21 @@ +# Using pudb + +## Setting Breakpoints + +You can add the following line anywhere in your code to start a pudb breakpoint: + +``` +import pudb; pudb.set_trace() +``` + +You can also press `b` in pudb to add a breakpoint with a keyboard shortcut. + +## pdb and pudb API similarity + +The standard pdb docs should work for pudb with one small exception. + +> At the programming language level, PuDB displays the same interface +> as Python’s built-in pdb module. Just replace pdb with pudb. (One +> exception: run is called runstatement.) + +=> https://docs.python.org/3/library/pdb.html?highlight=pdb#module-pdb diff --git a/topics/quality-control/qc-checks.gmi b/topics/quality-control/qc-checks.gmi new file mode 100644 index 0000000..0d04625 --- /dev/null +++ b/topics/quality-control/qc-checks.gmi @@ -0,0 +1,47 @@ +# Quality Control Checks + +1. ProbeSetId (Affymetrix format): + +We favour using Illumina, Affimetrix, and other platform formats. + +Custom formats require a new annotation file to be created. + +We usually use Ensemble ID or Gene IDs. + +1.1 Ensemble transcript IDs usually have duplicates that need to be pruned. + +ENSMBL1234 + +## Example Gene Symbol to ProbeSetId + +AFFX-BkGr-GC03_st -> TCO500002136.mm.2 + +2. Inbred Strain names should prefer long form: + +B6 -> C57BL/6 +D2 -> DBA/2 + +3. Probeset IDs that don't have any values should be pruned: + +For example an Affymetrix data set might have ~28,000 entries and the data set that +is allowed into the GeneNetwork will be 22,000 entries. + +4. The standard error between male and female mice has to be computed. + +5. SE values have to be computed to 6 or greater decimal places. + +6. The average between male and female mice has to be computed to 3 decimal places. + +7. Datasets/studies having the same ProbeSetID should be grouped together. + +8. There should be no trailing spaces in data cells. + +9. Entries should have the same capitalization style. + +10. Assesing Phenotypes for normality with Shapiro-Wilk Test. + +11. Check for annotations file. + +12. Check for CRLF. + +13. Check for UTF-8 encoding. diff --git a/topics/quality-control/shapiro-wilk-test.gmi b/topics/quality-control/shapiro-wilk-test.gmi new file mode 100644 index 0000000..b1bdbfd --- /dev/null +++ b/topics/quality-control/shapiro-wilk-test.gmi @@ -0,0 +1,33 @@ +# Shapiro Wilk Test + +This document contains more info about QC step 11. + +https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html +https://vedexcel.com/how-to-perform-a-shapiro-wilk-test-in-python/ + +``` +Here is a simple QC procedure we may want to consider that was used by +Megan and Camron in a recent paper that I have attached. + + +Phenotypes were assessed for normality using the Shapiro–Wilk Test. Because +some of the data residuals devi- ated significantly from normality, we used +the orderNorm function to perform Ordered Quantile normalization43 on all +phenotypes. + +-- +Rob +``` + +``` +QTL analysis was performed in F2 mice using the R package R/qtl +(RRID:SCR_009085) as previously described.11,29,42 Quality checking +of genotypes and QTL analysis were performed in R (https://www.r- +project.org/) using R/bestNormalize (https://github.com/petersonR/ +bestNormalize) and R/qtl.42 Phenotypes were assessed for normality +using the Shapiro–Wilk Test. Because some of the data residuals devi- +ated significantly from normality, we used the orderNorm function to +perform Ordered Quantile normalization43 on all phenotypes. QTL +-- +A quantitative trait variant in Gabra2 underlies increased methamphetamine stimulant sensitivity +``` diff --git a/topics/quality-control/ui-design.gmi b/topics/quality-control/ui-design.gmi new file mode 100644 index 0000000..84f7748 --- /dev/null +++ b/topics/quality-control/ui-design.gmi @@ -0,0 +1,41 @@ +# UI Design + +1. Input/Receive Data in UI (drag and drop/upload submit form) + +2. Select Mouse + +"What type of Group are you using?" + +> (AKXD, BXH, Mouse Diversity Panel, BXD) + +3. "What is your platform?" + +> (Aff, Ilumina, ...) + +If Affymetrix (Aff) is selected then there should be various options +like Clarion S. + +If the platform you chose is not available: + + Tell PI that they should solicit for their platform to be added to the list. + + They can contact us via email. + +4. Allow excel file upload? + +## More Example UI Interactions and Checks + +"If your dataset does not comply with GN then you can try uploading your +dataset so that we can inspect it." + +"Your dataset has two erroneous entries: Gene Accession Gene." + +"The last two columns have the wrong format for the strain name." + +"Here's our format of how your dataset should look like." + +> ProbeSetID Strains ... + +"Inbred Set ID 1 is the same as BXD" + +> These are the strains: ... diff --git a/topics/systems/machine-room.gmi b/topics/systems/machine-room.gmi index bece333..d75883a 100644 --- a/topics/systems/machine-room.gmi +++ b/topics/systems/machine-room.gmi @@ -6,9 +6,10 @@ ## Tasks -* [ ] Make tux02e visible from outside -* [ ] Network switch 10Gbs -* [ ] Add disks to tux01 - Sep 28 To Brock +* [X] Make tux02e visible from outside +* [ ] Network switch 10Gbs - add hosts +* [ ] Add disks to tux01 and tux02 - need to reboot +* [ ] Set up E-mail relay for tux01 and tux02 smtp.uthsc.edu, port 25 => tux02-production.gmi setup new production machine => decommission-machines.gmi Decommission machines diff --git a/topics/systems/sheepdog.gmi b/topics/systems/sheepdog.gmi index 478adf1..137a5dd 100644 --- a/topics/systems/sheepdog.gmi +++ b/topics/systems/sheepdog.gmi @@ -13,6 +13,7 @@ that need resolving at some point. ## Tasks * [X] add locking functionality for tags - added borg with-lock (test) +* [X] chgrp functionality in sheepdog_borg * [ ] check whether rsync dir exists, repo valid and/or no lock before proceeding * [ ] send digest E-mails * [ ] smart state E-mails on services going down diff --git a/topics/testing/genenetwork-checklist.gmi b/topics/testing/genenetwork-checklist.gmi new file mode 100644 index 0000000..ba19733 --- /dev/null +++ b/topics/testing/genenetwork-checklist.gmi @@ -0,0 +1,59 @@ +# GN Checklist + +This is the checklist to run through when testing GN2. + +## Tags + +* assigned: pjotrp + +## Tasks + +This is a check list for testing whether GN2 is completely deployed + +* [ ] visit main page +=> https://luna.genenetwork.org/ +* [ ] simple search (e.g. shh in hippocampus mRNA) +=> https://luna.genenetwork.org/search?species=mouse&group=BXD&type=Hippocampus+mRNA&dataset=HC_M2_0606_P&search_terms_or=shh&search_terms_and=&FormID=searchResult +* [ ] try your own search term +* [ ] wild card seach (e.g. sh* in hippocampus mRNA) +=> https://luna.genenetwork.org/search?species=mouse&group=BXD&type=Hippocampus+mRNA&dataset=HC_M2_0606_P&search_terms_or=sh*&search_terms_and=&FormID=searchResult +* [ ] try your own wild-card search +* [ ] visit a trait page +=> https://luna.genenetwork.org/show_trait?trait_id=1427571_at&dataset=HC_M2_0606_P + + +will continue... + + +curl -X POST -b 'anon_user_v1=4d2ce6d3-0d4c-4349-b491-a4e179fdc163:d5fdc040c737e2a775d2' https://luna.genenetwork.org/run_mapping + +https://luna.genenetwork.org/run_mapping +User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0 +Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 +Accept-Language: en-US,en;q=0.5 +Accept-Encoding: gzip, deflate, br +Content-Type: application/x-www-form-urlencoded +Content-Length: 12345 +Origin: https://luna.genenetwork.org +Connection: keep-alive +Referer: https://luna.genenetwork.org/loading +Cookie: anon_user_v1=4d2ce6d3-0d4c-4349-b491-a4e179fdc163:d5fdc040c737e2a775d2; 404 Not Found: The requested URL=Wild-Type-Mouse.gif +Upgrade-Insecure-Requests: 1 +Sec-GPC: 1 + +https://luna.genenetwork.org/run_mapping +POST /run_mapping HTTP/1.1 +Host: luna.genenetwork.org +User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0 +Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 +Accept-Language: en-US,en;q=0.5 +Accept-Encoding: gzip, deflate, br +Referer: https://luna.genenetwork.org/loading +Content-Type: application/x-www-form-urlencoded +Content-Length: 12345 +Origin: https://luna.genenetwork.org +Connection: keep-alive +Cookie: anon_user_v1=4d2ce6d3-0d4c-4349-b491-a4e179fdc163:d5fdc040c737e2a775d2; 404 Not Found: The requested URL=Wild-Type-Mouse.gif +Upgrade-Insecure-Requests: 1 +Sec-GPC: 1 +Cache-Control: max-age=0 |
