diff options
34 files changed, 1102 insertions, 35 deletions
diff --git a/issues/cleanup-base-file-gn2.gmi b/issues/cleanup-base-file-gn2.gmi index 1c086ea..8a05323 100644 --- a/issues/cleanup-base-file-gn2.gmi +++ b/issues/cleanup-base-file-gn2.gmi @@ -5,7 +5,7 @@ * Assigned: alexm * Keywords: base, HTML, JavaScript, cleanup * type: Refactoring -* Status: In progress +* Status: closed, completed, done ## Description @@ -13,9 +13,9 @@ The base file should contain no custom JavaScript since it is inherited in almos ## Tasks -* [] Remove global search code from the base file and move it to the index page -* [] Fix formatting and linting issues in the base file (e.g., tags) -* [] Inherit from index page for all gn2 templates +* [x] Remove global search code from the base file and move it to the index page +* [x] Fix formatting and linting issues in the base file (e.g., tags) +* [x] Inherit from index page for all gn2 templates ## Notes diff --git a/issues/create-custom-rif-xapian-index.gmi b/issues/create-custom-rif-xapian-index.gmi new file mode 100644 index 0000000..a0b9039 --- /dev/null +++ b/issues/create-custom-rif-xapian-index.gmi @@ -0,0 +1,16 @@ +# Create Custom RIF XAPIAN Index + +## Tags + +* assigned: bonfacem +* priority: medium +* status: in-progress +* deadline: 2024-10-23 Wed + +## Description + +Given the GN Wiki search page: + +=> https://cd.genenetwork.org/genewiki GeneWiki Entries Search + +We only search by symbol. Add custom XAPIAN index to perform more powerful search. diff --git a/issues/genenetwork/cannot-connect-to-mariadb.gmi b/issues/genenetwork/cannot-connect-to-mariadb.gmi new file mode 100644 index 0000000..ca4bd9f --- /dev/null +++ b/issues/genenetwork/cannot-connect-to-mariadb.gmi @@ -0,0 +1,99 @@ +# Cannot Connect to MariaDB + + +## Description + +GeneNetwork3 is failing to connect to mariadb with the error: + +``` +⋮ +2024-11-05 14:49:00 Traceback (most recent call last): +2024-11-05 14:49:00 File "/gnu/store/83v79izrqn36nbn0l1msbcxa126v21nz-profile/lib/python3.10/site-packages/flask/app.py", line 1523, in full_dispatch_request +2024-11-05 14:49:00 rv = self.dispatch_request() +2024-11-05 14:49:00 File "/gnu/store/83v79izrqn36nbn0l1msbcxa126v21nz-profile/lib/python3.10/site-packages/flask/app.py", line 1509, in dispatch_request +2024-11-05 14:49:00 return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) +2024-11-05 14:49:00 File "/gnu/store/83v79izrqn36nbn0l1msbcxa126v21nz-profile/lib/python3.10/site-packages/gn3/api/menu.py", line 13, in generate_json +2024-11-05 14:49:00 with database_connection(current_app.config["SQL_URI"], logger=current_app.logger) as conn: +2024-11-05 14:49:00 File "/gnu/store/lzw93sik90d780n09svjx5la1bb8g3df-python-3.10.7/lib/python3.10/contextlib.py", line 135, in __enter__ +2024-11-05 14:49:00 return next(self.gen) +2024-11-05 14:49:00 File "/gnu/store/83v79izrqn36nbn0l1msbcxa126v21nz-profile/lib/python3.10/site-packages/gn3/db_utils.py", line 34, in database_connection +2024-11-05 14:49:00 connection = mdb.connect(db=db_name, +2024-11-05 14:49:00 File "/gnu/store/83v79izrqn36nbn0l1msbcxa126v21nz-profile/lib/python3.10/site-packages/MySQLdb/__init__.py", line 121, in Connect +2024-11-05 14:49:00 return Connection(*args, **kwargs) +2024-11-05 14:49:00 File "/gnu/store/83v79izrqn36nbn0l1msbcxa126v21nz-profile/lib/python3.10/site-packages/MySQLdb/connections.py", line 195, in __init__ +2024-11-05 14:49:00 super().__init__(*args, **kwargs2) +2024-11-05 14:49:00 MySQLdb.OperationalError: (2002, "Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)") +``` + +We have previously defined the default socket file[^1][^2] as "/run/mysqld/mysqld.sock". + +## Troubleshooting Logs + +### 2024-11-05 + +I attempted to just bind `/run/mysqld/mysqld.sock` to `/tmp/mysql.sock` by adding the following mapping in GN3's `gunicorn-app` definition: + +``` +(file-system-mapping + (source "/run/mysqld/mysqld.sock") + (target "/tmp/mysql.sock") + (writable? #t)) +``` + +but that does not fix things. + +I had tried to change the mysql URI to use IP addresses, i.e. + +``` +SQL_URI="mysql://webqtlout:webqtlout@128.169.5.119:3306/db_webqtl" +``` + +but that simply changes the error from the above to the one below: + +``` +2024-11-05 15:27:12 MySQLdb.OperationalError: (2002, "Can't connect to MySQL server on '128.169.5.119' (115)") +``` + +I tried with both `127.0.0.1` and `128.169.5.119`. + +My hail-mary was to attempt to expose the `my.cnf` file generated by the `mysql-service-type` definition to the "pola-wrapper", but that is proving tricky, seeing as the file is generated elsewhere[^4] and we do not have a way of figuring out the actual final path of the file. + +I tried: + +``` +(file-system-mapping + (source (mixed-text-file "my.cnf" + (string-append "[client]\n" + "socket=/run/mysqld/mysqld.sock"))) + (target "/etc/mysql/my.cnf")) +``` + +but that did not work either. + +### 2024-11-07 + +Start digging into how GNU Guix services are defined[^5] to try and understand why the file mapping attempt did not work. + +=> http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/system/file-systems.scm?id=2394a7f5fbf60dd6adc0a870366adb57166b6d8b#n575 +Looking at the code linked above specifically at lines 575 to 588, and 166, it seems, to me, that the mappings attempt should have worked. + +Try it again, taking care to verify that the paths are correct, with: + +``` +(file-system-mapping + (source (mixed-text-file "my.cnf" + (string-append "[client-server]\n" + "socket=/run/mysqld/mysqld.sock"))) + (target "/etc/my.cnf")) +``` + +Try rebuilding on tux04: started getting `Segmentation fault` errors out of the blue for many guix commands 🤦🏿. +Try building container on local dev machine: this took a long time - quit and continue later. + +### Footnotes + +[^1] Lines 47 to 49 of https://git.genenetwork.org/gn-machines/tree/production.scm?id=46a1c4c8d01198799e6ac3b99998dca40d2c7094#n47 +[^2] Guix's mysql-service-type configurations https://guix.gnu.org/manual/en/html_node/Database-Services.html#index-mysql_002dconfiguration +[^3] https://mariadb.com/kb/en/server-system-variables/#socket +[^4] https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/databases.scm?id=4c56d0cccdc44e12484b26332715f54768738c5f#n576 +[^5] https://guix.gnu.org/manual/en/html_node/Defining-Services.html diff --git a/issues/genenetwork/containerising-production-issues.gmi b/issues/genenetwork/containerising-production-issues.gmi new file mode 100644 index 0000000..ed5702a --- /dev/null +++ b/issues/genenetwork/containerising-production-issues.gmi @@ -0,0 +1,33 @@ +# Containerising Production: Issues + +## Tags + +* type: bug +* assigned: fredm +* priority: critical +* status: closed, completed +* keywords: production, container, tux04 +* interested: alexk, aruni, bonfacem, fredm, pjotrp, soloshelby, zsloan, jnduli + +## Description + +We have recently got production into a container and deployed it: It has come up, however, that there are services that are useful to get a full-featured GeneNetwork system running that are not part of the container. + +This is, therefore, a meta-issue, tracking all issues that relate to the deployment of the disparate services that make up GeneNetwork. + +## Documentation + +=> https://issues.genenetwork.org/topics/genenetwork/genenetwork-services + +The link above documents the various services that make up the GeneNetwork service. + +## Issues + +* [x] Move user directories to a large partition +=> ./handle-tmp-dirs-in-container [x] Link TMPDIR in container to a directory on a large partition +=> ./markdown-editing-service-not-deployed [ ] Define and deploy Markdown Editing service +=> ./umhet3-samples-timing-slow [ ] Figure out and fix UM-HET3 Samples mappings on Tux04 +=> ./setup-mailing-on-tux04 [x] Setting up email service on Tux04 +=> ./virtuoso-shutdown-clears-data [x] Virtuoso seems to lose data on restart +=> ./python-requests-error-in-container [x] Fix python's requests library certificates error +=> ./cannot-connect-to-mariadb [ ] GN3 cannot connect to mariadb server diff --git a/issues/genenetwork/handle-tmp-dirs-in-container.gmi b/issues/genenetwork/handle-tmp-dirs-in-container.gmi new file mode 100644 index 0000000..5f6eb92 --- /dev/null +++ b/issues/genenetwork/handle-tmp-dirs-in-container.gmi @@ -0,0 +1,22 @@ +# Handle Temporary Directories in the Container + +## Tags + +* type: feature +* assigned: fredm +* priority: critical +* status: closed, completed +* keywords: production, container, tux04 +* interested: alexk, aruni, bonfacem, pjotrp, zsloan + +## Description + +The container's temporary directories should be in a large partition on the host to avoid a scenario where the writes fill up one of the smaller drives. + +Currently, we use the `/tmp` directory by default, but we should look into transitioning away from that — `/tmp` is world readable and world writable and therefore needs careful consideration to keep safe. + +Thankfully, we are running our systems within a container, and can bind the container's `/tmp` directory to a non-world-accessible directory, keeping things at least contained. + +### Fixes + +=> https://git.genenetwork.org/gn-machines/commit/?id=7306f1127df9d4193adfbfa51295615f13d32b55 diff --git a/issues/genenetwork/markdown-editing-service-not-deployed.gmi b/issues/genenetwork/markdown-editing-service-not-deployed.gmi new file mode 100644 index 0000000..e7a1717 --- /dev/null +++ b/issues/genenetwork/markdown-editing-service-not-deployed.gmi @@ -0,0 +1,34 @@ +# Markdown Editing Service: Not Deployed + +## Tags + +* type: bug +* status: open +* assigned: fredm +* priority: critical +* keywords: production, container, tux04 +* interested: alexk, aruni, bonfacem, fredm, pjotrp, zsloan + +## Description + +The Markdown Editing service is not working on production. + +* Link: https://genenetwork.org/facilities/ +* Repository: https://git.genenetwork.org/gn-guile + +Currently, the code is being run directly on the host, rather than inside the container. + +Some important things to note: + +* The service requires access to a checkout of https://github.com/genenetwork/gn-docs +* Currently, the service is hard-coded to use a specific port: we should probably fix that. + +## Reopened: 2024-11-01 + +While the service was deployed, the edit functionality is not working right, specifically, pushing the edits upstream to the remote seems to fail. + +If you do an edit and refresh the page, it will show up in the system, but it will not proceed to be pushed up to the remote. + +Set `CGIT_REPO_PATH="https://git.genenetwork.org/gn-guile"` which seems to allow the commit to work, but we do not actually get the changes pushed to the remote in any useful sense. + +It seems to me, that we need to configure the environment in such a way that it will be able to push the changes to remote. diff --git a/issues/genenetwork/python-requests-error-in-container.gmi b/issues/genenetwork/python-requests-error-in-container.gmi new file mode 100644 index 0000000..0289762 --- /dev/null +++ b/issues/genenetwork/python-requests-error-in-container.gmi @@ -0,0 +1,174 @@ +# Python Requests Error in Container + +## Tags + +* type: bug +* assigned: fredm +* priority: critical +* status: closed, completed, fixed +* interested: alexk, aruni, bonfacem, pjotrp, zsloan +* keywords: production, container, tux04, python, requests + +## Description + +Building the container with the +=> https://git.genenetwork.org/guix-bioinformatics/commit/?id=eb7beb340a9731775e8ad177e47b70dba2f2a84f upgraded guix definition +leads to python's requests library failing. + +``` +2024-10-30 16:04:13 OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt +``` + +If you login to the container itself, however, you find that the file `/etc/ssl/certs/ca-certificates.crt` actually exists and has content. + +Possible fixes suggested are to set up correct envvars for the requests library, such as `REQUESTS_CA_BUNDLE` + +See +=> https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification + +### Troubleshooting Logs + +Try reproducing the issue locally: + +``` +$ guix --version +hint: Consider installing the `glibc-locales' package and defining `GUIX_LOCPATH', along these lines: + + guix install glibc-locales + export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale" + +See the "Application Setup" section in the manual, for more info. + +guix (GNU Guix) 2394a7f5fbf60dd6adc0a870366adb57166b6d8b +Copyright (C) 2024 the Guix authors +License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> +This is free software: you are free to change and redistribute it. +There is NO WARRANTY, to the extent permitted by law. +$ +$ guix shell --container --network python python-requests coreutils +[env]$ ls "${GUIX_ENVIRONMENT}/etc" +ld.so.cache profile +``` + +We see from the above that there are no certificates in the environment with just python and python-requests. + +Okay. Now let's write a simple python script to test things out with: + +``` +import requests + +resp = requests.get("https://github.com") +print(resp) +``` + +and run it! + +``` +$ guix shell --container --network python python-requests coreutils -- python3 test.py +Traceback (most recent call last): + File "/tmp/test.py", line 1, in <module> + import requests + File "/gnu/store/b6ny4p29f32rrnnvgx7zz1nhsms2zmqk-profile/lib/python3.10/site-packages/requests/__init__.py", line 164, in <module> + from .api import delete, get, head, options, patch, post, put, request + File "/gnu/store/b6ny4p29f32rrnnvgx7zz1nhsms2zmqk-profile/lib/python3.10/site-packages/requests/api.py", line 11, in <module> + from . import sessions + File "/gnu/store/b6ny4p29f32rrnnvgx7zz1nhsms2zmqk-profile/lib/python3.10/site-packages/requests/sessions.py", line 15, in <module> + from .adapters import HTTPAdapter + File "/gnu/store/b6ny4p29f32rrnnvgx7zz1nhsms2zmqk-profile/lib/python3.10/site-packages/requests/adapters.py", line 81, in <module> + _preloaded_ssl_context.load_verify_locations( +FileNotFoundError: [Errno 2] No such file or directory +``` + +Uhmm, what is this new error? + +Add `nss-certs` and try again. + +``` +$ guix shell --container --network python python-requests nss-certs coreutils +[env]$ ls ${GUIX_ENVIRONMENT}/etc/ssl/ +certs +[env]$ python3 test.py +Traceback (most recent call last): + File "/tmp/test.py", line 1, in <module> + import requests + File "/gnu/store/17dw8qczqqz9fmj2kxzsbfqn730frqd7-profile/lib/python3.10/site-packages/requests/__init__.py", line 164, in <module> + from .api import delete, get, head, options, patch, post, put, request + File "/gnu/store/17dw8qczqqz9fmj2kxzsbfqn730frqd7-profile/lib/python3.10/site-packages/requests/api.py", line 11, in <module> + from . import sessions + File "/gnu/store/17dw8qczqqz9fmj2kxzsbfqn730frqd7-profile/lib/python3.10/site-packages/requests/sessions.py", line 15, in <module> + from .adapters import HTTPAdapter + File "/gnu/store/17dw8qczqqz9fmj2kxzsbfqn730frqd7-profile/lib/python3.10/site-packages/requests/adapters.py", line 81, in <module> + _preloaded_ssl_context.load_verify_locations( +FileNotFoundError: [Errno 2] No such file or directory +[env]$ +[env]$ export REQUESTS_CA_BUNDLE="${GUIX_ENVIRONMENT}/etc/ssl/certs/ca-certificates.crt" +[env]$ $ python3 test.py +Traceback (most recent call last): + File "/tmp/test.py", line 1, in <module> + import requests + File "/gnu/store/17dw8qczqqz9fmj2kxzsbfqn730frqd7-profile/lib/python3.10/site-packages/requests/__init__.py", line 164, in <module> + from .api import delete, get, head, options, patch, post, put, request + File "/gnu/store/17dw8qczqqz9fmj2kxzsbfqn730frqd7-profile/lib/python3.10/site-packages/requests/api.py", line 11, in <module> + from . import sessions + File "/gnu/store/17dw8qczqqz9fmj2kxzsbfqn730frqd7-profile/lib/python3.10/site-packages/requests/sessions.py", line 15, in <module> + from .adapters import HTTPAdapter + File "/gnu/store/17dw8qczqqz9fmj2kxzsbfqn730frqd7-profile/lib/python3.10/site-packages/requests/adapters.py", line 81, in <module> + _preloaded_ssl_context.load_verify_locations( +FileNotFoundError: [Errno 2] No such file or directory +``` + +Welp! Looks like this error is a whole different thing. + +Let us try with the genenetwork2 package. + +``` +$ guix shell --container --network genenetwork2 coreutils +[env]$ ls "${GUIX_ENVIRONMENT}/etc" +bash_completion.d jupyter ld.so.cache profile +``` + +This does not seem to have the certificates in place either, so let's add nss-certs + +``` +$ guix shell --container --network genenetwork2 coreutils nss-certs +[env]$ ls "${GUIX_ENVIRONMENT}/etc" +bash_completion.d jupyter ld.so.cache profile ssl +[env]$ python3 test.py +Traceback (most recent call last): + File "/tmp/test.py", line 3, in <module> + resp = requests.get("https://github.com") + File "/gnu/store/qigjz4i0dckbsjbd2has0md2dxwsa7ry-profile/lib/python3.10/site-packages/requests/api.py", line 73, in get + return request("get", url, params=params, **kwargs) + File "/gnu/store/qigjz4i0dckbsjbd2has0md2dxwsa7ry-profile/lib/python3.10/site-packages/requests/api.py", line 59, in request + return session.request(method=method, url=url, **kwargs) + File "/gnu/store/qigjz4i0dckbsjbd2has0md2dxwsa7ry-profile/lib/python3.10/site-packages/requests/sessions.py", line 587, in request + resp = self.send(prep, **send_kwargs) + File "/gnu/store/qigjz4i0dckbsjbd2has0md2dxwsa7ry-profile/lib/python3.10/site-packages/requests/sessions.py", line 701, in send + r = adapter.send(request, **kwargs) + File "/gnu/store/qigjz4i0dckbsjbd2has0md2dxwsa7ry-profile/lib/python3.10/site-packages/requests/adapters.py", line 460, in send + self.cert_verify(conn, request.url, verify, cert) + File "/gnu/store/qigjz4i0dckbsjbd2has0md2dxwsa7ry-profile/lib/python3.10/site-packages/requests/adapters.py", line 263, in cert_verify + raise OSError( +OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt +``` + +We get the expected certificates error! This is good. Now define the envvar and try again. + +``` +[env]$ export REQUESTS_CA_BUNDLE="${GUIX_ENVIRONMENT}/etc/ssl/certs/ca-certificates.crt" +[env]$ python3 test.py +<Response [200]> +``` + +Success!!! + +Adding nss-certs and setting the `REQUESTS_CA_BUNDLE` fixes things. We'll need to do the same for the container, for both the genenetwork2 and genenetwork3 packages (and any other packages that use requests library). + +### Fixes + +=> https://git.genenetwork.org/guix-bioinformatics/commit/?id=fec68c4ca87eeca4eb9e69e71fc27e0eae4dd728 +=> https://git.genenetwork.org/guix-bioinformatics/commit/?id=c3bb784c8c70857904ef97ecd7d36ec98772413d +The two commits above add nss-certs package to all the flask apps, which make use of the python-requests library, which requires a valid CA certificates bundle in each application's environment. + +=> https://git.genenetwork.org/gn-machines/commit/?h=production-container&id=04506c4496e5ca8b3bc38e28ed70945a145fb036 +The commit above defines the "REQUESTS_CA_BUNDLE" environment variable for all the flask applications that make use of python's requests library. diff --git a/issues/genenetwork/setup-mailing-on-tux04.gmi b/issues/genenetwork/setup-mailing-on-tux04.gmi new file mode 100644 index 0000000..45605d9 --- /dev/null +++ b/issues/genenetwork/setup-mailing-on-tux04.gmi @@ -0,0 +1,16 @@ +# Setup Mailing on Tux04 + +## Tags + +* type: bug +* status: closed +* assigned: fredm +* priority: critical +* interested: pjotrp, zsloan +* keywords: production, container, tux04 + +## Description + +We use emails to verify user accounts and allow changing of user passwords. We therefore need to setup a way to send emails from the system. + +I updated the configurations to use UTHSC's mail server diff --git a/issues/genenetwork/umhet3-samples-timing-slow.gmi b/issues/genenetwork/umhet3-samples-timing-slow.gmi new file mode 100644 index 0000000..a3a33a7 --- /dev/null +++ b/issues/genenetwork/umhet3-samples-timing-slow.gmi @@ -0,0 +1,72 @@ +# UM-HET3 Timing: Slow + +## Tags + +* type: bug +* status: open +* assigned: fredm +* priority: critical +* interested: fredm, pjotrp, zsloan +* keywords: production, container, tux04, UM-HET3 + +## Description + +In email from @robw: + +``` +> > Not sure why. Am I testing the wrong way? +> > Are we using memory and RAM in the same way on the two machines? +> > Here are data on the loading time improvement for Tux2: +> > I tested this using a "worst case" trait that we know when—the 25,000 +> > UM-HET3 samples: +> > [1]https://genenetwork.org/show_trait?trait_id=10004&dataset=HET3-ITPPu +> > blish +> > Tux02: 15.6, 15.6, 15.3 sec +> > Fallback: 37.8, 38.7, 38.5 sec +> > Here are data on Gemma speed/latency performance: +> > Also tested "worst case" performance using three large BXD data sets +> > tested in this order: +> > [2]https://genenetwork.org/show_trait?trait_id=10004&dataset=BXD-Longev +> > ityPublish +> > [3]https://genenetwork.org/show_trait?trait_id=10003&dataset=BXD-Longev +> > ityPublish +> > [4]https://genenetwork.org/show_trait?trait_id=10002&dataset=BXD-Longev +> > ityPublish +> > Tux02: 107.2, 329.9 (ouch), 360.0 sec (double ouch) for 1004, 1003, and +> > 1002 respectively. On recompute (from cache) 19.9, 19.9 and 20.0—still +> > too slow. +> > Fallback: 154.1, 115.9 for the first two traits (trait 10002 already in +> > the cache) +> > On recompute (from cache) 59.6, 59.0 and 59.7. Too slow from cache. +> > PROBLEM 2: Tux02 is unable to map UM-HET3. I still get an nginx 413 +> > error: Entity Too Large. +> +> Yeah, Fred should fix that one. It is an nginx setting - we run 2x +> nginx. It was reported earlier. +> +> > I need this to work asap. Now mapping our amazing UM-HET3 data. I can +> > use Fallback, but it is painfully slow and takes about 214 sec. I hope +> > Tux02 gets that down to a still intolerable slow 86 sec. +> > Can we please fix and confirm by testing. The Trait is above for your +> > testing pleasure. +> > Even 86 secs is really too slow and should motivate us (or users like +> > me) to think about how we are using all of those 24 ultra-fast cores on +> > the AMD 9274F. Why not put them all to use for us and users. It is not +> > good enough just to have "it work". It has to work in about 5–10 +> > seconds. +> > Here are my questions for you guys: Are we able to use all 24 cores +> > for any one user? How does each user interact with the CPU? Can we +> > handle a class of 24 students with 24 cores, or is it "complicated"? +> > PROBLEM 3: Zach, Fred. Are we computing render time or transport +> > latency correctly? Ideally the printout at the bottom of mapping pages +> > would be true latency as experienced by the user. As far as I can tell +> > with a stop watch our estimates of time are incorrect by as much as 3 +> > secs. And note that the link +> > to [5]http://joss.theoj.org/papers/10.21105/joss.00025 is not working +> > correctly in the footer (see image below). Oddly enough it works fine +> > on Tux02 +> +> Fred, take a note. +``` + +Figure out what this is about and fix it. diff --git a/issues/genenetwork/virtuoso-shutdown-clears-data.gmi b/issues/genenetwork/virtuoso-shutdown-clears-data.gmi new file mode 100644 index 0000000..2e01238 --- /dev/null +++ b/issues/genenetwork/virtuoso-shutdown-clears-data.gmi @@ -0,0 +1,98 @@ +# Virtuoso: Shutdown Clears Data + +## Tags + +* type: bug +* assigned: fredm +* priority: critical +* status: closed, completed +* interested: bonfacem, pjotrp, zsloan +* keywords: production, container, tux04, virtuoso + +## Description + +It seems that virtuoso has the bad habit of clearing data whenever it is stopped/restarted. + +This issue will track the work necessary to get the service behaving correctly. + +According to the documentation on +=> https://vos.openlinksw.com/owiki/wiki/VOS/VirtBulkRDFLoader the bulk loading process + +``` +The bulk loader also disables checkpointing and the scheduler, which also need to be re-enabled post bulk load +``` + +That needs to be handled. + +### Notes + +After having a look at +=> https://docs.openlinksw.com/virtuoso/ch-server/#databaseadmsrv the configuration documentation +it occurs to me that the reason virtuoso supposedly clears the data is that the `DatabaseFile` value is not set, so it defaults to a new database file every time the server is restarted (See also the `Striping` setting). + +### Troubleshooting + +Reproduce locally: + +We begin by getting a look at the settings for the remote virtuoso +``` +$ ssh tux04 +fredm@tux04:~$ cat /gnu/store/bg6i4x96nm32gjp4qhphqmxqc5vggk3h-virtuoso.ini +[Parameters] +ServerPort = localhost:8981 +DirsAllowed = /var/lib/data +NumberOfBuffers = 4000000 +MaxDirtyBuffers = 3000000 +[HTTPServer] +ServerPort = localhost:8982 +``` + +Copy these into a file locally, and adjust the `NumberOfBuffers` and `MaxDirtyBuffers` for smaller local dev environment. Also update `DirsAllowed`. + +We end up with our local configuration in `~/tmp/virtuoso/etc/virtuoso.ini` with the content: + +``` +[Parameters] +ServerPort = localhost:8981 +DirsAllowed = /var/lib/data +NumberOfBuffers = 10000 +MaxDirtyBuffers = 6000 +[HTTPServer] +ServerPort = localhost:8982 +``` + +Run virtuoso! +``` +$ cd ~/tmp/virtuoso/var/lib/virtuoso/ +$ ls +$ ~/opt/virtuoso/bin/virtuoso-t +foreground +configfile ~/tmp/virtuoso/etc/virtuoso.ini +``` + +Here we start by changing into the `~/tmp/virtuoso/var/lib/virtuoso/` directory which will be where virtuoso will put its state. Now in a different terminal list the files created int the state directory: + +``` +$ ls ~/tmp/virtuoso/var/lib/virtuoso +virtuoso.db virtuoso.lck virtuoso.log virtuoso.pxa virtuoso.tdb virtuoso.trx +``` + +That creates the database file (and other files) with the documented default values, i.e. `virtuoso.*`. + +We cannot quite reproduce the issue locally, since every reboot will have exactly the same value for the files locally. + +Checking the state directory for virtuoso on tux04, however: + +``` +fredm@tux04:~$ sudo ls -al /export2/guix-containers/genenetwork/var/lib/virtuoso/ | grep '\.db$' +-rw-r--r-- 1 986 980 3787456512 Oct 28 14:16 js1b7qjpimdhfj870kg5b2dml640hryx-virtuoso.db +-rw-r--r-- 1 986 980 4152360960 Oct 28 17:11 rf8v0c6m6kn5yhf00zlrklhp5lmgpr4x-virtuoso.db +``` + +We see that there are multiple db files, each created when virtuoso was restarted. There is an extra (possibly) random string prepended to the `virtuoso.db` part. This happens for our service if we do not actually provide the `DatabaseFile` configuration. + + +## Fixes + +=> https://github.com/genenetwork/gn-gemtext-threads/commit/8211c1e49498ba2f3b578ed5b11b15c52299aa08 Document how to restart checkpointing and the scheduler after bulk loading +=> https://git.genenetwork.org/guix-bioinformatics/commit/?id=2dc335ca84ea7f26c6977e6b432f3420b113f0aa Add configs for scheduler and checkpointing +=> https://git.genenetwork.org/guix-bioinformatics/commit/?id=7d793603189f9d41c8ee87f8bb4c876440a1fce2 Set up virtuoso database configurations +=> https://git.genenetwork.org/gn-machines/commit/?id=46a1c4c8d01198799e6ac3b99998dca40d2c7094 Explicitly name virtuoso database files. diff --git a/issues/genenetwork2/handle-oauth-errors-better.gmi b/issues/genenetwork2/handle-oauth-errors-better.gmi new file mode 100644 index 0000000..462ded5 --- /dev/null +++ b/issues/genenetwork2/handle-oauth-errors-better.gmi @@ -0,0 +1,17 @@ +# Handle OAuth Errors Better + +## Tags + +* type: bug +* status: open +* priority: high +* assigned: fredm +* interested: zachs, robw +* keywords: gn2, genenetwork2, ui, user interface, oauth, oauth errors + +## Description + +When a session expires, for whatever reason, a notification is displayed to the user as shown in the image below: +=> ./session_expiry_oauth_error.png + +The message is a little jarring to the end user. Make it gentler, and probably more informative, so the user is not as surprised. diff --git a/issues/genenetwork2/session_expiry_oauth_error.png b/issues/genenetwork2/session_expiry_oauth_error.png Binary files differnew file mode 100644 index 0000000..34e2dda --- /dev/null +++ b/issues/genenetwork2/session_expiry_oauth_error.png diff --git a/issues/genenetwork3/broken-aliases.gmi b/issues/genenetwork3/broken-aliases.gmi new file mode 100644 index 0000000..5735a1c --- /dev/null +++ b/issues/genenetwork3/broken-aliases.gmi @@ -0,0 +1,27 @@ +# Broken Aliases + +## Tags + +* type: bug +* status: open +* priority: high +* assigned: fredm +* interested: pjotrp +* keywords: aliases, aliases server + + +## Repository + +=> https://github.com/genenetwork/gn3 + +## Bug Report + +### Actual + +* Go to https://genenetwork.org/gn3/gene/aliases2/Shh,Brca2 +* Not that an exception is raised, with a "404 Not Found" message + +### Expected + +* We expected a list of aliases to be returned for the given symbols as is done in https://fallback.genenetwork.org/gn3/gene/aliases2/Shh,Brca2 + diff --git a/issues/genenetwork3/check-for-mandatory-settings.gmi b/issues/genenetwork3/check-for-mandatory-settings.gmi new file mode 100644 index 0000000..16a2f8a --- /dev/null +++ b/issues/genenetwork3/check-for-mandatory-settings.gmi @@ -0,0 +1,40 @@ +# Check for Mandatory Settings + +## Tags + +* status: open +* priority: high +* type: bug, improvement +* interested: fredm, bonz +* assigned: jnduli, rookie101 +* keywords: GN3, gn3, genenetwork3, settings, config, configs, configurations + +## Explanation + +Giving defaults to some important settings leads to situations where the correct configuration is not set up correctly leading at best to failure, and at worst, to subtle failures that can be difficult to debug: e.g. When a default URI to a server points to an active domain, just not the correct one. + +We want to make such (arguably, sensitive) configurations explicit, and avoid giving them defaults. We want to check that they are set up before allowing the application to run, and fail loudly and obnoxiously if they are not provided. + +Examples of configuration variables that should be checked for: + +* All external URIs (external to app/repo under consideration) +* All secrets (secret keys, salts, tokens, etc) + +We should also eliminate from the defaults: + +* Computed values +* Calls to get values from ENVVARs (`os.environ.get(…)` calls) + +### Note on ENVVARs + +The environment variables should be used for overriding values under specific conditions, therefore, it should both be explicit and the last thing loaded to ensure they actually override settings. + +=> https://git.genenetwork.org/gn-auth/tree/gn_auth/__init__.py?id=3a276642bea934f0a7ef8f581d8639e617357a2a#n70 See this example for a possible way of allowing ENVVARs to override settings. + +The example above could be improved by maybe checking for environment variables starting with a specific value, e.g. the envvar `GNAUTH_SECRET_KEY` would override the `SECRET_KEY` configuration. This allows us to override settings without having to change the code. + +## Tasks + +* [ ] Explicitly check configs for ALL external URIs +* [ ] Explicitly check configs for ALL secrets +* [ ] Explicitly load ENVVARs last to override settings diff --git a/issues/genenetwork3/generate-heatmaps-failing.gmi b/issues/genenetwork3/generate-heatmaps-failing.gmi new file mode 100644 index 0000000..03256e6 --- /dev/null +++ b/issues/genenetwork3/generate-heatmaps-failing.gmi @@ -0,0 +1,30 @@ +# Generate Heatmaps Failing + +## Tags + +* type: bug +* status: open +* priority: medium +* assigned: fredm, zachs, zsloan +* keywords: genenetwork3, gn3, GN3, heatmaps + +## Reproduce + +* Go to https://genenetwork.org/ +* Under "Select and Search" menu, enter "synap*" for the "Get Any" field +* Click "Search" +* In search results page, select first 10 traits +* Click "Add" +* Under "Create a new collection" enter the name "newcoll" and click "Create collection" +* In the collections page that shows up, click "Select All" once +* Ensure all the traits are selected +* Click "Generate Heatmap" and wait +* Note how system fails silently with no heatmap presented + +### Notes + +On https://gn2-fred.genenetwork.org the heatmaps fails with a note ("ERROR: undefined"). In the logs, I see "Module 'scipy' has no attribute 'array'" which seems to be due to a change in numpy. +=> https://github.com/MaartenGr/BERTopic/issues/1791 +=> https://github.com/scipy/scipy/issues/19972 + +This issue should not be present with python-plotly@5.20.0 but since guix-bioinformatics pins the guix version to `b0b988c41c9e0e591274495a1b2d6f27fcdae15a`, we are not able to pull in newer versions of packages from guix. diff --git a/issues/genotype_search_bug.gmi b/issues/genotype_search_bug.gmi index 40dba2d..0f05f4e 100644 --- a/issues/genotype_search_bug.gmi +++ b/issues/genotype_search_bug.gmi @@ -4,7 +4,7 @@ * type: bug * priority: medium -* status: open +* status: closed * assigned: zsloan * keywords: bug, search diff --git a/issues/gn-auth/implement-redirect-on-login.gmi b/issues/gn-auth/implement-redirect-on-login.gmi index 061957d..342b2e6 100644 --- a/issues/gn-auth/implement-redirect-on-login.gmi +++ b/issues/gn-auth/implement-redirect-on-login.gmi @@ -5,7 +5,7 @@ * assigned: alexm * priority: medium * status: in progress -* keywords: gn-auth, auth, redirect, login, completed, closed +* keywords: gn-auth, auth, redirect, login, completed, closed, done * type: feature-request ## Description diff --git a/issues/gnqa/gnqa_integration_to_global_search_Design.gmi b/issues/gnqa/gnqa_integration_to_global_search_Design.gmi index 4ef57f1..0d5afd0 100644 --- a/issues/gnqa/gnqa_integration_to_global_search_Design.gmi +++ b/issues/gnqa/gnqa_integration_to_global_search_Design.gmi @@ -4,7 +4,7 @@ * assigned: jnduli, alexm * keywords: llm, genenetwork2 * type: feature -* status: complete +* status: complete, closed, done ## Description This document outlines the design proposal for integrating GNQA into the Global Search feature. diff --git a/issues/mgamma/mgamma-design.gmi b/issues/mgamma/mgamma-design.gmi index 23e02d5..ed4c061 100644 --- a/issues/mgamma/mgamma-design.gmi +++ b/issues/mgamma/mgamma-design.gmi @@ -7,3 +7,31 @@ We have a lot of experience running and hacking the GEMMA tool in GeneNetwork.or GEMMA proves to give great GWA results and has a decent speed for a single threaded implementation - even though the matrix calls to openblas use multiple threads. The source code base of GEMMA, however, proves hard to build on. This is why we are creating a next generation tool that has a focus on *performance and hackability*. After several attempts using R, D, Julia, python, Ruby we have in 2023 settled on Guile+C+Zig. Guile provides a REPL and great hackabability. C+Zig we'll use for performance. The other languages are all great, but we think we can work faster in this setup. + +Well, it is the end of 2024 and we have ditched that effort. Who said life was easy! The guile interface proved problematic - and Zig went out of favour because of its bootstrap story which prevents it becoming part of Guix, Debian etc. Also I discovered new tensor MPUs support f64 - so we may want to support vector and matrix computations on these cores. + +To write a gemma replacement I am now favouring to chunk up existing gemma and make sure its components can talk with alternative implementations. We may use a propagated network approach. Critical is to keep the data in RAM, so it may need some message passing interface with memory that can be shared. The chunking into CELLs (read propagator network PN) is a requirement because we kept tripping over state in GEMMA. So a PN should make sure we can run two implementations of the same CELL and compare outcomes for testing. Also it will allow us to test AVX, tensor and (say) MKL or CUDA implementations down the line. Also it should allow us to start using new functionality on GN faster. It would also be fun to have an implementation run on the RISC-V manycore. + +So, what do we want out of our languages: + +* Nice matrix interface (Julia) +* Support for AVX (Julia) +* Possibility to drop to low level C programming (Julia+prescheme+C?) +* High level -- PN -- glue (Julia+Guile?) + +Julia looks like a great candidate, even though it has notable downsides including the big 'server' blob deployment and the garbage collector (the latter also being a strength, mind). Alternatives could be Rust and Prescheme which have no such concerns, but lack the nice matrix notation. + +The approach will be to start with Julia and reimplementing GEMMA functions so they can be called from Julia and/or guile. + +Oh, I just found out that Julia, like zig, is no longer up-to-date on Debian. And the Guix version is 2 years old. That is really bad. If these languages don't get supported on major distros it is a dead end! + +=> https://mastodon.social/@pjotrprins/113379842047170785 + +What to now? + +* Nice matrix interface (?) +* Support for AVX (?) +* Possibility to drop to low level C programming (?+prescheme+C?) +* High level -- PN -- glue (?+Guile?) + +Current candidates for ? are Nim and Rust. Neither has a really nice matrix interface - though Nim's is probably what I prefer and it is close to python. Chicken may work too when I get fed with mentioned two languages. diff --git a/issues/old_session_bug.gmi b/issues/old_session_bug.gmi index 649ea46..925b9f6 100644 --- a/issues/old_session_bug.gmi +++ b/issues/old_session_bug.gmi @@ -2,7 +2,7 @@ ## Tags -* status: open +* status: closed * priority: medium * type: bug * assigned: zsloan, fredm diff --git a/issues/rdf/automate-rdf-generation-and-ingress.gmi b/issues/rdf/automate-rdf-generation-and-ingress.gmi new file mode 100644 index 0000000..ef4ba9f --- /dev/null +++ b/issues/rdf/automate-rdf-generation-and-ingress.gmi @@ -0,0 +1,37 @@ +# Update RDF Generation and Ingress to Virtuoso + +## Tags + +* assigned: bonfacem +* priority: high +* tags: in-progress +* deadline: 2024-10-23 Wed + +We need to update Virtuoso in production. At the moment this is done manually. For the current set-up, we need to update the recent modified RIF+WIKI models: + + +``` +# Generate the RDF triples +time guix shell guile-dbi guile-hashing -m manifest.scm -- ./pre-inst-env ./examples/generif.scm --settings conf.scm --output /home/bonfacem/ttl-files/generif-metadata-new.ttl --documentation ./docs/generif-metadata.md + +# Make sure they are valid +guix shell -m manifest.scm -- rapper --input turtle --count /home/bonfacem/ttl-files/generif-metadata-new.ttl + +# Copy the files over to the exposed virtuoso path +cp /home/bonfacem/ttl-files/generif-metadata-new.ttl </some/dir/> + +# Get into Virtuoso (with a password) +guix shell virtuoso-ose -- isql <port-number> + +# Load the files to be loaded +# Assuming that '/var/lib/data' is where the files are +ld_dir('/var/lib/data', 'generif-metadata-new.ttl', 'http://genenetwork.org'); + +# Load the files +rdf_loader_run(); +CHECKPOINT; +``` + +Above steps should be automated and tested in CD before roll-out in production. Key considerations: + +- Pick latest important changes from git, so that we can pick what files to run instead of generating all the ttl files all the time. diff --git a/issues/redesign-global-search-design.gmi b/issues/redesign-global-search-design.gmi index a518fd5..df63791 100644 --- a/issues/redesign-global-search-design.gmi +++ b/issues/redesign-global-search-design.gmi @@ -4,16 +4,20 @@ * assigned: alexm, zac * keywords: global search, design, HTML * type: enhancement -* status: In progress +* status: closed, completed, done ## Description Rob suggested we model the global search on the NCBI PubMed interface. We should remove the `?` button, which seems to be confusing for users, and have a better user guide. ## Tasks -* [] Redesign the global search to fit the NCBI PubMed model. -* [] Replace the "?" button that acts as a user guide +* [x] Redesign the global search to fit the NCBI PubMed model. +* [x] Replace the "?" button that acts as a user guide ## Related issues: -=> https://issues.genenetwork.org/issues/cleanup-base-file-gn2
\ No newline at end of file +=> https://issues.genenetwork.org/issues/cleanup-base-file-gn2 + +## Notes +PR that seeks to address this issue: +=> https://github.com/genenetwork/genenetwork2/pull/880
\ No newline at end of file diff --git a/issues/replace-neo4j-with-virtuoso.gmi b/issues/replace-neo4j-with-virtuoso.gmi new file mode 100644 index 0000000..450fb70 --- /dev/null +++ b/issues/replace-neo4j-with-virtuoso.gmi @@ -0,0 +1,8 @@ +# Replace Neo4J with Virtuoso + +## Tags + +* assigned: bonfacem, soloshelby +* deadline: 2024-10-25 Fri + +Currently, the RAG ingests TTL files into Neo4J. Replace this with Virtuoso. diff --git a/issues/set-up-gn-guile-in-tux02.gmi b/issues/set-up-gn-guile-in-tux02.gmi index 2ac0cd6..29eca68 100644 --- a/issues/set-up-gn-guile-in-tux02.gmi +++ b/issues/set-up-gn-guile-in-tux02.gmi @@ -5,10 +5,11 @@ * assigned: bonfacem * priority: high * status: in-progress +* deadline: 2024-10-23 Wed ## Tasks -* [ ] Create gn-guile container. +* [-] Create gn-guile container. * [X] Merge gn2 UI PR. => https://github.com/genenetwork/genenetwork2/pull/854 Feature/gn editor UI -* [ ] Test out auth editing in CD. +* [-] Test out auth editing in CD. diff --git a/issues/set-up-virtuoso-on-production.gmi b/issues/set-up-virtuoso-on-production.gmi index a81b480..614565a 100644 --- a/issues/set-up-virtuoso-on-production.gmi +++ b/issues/set-up-virtuoso-on-production.gmi @@ -77,6 +77,8 @@ SQL> DELETE FROM rdf_quad WHERE g = iri_to_id('http://genenetwork.org'); SQL> ld_dir('/var/lib/virtuoso', '*.ttl', 'http://genenetwork.org'); SQL> rdf_loader_run(); SQL> CHECKPOINT; +SQL> checkpoint_interval(60); +SQL> scheduler_interval(10); ``` * Verify you have some RDF data by running: diff --git a/issues/xapian_bug.gmi b/issues/xapian_bug.gmi index f11b604..068d8eb 100644 --- a/issues/xapian_bug.gmi +++ b/issues/xapian_bug.gmi @@ -5,6 +5,7 @@ * assigned: zsloan * priority: high * type: search +* status: closed * keywords: xapian, gn2, gn3 ## Description diff --git a/tasks/pjotrp.gmi b/tasks/pjotrp.gmi index 9c2f57f..680c2a1 100644 --- a/tasks/pjotrp.gmi +++ b/tasks/pjotrp.gmi @@ -12,21 +12,29 @@ The tasks here should probably be broken out into appropriately tagged issues, w => https://issues.genenetwork.org +Generally work applies to NIH/R073237482 and other grants. + # Current ## Systems -- [X] mount NFS bacchus on octopus -- [ ] drop tux02 backups on balg01 -- [ ] reintroduce borg-borg +* [ ] make sure production is up to scratch +* [ ] backup tux04 +* [ ] add drives to Octopus 1U01HG013760 +* [ ] add Ceph for distributed network storage 1U01HG013760 +* [ ] drop tux02 backups on balg01 +* [ ] reintroduce borg-borg ## Ongoing tasks (current/urgent) * [ ] Precompute * [+] Set up stable GeneNetwork server instance with new hardware (see below) => /topics/systems/fire-up-genenetwork-system-container.gmi +* [ ] Check email setup tux04 * [+] Julia as part of GN3 deployment => /topics/deploy/julia.gmi +* [ ] Work on pangenome genotyping 1U01HG013760 +* [+] Moni builds 1U01HG013760 # Tasks @@ -61,7 +69,7 @@ Longer term Later -* [ ] Mempang25 +* [ ] Mempang25 1U01HG013760 + [ ] Invites + [ ] Payments + [ ] Rooms @@ -94,6 +102,7 @@ Later ### Done +* [X] mount NFS bacchus on octopus * [X] Guix project server * [X] Build junior team * [X] Complete Octopus setup with Erik, Andrea and Efraim - adding all machines diff --git a/topics/genenetwork/genenetwork-services.gmi b/topics/genenetwork/genenetwork-services.gmi new file mode 100644 index 0000000..717fdd8 --- /dev/null +++ b/topics/genenetwork/genenetwork-services.gmi @@ -0,0 +1,122 @@ +# GeneNetwork Services + +## Tags + +* type: documentation +* keywords: documentation, docs, doc, services, genenetwork services + +## GeneNetwork Core Services + +GeneNetwork is composed of a number of different services. This document attempts to document all the services that make up GeneNetwork and document what links give access to the services. + +### GeneNetwork2 + +This is the main user-interface to the entire GeneNetwork system. + +#### Links + +=> https://github.com/genenetwork/genenetwork2 Repository +=> https://genenetwork.org/ GN2 on production +=> https://fallback.genenetwork.org/ GN2 on old production +=> https://cd.genenetwork.org/ GN2 on CI/CD +=> https://staging.genenetwork.org/ GN2 on staging + +### GeneNetwork3 + +This is the main API server for GeneNetwork. + +#### Links + +=> https://github.com/genenetwork/genenetwork3 Repository +=> https://genenetwork.org/api3/ GN3 on production +=> https://fallback.genenetwork.org/api3/ GN3 on old production +=> https://cd.genenetwork.org/api3/ GN3 on CI/CD +=> https://staging.genenetwork.org/api3/ GN3 on staging + +### Sparql Service + +The SparQL service is served from a Virtuoso-OSE service. + +=> https://issues.genenetwork.org/topics/deploy/our-virtuoso-instances We have notes on our virtuoso instances here. + + +#### Links + +=> https://github.com/genenetwork/genenetwork3 Repository +=> https://sparql.genenetwork.org/sparql/ sparql-service on production +* ??? sparql-service on old production +* ??? sparql-service on CI/CD +* ??? sparql-service on staging + +### GN-Auth + +This is the authorisation server for the GeneNetwork system. + +#### Links + +=> https://git.genenetwork.org/gn-auth/ Repository +=> https://auth.genenetwork.org/ gn-auth on production +=> https://fallback.genenetwork.org/gn-auth/ gn-auth on old production +* ??? gn-auth on CI/CD +=> https://staging-auth.genenetwork.org/ gn-auth on staging + +### GN-Uploader + +This service is to be used for uploading data to GeneNetwork. It is currently in development (best case, alpha). + +#### Links + +=> https://git.genenetwork.org/gn-uploader/ Repository +* ??? gn-uploader on production +* ??? gn-uploader on old production +* ??? gn-uploader on CI/CD +=> https://staging-uploader.genenetwork.org/ gn-uploader on staging + +### Aliases Server + +An extra server to respond with aliases for genetic (etc.) symbols. + +This is currently a project in racket, but we should probably pull in the features in this repository into one of the others (probably GeneNetwork3) and trash this repository. + +#### Links + +=> https://github.com/genenetwork/gn3 Repository +=> https://genenetwork.org/gn3/ aliases-server on production +=> https://fallback.genenetwork.org/gn3/ aliases-server on old production +=> https://cd.genenetwork.org/gn3/ aliases-server on CI/CD +=> https://staging.genenetwork.org/gn3/ aliases-server on staging + +### Markdown Editing Server + +#### Links + +=> https://git.genenetwork.org/gn-guile/ Repository +=> https://genenetwork.org/facilities/ markdown-editing-server on production +=> https://fallback.genenetwork.org/facilities/ markdown-editing-server on old production +=> https://cd.genenetwork.org/facilities/ markdown-editing-server on CI/CD +=> https://staging.genenetwork.org/facilities/ markdown-editing-server on staging + +## Support Services + +These are other services that support the development and maintenance of the core services. + +### Issue Tracker + +We use a text-based issue tracker that is accessible via +=> https://issues.genenetwork.org/ + +The repository for this service is at +=> https://github.com/genenetwork/gn-gemtext-threads/ + +### Repositories Server + +This is where a lot of the genenetwork repositories live. You can access it at +=> https://git.genenetwork.org/ + +### Continuous Integration Service + +… + +=> https://ci.genenetwork.org/ + +### … diff --git a/topics/guix/guix-profiles.gmi b/topics/guix/guix-profiles.gmi index 578bb82..8cf41d8 100644 --- a/topics/guix/guix-profiles.gmi +++ b/topics/guix/guix-profiles.gmi @@ -16,7 +16,7 @@ Alternatively put the following into a channels.scm file. ``` (list (channel (name 'gn-bioinformatics) - (url "https://gitlab.com/genenetwork/guix-bioinformatics") + (url "https://git.genenetwork.org/guix-bioinformatics") (branch "master"))) ``` Build a profile using diff --git a/topics/gunicorn/deploying-app-under-url-prefix.gmi b/topics/gunicorn/deploying-app-under-url-prefix.gmi new file mode 100644 index 0000000..b2e382f --- /dev/null +++ b/topics/gunicorn/deploying-app-under-url-prefix.gmi @@ -0,0 +1,121 @@ +# Deploying Your Flask Application Under a URL Prefix With GUnicorn + +## TAGS + +* type: doc, documentation, docs +* author: fredm, zachs +* keywords: flask, gunicorn, SCRIPT_NAME, URL prefix + +## Introduction + +You have your application and are ready to deploy it, however, for some reason, you want to deploy it under a URL prefix, rather than at a top-level-domain. + +This short article details the things you need to set up. + +## Set up Your WebServer (Nginx) + +You need to tell your webserver to serve the application under a particular url prefix. You do this using that particular webserver's reverse-proxying configurations: For this article, we will use Nginx as the server. + +Normally, you'd simply do something like: + +``` +server { + server_name your.server.domain + + ⋮ + + location /the-prefix/ { + proxy_pass http://127.0.0.1:8080/; + proxy_set_header Host $host; + ⋮ + } + + ⋮ +} +``` + +Here, your top-level domain will be https://your.server.domain and you therefore want to access your shiny new application at https://your.server.domain/the-prefix/ + +For a simple application, with no sessions or anything, this should work, somewhat, though you might run into trouble with things like static files (e.g. css, js, etc) if the application does not use the same ones as that one on the TLD. + +If you are using sessions, you might also run into an issue where there is an interaction in the session management of both applications, especially if the application on the TLD makes use of services from the application at the url prefix. This is mostly due to redirects from the url-prefix app getting lost and hitting the TLD app. + +To fix this, we change the configuration above to: + +``` +server { + server_name your.server.domain + + ⋮ + + location /the-prefix/ { + proxy_pass http://127.0.0.1:8080/the-prefix/; + proxy_set_header Host $host; + ⋮ + } + + ⋮ +} +``` + +but now, you get errors, since there is no endpoint in your shiny new app that in at the route /the-prefix/***. + +Enter Gunicorn! + + +## Setting up SCRIPT_NAME for GUnicorn + +### The "Hacky" Way + +At the point of invocation of GUnicorn, we set the SCRIPT_NAME environment variable to the value "/the-prefix" — note that there is no trailing slash; this is very important. You should now have something like: + +``` +$ export SCRIPT_NAME="/the-prefix" +$ gunicorn --bind 0.0.0.0:8082 --workers … +``` + +The first line tells GUnicorn what the URL prefix is. It will use this to compute what URL to pass to the flask application. + +Example, say you try accessing the endpoint + +``` +https://your.server.domain/the-prefix/auth/authorise?response_type=code&client_id=some-id&redirect_uri=some-uri +``` + +Gunicorn will split that URL into 2 parts using the value of the SCRIPT_NAME environment variable, giving you: + +* https://your.server.domain +* /auth/authorise?response_type=code&client_id=some-id&redirect_uri=some-uri + +It will then pass on the second part to flask. This is why the value of SCRIPT_NAME should not have a trailing slash. + +Note that using the SCRIPT_NAME environment variable is a convenience feature provided by GUnicorn, not a WSGI feature. If you ever change your WSGI server, there is no guarantee this fix will work. + +### Using WSGI Routing MiddleWare + +A better way is to make use of a WSGI routing middleware. You could do this by defining a separate WSGI entry point in your application's repository. + +``` +# wsgi_url_prefix.py +from werkzeug.wrappers import Response +from werkzeug.middleware.dispatcher import DispatcherMiddleware + +from app import create_app + +def init_prefixed_app(theapp): + theapp.wsgi_app = DispatcherMiddleware( + Response("Not Found", 404), + { + "/the-prefix": the_app.wsgi_app + }) + return theapp + + +app = init_prefixed_app(create_app()) +``` + +## References + +=> https://docs.gunicorn.org/en/latest/faq.html#how-do-i-set-script-name +=> https://dlukes.github.io/flask-wsgi-url-prefix.html +=> https://www.reddit.com/r/Python/comments/juwj3x/comment/gchdsld/ diff --git a/topics/meetings/jnduli_bmunyoki.gmi b/topics/meetings/jnduli_bmunyoki.gmi index c4832c6..8afe4f1 100644 --- a/topics/meetings/jnduli_bmunyoki.gmi +++ b/topics/meetings/jnduli_bmunyoki.gmi @@ -1,22 +1,39 @@ # Meeting Notes -## 2024-10-11 -* @priscilla @flisso: Try out API endpoints that don't require auth. -* @jnduli Harden hook system for gn-auth. +## 2024-10-18 +* @priscilla @flisso: Set up mariadb and virtuoso to test out some GN3 endpoints. +* @priscilla @flisso @bmunyoki: Improve docs while hacking on the above. * @jnduli Remove gn-auth code from GN3. -* @jnduli @bonfacem Finish up RIF Editing project. -* @jnduli @alexm Create issue on describing the monitoring system. -* @jnduli @alexm Create issue on prompt engineering in GN to improve what we already have. -* @alex Work on R/Qtl. NOTE: @jnduli/@bonfacem help out with this. -* @alex: Review @bmunyoki's work on RIF/Indexing. -* @flisso: Make sure we have C Elegans dataset and MIKK genotypes to production. -* @flisso: Make sure we have HS Rats in testing stage. -* @flisso: Make progress in learning back-end coding WRT GN. +* @jnduli Resolve current issue with broken auth in gn-qa. +* @jnduli @alexm Work on the R/Qtl design doc. +* @flisso MIKK genotyping. +* @flisso Make sure we have C Elegans and HS Rats dataset to testing, and have the genotyping pipeline working. * @shelbys: Modify existing Grant write-up for pangenomes. -* @shelbys: Finish getting all the R2R scores from the first study. +* @shelbys @bonfacem: Getting RDF into R2R. * @bonfacem RIF Indexing for RIF page in Xapian. * @bonfacem Work on properly containerizing gn-guile. +## 2024-10-11 +* WIP @priscilla @flisso: Try out API endpoints that don't require auth. NOTE: Priscilla got to set-up guix channels for gn3. Felix ran into problems. Priscilla set up the MySQL in her Ubuntu system. +* NOT DONE: @jnduli Harden hook system for gn-auth. +* WIP: @jnduli Remove gn-auth code from GN3. NOTE: Sent latest patches to Fred. Running issue, some patches may have caused gn-qa to fail. +* DONE: @jnduli @bonfacem Finish up RIF Editing project. +* NOT DONE: @jnduli @alexm Create issue on describing the monitoring system. +* NOT DONE: @jnduli @alexm Create issue on prompt engineering in GN to improve what we already have. +* WIP: @alex Work on R/Qtl. NOTE: @jnduli/@bonfacem help out with this. NOTE: Finished writing the design doc for gn-qa. +* DONE: Looked at documentation for R/Qtl. +* NOT DONE: @alex: Review @bmunyoki's work on RIF/Indexing. +* WIP: @flisso: Make sure we have C Elegans dataset and MIKK genotypes to production. NOTE: Issues with data entry scripts. Fred/Zach working to set up test environment. +* WIP: @flisso: MIKK genotyping. NOTE: Still testing the pipeline. Halfway there. +* NOT DONE: @flisso: Make sure we have HS Rats in testing stage. +* WIP: @flisso: Make progress in learning back-end coding WRT GN. NOTE: Issue setting up GN3. +* WIP: @shelbys: Modify existing Grant write-up for pangenomes. NOTE: Reviewed by Pj and Eric. More mods based of feedback. Paper got accepted by BioArxiv. Added some docs to R2R evaluation code. +* DONE: @shelbys: Finish getting all the R2R scores from the first study. NOTE: Got scores for all the scores from first papers using R2R instead of Fahamu. +* NOT DONE: @bonfacem RIF Indexing for RIF page in Xapian. +* WIP: @bonfacem Work on properly containerizing gn-guile. +* DONE: @bonfacem Fix the gn-transform-database in CI. Sent patches to Arun for review. +* DONE: @bonfacem Fixed broken utf-8 characters in gn-gemtext. + ## 2024-10-04 * IN PROGRESS: @priscilla @bonfacem Setting up GN3. @priscilla try out API endpoints that don't require auth. NOTE: @priscilla Able to set up guix as a package manager. Trouble with Guix set-up with GN3. @bonfacem good opportunity to improve docs in GN3. * IN PROGRESS: @jnduli Harden hook system for gn-auth. diff --git a/topics/octopus/set-up-guix-for-new-users.gmi b/topics/octopus/set-up-guix-for-new-users.gmi new file mode 100644 index 0000000..f459559 --- /dev/null +++ b/topics/octopus/set-up-guix-for-new-users.gmi @@ -0,0 +1,38 @@ +# Set up Guix for new users + +This document describes how to set up Guix for new users on a machine in which Guix is already installed (such as octopus01). + +## Create a per-user profile for yourself by running your first guix pull + +"Borrow" some other user's guix to run guix pull. In the example below, we use root's guix, but it might as well be any guix. +``` +$ /var/guix/profiles/per-user/root/current-guix/bin/guix pull +``` +This should create your very own Guix profile at ~/.config/guix/current. You may invoke guix from this profile as +``` +$ ~/.config/guix/current/bin/guix ... +``` +But, you'd normally want to make this more convenient. So, add ~/.config/guix/current/bin to your PATH. To do this, add the following to your ~/.profile +``` +GUIX_PROFILE=~/.config/guix/current +. $GUIX_PROFILE/etc/profile +``` +Thereafter, you may run any guix command simply as +``` +$ guix ... +``` + +## Pulling from a different channels.scm + +By default, guix pull pulls the latest commit of the main upstream Guix channel. You may want to pull from additional channels as well. Put the channels you want into ~/.config/guix/channels.scm, and then run guix pull. For example, here's a channels.scm if you want to use the guix-bioinformatics channel. +``` +$ cat ~/.config/guix/channels.scm +(list (channel + (name 'gn-bioinformatics) + (url "https://git.genenetwork.org/guix-bioinformatics") + (branch "master"))) +``` +And, +``` +$ guix pull +``` diff --git a/topics/octopus/slurm-upgrade.gmi b/topics/octopus/slurm-upgrade.gmi index 1f09a79..822f68e 100644 --- a/topics/octopus/slurm-upgrade.gmi +++ b/topics/octopus/slurm-upgrade.gmi @@ -14,7 +14,7 @@ slurm can only be upgraded safely in small version increments. For example, it i Stop the slurmdbd, slurmctld and slurmd services. ``` -# systemctl stop slurmdbd slurmctld slurmd +# systemctl stop slurmdbd slurmctld slurmd slurmrestd ``` Backup the slurm StateSaveLocation (/var/spool/slurmd/ctld in our case) and the slurm configuration directory. ``` @@ -50,6 +50,7 @@ Reload the new systemd configuration files. Then, start the slurmdbd, slurmctld # systemctl start slurmdbd # systemctl start slurmctld # systemctl start slurmd +# systemctl start slurmrestd ``` ## Upgrade slurm on the worker nodes @@ -74,14 +75,14 @@ Copy over any configuration file changes from octopus01. Then, reload the new sy It is a lot of typing to run the same command on all worker nodes. You could make this a little less cumbersome with the following bash for loop. ``` -for node in octopus02 octopus03 octopus05 octopus06 octopus07 octopus08 octopus09 octopus10 octopus11 tux06 tux07 tux08 tux09; +for node in octopus02 octopus03 octopus05 octopus06 octopus07 octopus08 octopus09 octopus10 octopus11 tux05 tux06 tux07 tux08 tux09; do ssh $node your command done ``` You can even do this for sudo commands using the -S flag of sudo that makes it read the password from stdin. Assuming your password is in the pass password manager, the bash for loop would then look like: ``` -for node in octopus02 octopus03 octopus05 octopus06 octopus07 octopus08 octopus09 octopus10 octopus11 tux06 tux07 tux08 tux09; +for node in octopus02 octopus03 octopus05 octopus06 octopus07 octopus08 octopus09 octopus10 octopus11 tux05 tux06 tux07 tux08 tux09; do pass octopus | ssh $node sudo -S your command done diff --git a/topics/systems/restore-backups.gmi b/topics/systems/restore-backups.gmi index 518c56d..b97af2b 100644 --- a/topics/systems/restore-backups.gmi +++ b/topics/systems/restore-backups.gmi @@ -26,7 +26,7 @@ The last backup on 'tux02' is from October 2022 - after I did a reinstall. That According to sheepdog the drops are happening to 'space' and 'epysode', but 'tux02' is missing: -=> https://rabbit.genenetwork.org/sheepdog/index.html +=> http://sheepdog.genenetwork.org/sheepdog/status.html ## Mariadb |
