diff options
Diffstat (limited to 'topics/deploy')
| -rw-r--r-- | topics/deploy/our-virtuoso-instances.gmi | 2 | ||||
| -rw-r--r-- | topics/deploy/setting-up-or-migrating-production-across-machines.gmi | 156 | ||||
| -rw-r--r-- | topics/deploy/uthsc-email.gmi | 64 | ||||
| -rw-r--r-- | topics/deploy/uthsc-vpn-with-free-software.gmi | 23 | ||||
| -rw-r--r-- | topics/deploy/uthsc-vpn.scm | 104 |
5 files changed, 332 insertions, 17 deletions
diff --git a/topics/deploy/our-virtuoso-instances.gmi b/topics/deploy/our-virtuoso-instances.gmi index 0336018..3ac56ae 100644 --- a/topics/deploy/our-virtuoso-instances.gmi +++ b/topics/deploy/our-virtuoso-instances.gmi @@ -9,6 +9,8 @@ We run three instances of virtuoso. The public SPARQL endpoint is accessible at => https://sparql.genenetwork.org/sparql +These are now generally run as part of genenetwork2 containers(!) + ## Configuration All our virtuoso instances are deployed in Guix system containers. The configuration for these containers is at diff --git a/topics/deploy/setting-up-or-migrating-production-across-machines.gmi b/topics/deploy/setting-up-or-migrating-production-across-machines.gmi index 1f35dae..631a000 100644 --- a/topics/deploy/setting-up-or-migrating-production-across-machines.gmi +++ b/topics/deploy/setting-up-or-migrating-production-across-machines.gmi @@ -15,24 +15,168 @@ Recent events (Late 2024 and early 2025) have led to us needing to move the prod In this respect, a number of tasks rise to the front as necessary to accomplish for a successful migration. Each of the following sections will detail a task that's necessary for a successful migration. +## Copy Over Auth Database + +We need to synchronise the authorisation database. We can copy this over from the production system, or the backups + +* TODO: Indicate where the backups for the auth database are here! + +Steps (flesh out better): + +* Extract backup (or copy from existing production system) +* Stop the (new) container (if it's running) +* Backup the (new) container's auth-db file ( +* Place the auth db file in the correct place in the container's filesystem: +* Backup existing secrets +* Login to the `/auth/admin/dashboard` of the auth server (e.g. https://cd.genenetwork.org/auth/admin/dashboard) +* If client with the CLIENT_ID in the secrets exists +* 1. update the uris for that client, if it doesn't exist, create an entirely new client and replace both the CLIENT_ID and CLIENT_SECRET in the secrets file. +* 2. Click on the "Change Secret" button and generate a new secret. Replace the secret in the secrets file with the newly generated secret +* If client with the CLIENT_ID in the secrets DOES NOT exist, register a new client, setting up the appropriate URIs and endpoints, and then add/replace both the CLIENT_ID and CLIENT_SECRET in the secrets file. +* Restart (new) container + ## Set Up the Database -* Extract: detail this — link to existing document in this repo. Also, probably note that we symlink the extraction back to `/var/lib/mysql`? -* Configure: detail this — link to existing document in this repo +=> /topics/systems/restore-backups Extract the latest database from the backups. +=> /topics/deploy/installation Configure MariaDB according to this document. ## Set Up the File System * TODO: List the necessary directories and describe what purpose each serves. This will be from the perspective of the container — actual paths on the host system are left to the builders choice, and can vary wildly. * TODO: Prefer explicit binding rather than implicit — makes the shell scripts longer, but no assumptions have to be made, everything is explicitly spelled out. +The container(s) need access to various files and directories from the host system in order to work correctly. + +Filesystem bindings could be linked to wildly different paths on different physical host machines, therefore, we shall examine the bindings from the point of view of the paths within the container, rather than forcing a particular file system layout on the host systems themselves. + +Each of the sections below details a specific binding: + +### /var/genenetwork + +This binding must be READWRITE within the container. + +The purpose is to hold varying files that are specific to the genenetwork system(s). Examples of the files are: + +* "gn-meta" and "synteny" files for GN3 +* genotype files +* session files for various systems (GN2, gn-uploader, etc.) + +### /var/lib/acme + +This binding must be READWRITE within the container. + +This is used to store TLS certificates for the various services within the container by the ACME (Automatic Certificate Management Environment) script. + +### /var/lib/redis + +This binding must be READWRITE within the container. + +This is used by the redis daemon to persist its state(s). + +### /var/lib/virtuoso + +This binding must be READWRITE within the container. + +Used by the virtuoso daemon to save its state, and maybe some log files. + +### /export/data/virtuoso/ + +This binding must be READONLY within the container. (Really?) + +This is used for importing data into virtuoso, say by sharing Turtle (TTL) files within the binding. + +--- +At this point the binding is READONLY because any TTL files to load are imported from outside the container. If the transformation of data from MariaDB to TTL form is built into the production containers down the line, then this might change to READWRITE to allow the tranformation tool to write to it. + +### /var/log + +This binding must be READWRITE within the container. + +Allows logs from various services running in the container be accessible in the host system. This is useful for debugging issues with the running systems. + +### /etc/genenetwork + +This binding must be READWRITE within the container. + +Useful for storing various configuration files/data for the service(s) running inside the running container. + +### /var/lib/xapian + +This binding must be READWRITE within the container. + +Stores the processed search indexes for the xapian search system. + +### /var/lib/genenetwork/sqlite/gn-auth + +This binding must be READWRITE within the container. + +The authorisation database is stored here. The directory needs to be writable to avoid permissions issues within the container when attempting to write data into the database. + +### /var/lib/genenetwork/sqlite/genenetwork3 + +This binding must be READWRITE within the container. + +This stores various SQLite databases in use with GN3. These are: + +* Database for the GNQA system +* ... + +### /run/mysqld + +This binding must be READWRITE within the container. + +This binding is the link to the host directory that holds the socket file for the running MariaDB instance. + +### /opt/gn/tmp + +This binding must be READWRITE within the container. + +Holds temporary files for the various services that run within the container. Some of the generated files from various services are also stored here. + +**PROPOSAL**: Move all generated files here, or have a dedicated directory for holding generated files? + + +### /var/genenetwork/sessions + +This binding must be READWRITE within the container. + +Holds session files for various services within the container. See also the /var/genenetwork binding. + +### /var/lib/genenetwork/uploader + +This binding must be READWRITE within the container. + +**gn-uploader** specific data files. Types of data files that could go here are: + +* File uploads +* (Reusable) Cache files and generated files +* ... others? + +### /var/lib/genenetwork/sqlite/gn-uploader + +This binding must be READWRITE within the container. + +Holds various SQLite databases used with the **gn-uploader** service, e.g.: + +* Background jobs database +* ... + +### /var/lib/genenetwork/gn-guile + +This binding must be READWRITE within the container. + +Various data files for the **gn-guile** service, such as: + +* The bare **gn-docs** repository (Previously bound at `/export/data/gn-docs`: now deprecated). + ## Redis We currently (2025-06-11) use Redis for: -- Tracking user collection (this will be moved to SQLite database) -- Tracking background jobs (this is being moved out to SQLite databases) -- Tracking running-time (not sure what this is about) -- Others? +* Tracking user collection (this will be moved to SQLite database) +* Tracking background jobs (this is being moved out to SQLite databases) +* Tracking running-time (not sure what this is about) +* Others? We do need to copy over the redis save file whenever we do a migration, at least until the user collections and background jobs features have been moved completely out of Redis. diff --git a/topics/deploy/uthsc-email.gmi b/topics/deploy/uthsc-email.gmi new file mode 100644 index 0000000..05f2ba5 --- /dev/null +++ b/topics/deploy/uthsc-email.gmi @@ -0,0 +1,64 @@ +# UTHSC E-mail + +Similar to many organizations UT uses outlook and exchange for their E-mail. Thanks to mobile support it is possible to work with email using other tools outside the UT network. + +## Prospect E-mail client + +People have success using Prospect as an E-mail client. You can follow the instructions on the UT website that are similar to Android support. + +## Davmail IMAP bridge + +An interesting solution is to create an IMAP bridge. It is a little slower, but can be set up! That way you can use your favorite E-mail tool (and filters!). + +I have had success setting up davmail with the following settings and testing with thunderbird first: + +``` +apt install davmail openjfx thunderbird +``` + +Start davmail with + +``` +davmail -d +``` + +Stop davmail and edit the ~/.davmail.properties file with the following. + +``` +davmail.mode=O365Interactive +davmail.url=https://outlook.office365.com/EWS/Exchange.asmx +davmail.oauth.clientId=d3590ed6-52b3-4102-aeff-aad2292ab01c +davmail.enableOauth2=true +davmail.oauth.deviceCode=true +davmail.oauth.enableOauth2=true +davmail.oauth.redirectUri=urn:ietf:wg:oauth:2.0:oob +davmail.oauth.tenantId=common +davmail.imapPort=1143 +davmail.smtpPort=1025 +davmail.logFilePath=/home/yours/.davmail/davmail.log +log4j.logger.httpclient.wire=DEBUG +log4j.rootLogger=DEBUG +log4j.logger.org.apache.http.wire=DEBUG +``` + +Restart davmail and point thunderbird to + +``` +IMAP Server: localhost:1143 +SMTP Server: localhost:1025 +Username: your-email@uthsc.edu +``` + +Note that you should set the UT password in the 2FA browser when it pops up. Do *not* set it in Thunderbird, also when it asks for it to send out SMTP. + +When something fails make sure to track the log in ~/.davmail/davmail.log + +# Using Mutt + +Some useful links: + +=> https://jonathanh.co.uk/blog/exchange-mutt/ +=> https://movementarian.org/blog/posts/mutt-and-office365/ +=> https://www.vanormondt.net/~peter/blog/2021-03-16-mutt-office365-mfa.html + +If someone can get the last one to work we won't even need davmail any more! diff --git a/topics/deploy/uthsc-vpn-with-free-software.gmi b/topics/deploy/uthsc-vpn-with-free-software.gmi index 95fd1cd..aeba322 100644 --- a/topics/deploy/uthsc-vpn-with-free-software.gmi +++ b/topics/deploy/uthsc-vpn-with-free-software.gmi @@ -6,14 +6,23 @@ It is possible to connect to the UTHSC VPN using only free software. For this, y To connect, run openconnect-sso as follows. A browser window will pop up for you to complete the Duo authentication. Once done, you will be connected to the VPN. ``` -$ openconnect-sso --server uthscvpn1.uthsc.edu --authgroup UTHSC +$ openconnect-sso --server vpn-server --authgroup UTHSC ``` Note that openconnect-sso should be run as a regular user, not as root. After passing Duo authentication, openconnect-sso will try to gain root priviliges to set up the network routes. At that point, it will prompt you for your password using sudo. ## Recommended way -The recommended way is to use Arun's g-expression setup using guix. See below. It should just work, provided you have the -chained certificate that you can get from the browser or one of us. +The recommended way is to use Arun's g-expression setup using guix. See below. It should just work, provided you have the chained certificate that you can get from the browser or one of us and point to the right server. Simply + +``` +$(guix build -f uthsc-vpn.scm) +``` + +See + +=> ./uthsc-vpn.scm + +Get the final details from us. UT does not like it when we put it online even though there is no real risk. ## Avoid tunneling all your network traffic through the VPN (aka Split Tunneling) @@ -22,7 +31,7 @@ openconnect, by default, tunnels all your traffic through the VPN. This is not g For example, to connect to the UTHSC VPN but only access the hosts tux01 and tux02e through the VPN, run the following command. ``` -$ openconnect-sso --server uthscvpn1.uthsc.edu --authgroup UTHSC -- --script 'vpn-slice tux01 tux02e' +$ openconnect-sso --server vpn-server --authgroup UTHSC -- --script 'vpn-slice tux01 tux02e' ``` The vpn-slice script looks up the hostnames tux01 and tux02e on the VPN DNS and adds /etc/hosts entries and routes to your system. vpn-slice can also set up more complicated routes. To learn more, read the vpn-slice documentation. @@ -52,7 +61,7 @@ Then, run the openconnect-sso client as usual. ## Misconfigured UTHSC TLS certificate The UTHSC TLS certificate does not validate on some systems. You can work around this by downloading the certificate chain and adding it to your system: -* Navigate with browser to https://uthscvpn1.uthsc.edu/. Inspect the certificate in the browser (lock icon next to search bar) and export .pem file +* Navigate with browser to https://vpn-server/. Inspect the certificate in the browser (lock icon next to search bar) and export .pem file * Move it to /usr/local/share/ca-certificates (with .crt extension) or equivalent * On Debian/Ubuntu update the certificate store with update-ca-certificates You should see @@ -65,7 +74,7 @@ Thanks Niklas. See also However, adding certificates to your system manually is not good security practice. It is better to limit the added certificate to the openconnect process. You can do this using the REQUESTS_CA_BUNDLE environment variable like so: ``` -REQUESTS_CA_BUNDLE=/path/to/uthsc/certificate.pem openconnect-sso --server uthscvpn1.uthsc.edu --authgroup UTHSC +REQUESTS_CA_BUNDLE=/path/to/uthsc/certificate.pem openconnect-sso --server vpn-server --authgroup UTHSC ``` ## Putting it all together using Guix G-expressions @@ -93,6 +102,6 @@ export QTWEBENGINE_CHROMIUM_FLAGS=--disable-seccomp-filter-sandbox ## Acknowledgement -Many thanks to Pjotr Prins and Erik Garrison without whose earlier work this guide would not be possible. +Many thanks to Arun, Pjotr and Erik without whose earlier work this guide would not be possible. => https://github.com/pjotrp/linux-at-university-of-tennessee => https://github.com/ekg/openconnect-sso-docker diff --git a/topics/deploy/uthsc-vpn.scm b/topics/deploy/uthsc-vpn.scm index 82f67f5..f204cdf 100644 --- a/topics/deploy/uthsc-vpn.scm +++ b/topics/deploy/uthsc-vpn.scm @@ -1,15 +1,30 @@ -(use-modules ((gnu packages python-web) #:select (python-requests python-urllib3)) +(use-modules ((gnu packages check) + #:select (python-pytest python-pytest-asyncio python-pytest-httpserver)) + ((gnu packages freedesktop) #:select (python-pyxdg)) + ((gnu packages python-build) #:select (python-poetry-core python-toml)) + ((gnu packages python-crypto) + #:select (python-keyring python-pyotp)) + ((gnu packages python-web) #:select (python-requests python-urllib3)) + ((gnu packages python-xyz) + #:select (python-attrs python-charset-normalizer + python-colorama python-prompt-toolkit python-pysocks + python-structlog)) ((gnu packages guile-xyz) #:select (guile-ini guile-lib guile-smc)) - ((gnu packages vpn) #:select (openconnect-sso vpn-slice)) + ((gnu packages qt) #:select (python-pyqt-6 python-pyqtwebengine-6)) + ((gnu packages vpn) #:select (openconnect vpn-slice)) + ((gnu packages xml) #:select (python-lxml-4.9)) + (guix build-system pyproject) (guix build-system python) (guix download) (guix gexp) + (guix git-download) + ((guix licenses) #:prefix license:) (guix packages)) ;; Put in the hosts you are interested in here. (define %hosts (list "octopus01" - "spacex.uthsc.edu")) + "spacex")) (define (ini-file name scm) "Return a file-like object representing INI file with @var{name} and @@ -36,6 +51,22 @@ "01dkqv0rsjqyw4wrp6yj8h3bcnl7c678qkj845596vs7p4bqff4a")))) (build-system python-build-system))) +(define python-charset-normalizer-2.10 + (package + (inherit python-charset-normalizer) + (version "2.1.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "charset-normalizer" version)) + (sha256 + (base32 "04zlajr77f6c7ai59l46as1idi0jjgbvj72lh4v5wfpz2s070pjp")))) + (build-system python-build-system) + (arguments (list)) + (native-inputs + (modify-inputs (package-native-inputs python-charset-normalizer) + (delete "python-setuptools"))))) + (define python-requests-2.28 (package (inherit python-requests) @@ -52,13 +83,78 @@ (native-inputs (list)) (propagated-inputs (modify-inputs (package-propagated-inputs python-requests) + (replace "python-charset-normalizer" python-charset-normalizer-2.10) (replace "python-urllib3" python-urllib3-1.26))))) +(define-public openconnect-sso + (package + (name "openconnect-sso") + ;; 0.8.0 was released in 2021, the latest update on master HEAD is from + ;; 2023. + (properties '((commit . "94128073ef49acb3bad84a2ae19fdef926ab7bdf") + (revision . "0"))) + (version (git-version "0.8.0" + (assoc-ref properties 'revision) + (assoc-ref properties 'commit))) + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://github.com/vlaci/openconnect-sso") + (commit (assoc-ref properties 'commit)))) + (file-name (git-file-name name version)) + (sha256 + (base32 "08cqd40p9vld1liyl6qrsdrilzc709scyfghfzmmja3m1m7nym94")))) + (build-system pyproject-build-system) + (arguments + `(#:phases + (modify-phases %standard-phases + (add-after 'unpack 'use-poetry-core + (lambda _ + ;; Patch to use the core poetry API. + (substitute* "pyproject.toml" + (("poetry.masonry.api") + "poetry.core.masonry.api")))) + (add-after 'unpack 'patch-openconnect + (lambda* (#:key inputs #:allow-other-keys) + (substitute* "openconnect_sso/app.py" + (("\"openconnect\"") + (string-append "\"" + (search-input-file inputs "/sbin/openconnect") + "\"")))))))) + (inputs + (list openconnect + python-attrs + python-colorama + python-keyring + python-lxml-4.9 + python-prompt-toolkit + python-pyotp + python-pyqt-6 + python-pyqtwebengine-6 + python-pysocks + python-pyxdg + python-requests + python-structlog + python-toml)) + (native-inputs + (list python-poetry-core + python-pytest + python-pytest-asyncio + python-pytest-httpserver)) + (home-page "https://github.com/vlaci/openconnect-sso") + (synopsis "OpenConnect wrapper script supporting Azure AD (SAMLv2)") + (description + "This package provides a wrapper script for OpenConnect supporting Azure AD +(SAMLv2) authentication to Cisco SSL-VPNs.") + (license license:gpl3))) + ;; Login to the UTHSC VPN fails with an SSLV3_ALERT_HANDSHAKE_FAILURE ;; on newer python-requests. (define openconnect-sso-uthsc (package (inherit openconnect-sso) + (name "openconnect-sso-uthsc") (inputs (modify-inputs (package-inputs openconnect-sso) (replace "python-requests" python-requests-2.28))))) @@ -81,7 +177,7 @@ (setenv "REQUESTS_CA_BUNDLE" #$(local-file "uthsc-certificate.pem")) (invoke #$(file-append openconnect-sso-uthsc "/bin/openconnect-sso") - "--server" "uthscvpn1.uthsc.edu" + "--server" "$vpn-server" ; ask us for end-point or see UT docs "--authgroup" "UTHSC" "--" "--script" (string-join (cons #$(file-append vpn-slice "/bin/vpn-slice") |