# Update production checklist

Last migration round is the move to tux03 (Sept2025)!

# Tasks

* [X] Install underlying Debian
* [X] Get guix going
* [X] Check database settings
* [X] Check gemma working
* [X] Check global search
* [X] Check authentication
* [X] Check sending E-mails
* [X] Check SPARQL
* [X] Make sure info.genenetwork.org and 'space' can reach the DB
* [ ] Backups

The following are at the system level

* [X] Firewalling and other security measures (sshd)
* [X] Check tmpdirs (cleanup?)
* [X] Make sure journalctl persistent (check for reboots)
* [X] Update certificates in CRON (no longer if not part of Guix)
* [X] Run trim in CRON
* [ ] Monitors (sheepdog)

# Install underlying Debian

For our production systems we use Debian as a base install. Once installed:

* [X] set up git in /etc and limit permissions to root user
* [X] add ttyS0 support for grub and kernel - so out-of-band works
* [X] start ssh server and configure not to use with passwords
* [X] start nginx and check external networking
* [X] mount old root
* [X] Clean up /etc/profile (remove global profile.d loading)
* [X] set up E-mail routing

It may help to mount the old root if you have it. Now it is on

```
mount /dev/sdd2 /mnt/old-root/
```

# Get Guix going

* [X] Mount bind /gnu on a large partition
* [X] Move /gnu/store to larger partition
* [X] Install Guix daemon
* [X] Update Guix daemon and setup in systemd (if necessary)
* [X] Make available in /usr/local/guix-profiles

Next move the /gnu store to a large partion and hard mount it in /etc/fstab with

```
/export2/gnu /gnu none defaults,bind 0 0
```

We can bootstrap with the Debian guix package (though I prefer the guix-install.sh script these days, mostly because it is more modern).

=> https://guix.gnu.org/manual/en/html_node/Binary-Installation.html


Run guix pull

```
guix pull --url=https://codeberg.org/guix/guix  -p ~/opt/guix-pull
```

Use that also to install guix in /usr/local/guix-profiles

```
guix package -i guix -p /usr/local/guix-profiles/guix
```

and update the daemon in systemd accordingly. After that I tend to remove /usr/bin/guix

The Debian installer configures guix. I tend to remove the profiles from /etc/profile so people have a minimal profile.

# Check database

* [X] Install mariadb
* [X] Recover database
* [X] Test permissions
* [X] Mariadb update my.cnf

Basically recover the database from a backup is the best start and set permissions. We usually take the default mariadb unless production is already on a newer version - so we move to guix deployment.

On tux02 mariadb-10.5.8 is running. On Debian it is now 10.11.11-0+deb12u1, so we should be good. On Guix is 10.10 at this point.

```
apt-get install mariadb-server
```

Next unpack the database files and set permissions to the mysql user. And (don't forget) update the /etc/mysql config files.

Restart mysql until you see:

```
mysql -u webqtlout -p -e "show databases"
+---------------------------+
| Database                  |
+---------------------------+
| 20081110_uthsc_dbdownload |
| db_GeneOntology           |
| db_webqtl                 |
| db_webqtl_s               |
| go                        |
| information_schema        |
| kegg                      |
| mysql                     |
| performance_schema        |
| sys                       |
+---------------------------+
```

=> topics/systems/mariadb/mariadb.gmi

## Recover database

We use borg for backups. First restore the backup on the PCIe. Also a test for overheating!


# Check sending E-mails

The swaks package is quite useful to test for a valid receive host:

```
swaks --to testing-my-server@gmail.com --server smtp.network
=== Trying smtp.network:25...
=== Connected to smtp.network.
<-  220 mailrouter8.network ESMTP NO UCE
 -> EHLO tux04.network
<-  250-mailrouter8.network
<-  250-PIPELINING
<-  250-SIZE 26214400
<-  250-VRFY
<-  250-ETRN
<-  250-STARTTLS
<-  250-ENHANCEDSTATUSCODES
<-  250-8BITMIME
<-  250-DSN
<-  250 SMTPUTF8
 -> MAIL FROM:<root@tux04.network>
<-  250 2.1.0 Ok
 -> RCPT TO:<pjotr2020@thebird.nl>
<-  250 2.1.5 Ok
 -> DATA
<-  354 End data with <CR><LF>.<CR><LF>
 -> Date: Thu, 06 Mar 2025 08:34:24 +0000
 -> To: pjotr2020@thebird.nl
 -> From: root@tux04.network
 -> Subject: test Thu, 06 Mar 2025 08:34:24 +0000
 -> Message-Id: <20250306083424.624509@tux04.network>
 -> X-Mailer: swaks v20201014.0 jetmore.org/john/code/swaks/
 ->
 -> This is a test mailing
 ->
 ->
 -> .
<-  250 2.0.0 Ok: queued as 4157929DD
 -> QUIT
<-  221 2.0.0 Bye                                                                                                                             === Connection closed with remote host
```

An exim configuration can be

```
dc_eximconfig_configtype='smarthost'
dc_other_hostnames='genenetwork.org'
dc_local_interfaces='127.0.0.1 ; ::1'
dc_readhost=''
dc_relay_domains=''
dc_minimaldns='false'
dc_relay_nets=''
dc_smarthost='smtp.network'
CFILEMODE='644'
dc_use_split_config='false'
dc_hide_mailname='false'
dc_mailname_in_oh='true'
dc_localdelivery='maildir_home'
```

And this should work:

```
swaks --to myemailaddress --from john@network --server localhost
```

# Backups

* [ ] Create an ibackup user.
* [ ] Install borg (usually guix version)
* [ ] Create a borg passphrase

=> topics/systems/backups-with-borg.gmi
=> topics/systems/backup-drops.gmi