Move issues lost inside the topics directory.

1 files changed, 65 insertions, 0 deletions

diff --git a/issues/systems/tux02-production.gmi b/issues/systems/tux02-production.gmi
new file mode 100644
index 0000000..0b698ab
--- /dev/null
+++ b/issues/systems/tux02-production.gmi
@@ -0,0 +1,65 @@
+# Tux02 Production
+
+We are going to move production to tux02 - tux01 will be the staging machine. This machine is aimed to be rock solid. The idea is to have 4-6 times a year upgrades. Also we should be able to roll back on an upgrade and be able to create time machines.
+
+## Tags
+
+* assigned: pjotrp
+* status: in progress
+* priority: medium
+* type: system administration
+* keywords: systems, tux02, production
+
+## Tasks
+
+* [X] update guix guix-1.3.0-9.f743f20
+* [X] set up nginx (Debian)
+* [X] test ipmi console (172.23.30.40)
+* [X] test ports (nginx)
+* [?] set up network for external tux02e.uthsc.edu (128.169.4.52)
+* [X] set up deployment evironment
+* [X] sheepdog copy database backup from tux01 on a daily basis using ibackup user
+* [X] same for GN2 production environment
+* [X] sheepdog borg borg the backups
+* [X] start GN2 production services
+* [X] add GN3 aliases server
+* [X] add Genenetwork3 service
+      env FLASK_APP="main.py" flask run --port=8087
+* [X] add proxy
+* [ ] set up databases
+* [ ] set up https and letsencrypt
+* [X] set up firewalling
+* [ ] set up systemd
+* [ ] setup logrotate for production log files
+* [ ] run git automatically on /etc and backup without passwords
+* [ ] add borg backups
+* [ ] create check list for manual testing
+* [ ] look at performance
+
+## Info
+
+We have a protocol for updating GN2 on Tux02.
+
+### Restore database from backup
+
+Databases no longer get copied. We only restore from backup. First because these are reproducible [installs]. Second because the backup should be in a sane state(!).
+
+Restoring a database from backup (about an hour)
+
+```
+root@tux02:/export3/backup/tux01/borg# borg extract borg-tux01::borg-backup-mariadb-20211024-03:09-Sun --progress
+```
+
+Next move the dir to fast storage.
+
+#### Symlink /var/lib/mysql
+
+The database is symlinked. You can point that to the recovered backup. Restart the DB and run mysql_upgrade followed by our tests. E.g.
+
+```
+systemctl stop mariadb
+ln -s /export2/mysql/borg-backup-mariadb-20211024-03\:09-Sun /var/lib/mysql
+systemctl start mariadb
+/usr/local/guix-profiles/gn-latest-20211021/bin/mysql_upgrade -u webqtlout -pwebqtlout
+/export/backup/scripts/tux02/system_check.sh
+```