diff options
Diffstat (limited to 'topics/systems/tux02-production.gmi')
| -rw-r--r-- | topics/systems/tux02-production.gmi | 65 |
1 files changed, 0 insertions, 65 deletions
diff --git a/topics/systems/tux02-production.gmi b/topics/systems/tux02-production.gmi deleted file mode 100644 index 0b698ab..0000000 --- a/topics/systems/tux02-production.gmi +++ /dev/null @@ -1,65 +0,0 @@ -# Tux02 Production - -We are going to move production to tux02 - tux01 will be the staging machine. This machine is aimed to be rock solid. The idea is to have 4-6 times a year upgrades. Also we should be able to roll back on an upgrade and be able to create time machines. - -## Tags - -* assigned: pjotrp -* status: in progress -* priority: medium -* type: system administration -* keywords: systems, tux02, production - -## Tasks - -* [X] update guix guix-1.3.0-9.f743f20 -* [X] set up nginx (Debian) -* [X] test ipmi console (172.23.30.40) -* [X] test ports (nginx) -* [?] set up network for external tux02e.uthsc.edu (128.169.4.52) -* [X] set up deployment evironment -* [X] sheepdog copy database backup from tux01 on a daily basis using ibackup user -* [X] same for GN2 production environment -* [X] sheepdog borg borg the backups -* [X] start GN2 production services -* [X] add GN3 aliases server -* [X] add Genenetwork3 service - env FLASK_APP="main.py" flask run --port=8087 -* [X] add proxy -* [ ] set up databases -* [ ] set up https and letsencrypt -* [X] set up firewalling -* [ ] set up systemd -* [ ] setup logrotate for production log files -* [ ] run git automatically on /etc and backup without passwords -* [ ] add borg backups -* [ ] create check list for manual testing -* [ ] look at performance - -## Info - -We have a protocol for updating GN2 on Tux02. - -### Restore database from backup - -Databases no longer get copied. We only restore from backup. First because these are reproducible [installs]. Second because the backup should be in a sane state(!). - -Restoring a database from backup (about an hour) - -``` -root@tux02:/export3/backup/tux01/borg# borg extract borg-tux01::borg-backup-mariadb-20211024-03:09-Sun --progress -``` - -Next move the dir to fast storage. - -#### Symlink /var/lib/mysql - -The database is symlinked. You can point that to the recovered backup. Restart the DB and run mysql_upgrade followed by our tests. E.g. - -``` -systemctl stop mariadb -ln -s /export2/mysql/borg-backup-mariadb-20211024-03\:09-Sun /var/lib/mysql -systemctl start mariadb -/usr/local/guix-profiles/gn-latest-20211021/bin/mysql_upgrade -u webqtlout -pwebqtlout -/export/backup/scripts/tux02/system_check.sh -``` |