blob: 7de911ff28db53b0dbc3c83a9977700b67d61987 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
|
# Tux02 Production
We are going to move production to tux02 - tux01 will be the staging machine. This machine is aimed to be rock solid. The idea is to have 4-6 times a year upgrades. Also we should be able to roll back on an upgrade and be able to create time machines.
## Tags
* assigned: pjotrp
* status: in progress
* priority: medium
* type: system administration
* keywords: systems, tux02, production
## Tasks
* [X] update guix guix-1.3.0-9.f743f20
* [X] set up nginx (Debian)
* [X] test ipmi console (172.23.30.40)
* [X] test ports (nginx)
* [?] set up network for external tux02e.uthsc.edu (128.169.4.52)
* [X] set up deployment evironment
* [X] sheepdog copy database backup from tux01 on a daily basis using ibackup user
* [X] same for GN2 production environment
* [X] sheepdog borg borg the backups
* [X] start GN2 production services
* [X] add GN3 aliases server
* [X] add Genenetwork3 service
env FLASK_APP="main.py" flask run --port=8087
* [X] add proxy
* [X] set up databases
* [X] set up https and letsencrypt
* [X] set up firewalling
* [X] set up systemd
* [ ] setup logrotate for production log files
* [ ] run git automatically on /etc and backup without passwords
* [ ] add borg backups
* [ ] create check list for manual testing
* [ ] look at performance
## Info
We have a protocol for updating GN2 on Tux02.
### Restore database from backup
Databases no longer get copied. We only restore from backup. First because these are reproducible [installs]. Second because the backup should be in a sane state(!).
Restoring a database from backup (about an hour)
```
root@tux02:/export3/backup/tux01/borg# borg extract borg-tux01::borg-backup-mariadb-20211024-03:09-Sun --progress
```
Next move the dir to fast storage.
#### Symlink /var/lib/mysql
The database is symlinked. You can point that to the recovered backup. Restart the DB and run mysql_upgrade followed by our tests. E.g.
```
systemctl stop mariadb
ln -s /export2/mysql/borg-backup-mariadb-20211024-03\:09-Sun /var/lib/mysql
systemctl start mariadb
/usr/local/guix-profiles/gn-latest-20211021/bin/mysql_upgrade -u webqtlout -pwebqtlout
/export/backup/scripts/tux02/system_check.sh
```
|
