1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
|
# Restore backup
## Borg
We use borg backups. Load a recent borg, for example from my profile
```
source /home/wrk/opt/borg/etc/profile
```
The backup passphrase needs to be set.
```
source /home/ibackup/.borg-pass
```
Now run against a repo on 'tux02'
```
borg list /export3/backup/tux01/borg/borg-tux01/
borg-backup-mariadb-20221004-03:13-Tue Mon, 2022-10-03 22:13:11
```
The last backup on 'tux02' is from October 2022 - after I did a reinstall. That is not great.
According to sheepdog the drops are happening to 'space' and 'epysode', but 'tux02' is missing:
=> https://rabbit.genenetwork.org/sheepdog/index.html
## Mariadb
The first step is to find where the current mariadb is running. This will be a symlink in /var/lib/mysql -> some dir.
```
mysql -> /export2/mysql/borg-backup-mariadb-20211024-03:09-Sun
```
We typically run the database on an nvme partition. Check if there is enough space(!). It may be you need to remove the old database after making a backup (if it is not in borg).
Next, find the backup. Typically it is symlinked from /export/backup. On
```
tux02:/export5/backup/bacchus/drop/tux01
```
### Unpack from backup
We need a place to unpack the DB. On tux02 export5 is large, so let's try that and pick the last backup
```
cd /export5/mysql
tux02:/export5/mysql# borg list /export5/backup/bacchus/drop/tux01/borg-tux01
borg extract --progress /export5/backup/bacchus/drop/tux01/borg-tux01::borg-backup-mariadb-20230207-02:31-Tue
```
As it happened, this was an incomplete backup! Check original size with
```
borg info --last 3 /export5/backup/bacchus/drop/tux01/borg-tux01
```
so I went with
```
borg extract --progress /export5/backup/bacchus/drop/tux01/borg-tux01::borg-backup-mariadb-20230205-03:38-Sun
```
Mariadb's own ibdata1 is still at 100Gb. Too large, and we'll muck it out at some point. The total mariadb database at this point is 430Gb.
### Move DB in place
To move the new DB in place we first have to stop mariadb, move the old database out of the way, move the new one in, and restart the DB after making sure permissions and symlink are right.
Just to be sure we copy the old DB somewhere.
```
systemctl stop mariadb
cd /export2/mysql
cp -vau /export2/mysql/borg-backup-mariadb-20211024-03:09-Sun .
root@tux02:/export2/mysql# cp -vau /export5/mysql/home/backup/tux01_mariadb_new/latest/* borg-backup-mariadb-20230205-03:38-Sun
cd /var/lib
rm mysql
ln -s /export2/mysql/borg-backup-mariadb-20230205-03\:38-Sun mysql
systemctl start mariadb
systemctl status mariadb
```
Oops. Can't create test file /var/lib/mysql. We need to set permissions.
```
root@tux02:/var/lib# chown mysql.mysql mysql
```
And we are running.
Next stop update genotype files.
## Files
GN has a file repo for genotype files, lmdb and such, next to MariaDB. Typically
```
root@tux02:/home/gn2/production/genotype_files# du -sh .
38G .
```
the backup in drop lists, for example
```
root@tux02:/export5/backup/bacchus/drop/tux01# borg list borg-genenetwork/
borg-GN2-home-20230207-04:00-Tue Mon, 2023-02-06 22:00:02
borg-GN1-home-20230207-04:40-Tue Mon, 2023-02-06 22:40:01
borg-GN3-home-20230207-04:41-Tue Mon, 2023-02-06 22:41:27
borg-GN2s-home-20230207-04:41-Tue Mon, 2023-02-06 22:41:45
borg-ZACH-home-20230207-04:42-Tue Mon, 2023-02-06 22:42:01
borg-datafiles-home-20230207-04:42-Tue Mon, 2023-02-06 22:42:32
```
At this poing ZACH-home is the main dir, so this fetches all files
```
borg extract --verbose borg-genenetwork::borg-ZACH-home-20230207-04:42-Tue home/zas1024/gn2-zach/genotype_files
```
and we copy it to /home/gn2/production.
We also have a separate data files 'warehouse', but that is not linked up with the webserver right now.
|
