diff options
| -rw-r--r-- | issues/systems/fallbacks-and-backups.gmi | 4 | ||||
| -rw-r--r-- | topics/systems/backup-drops.gmi | 18 | ||||
| -rw-r--r-- | topics/systems/restore-backups.gmi | 37 |
3 files changed, 53 insertions, 6 deletions
diff --git a/issues/systems/fallbacks-and-backups.gmi b/issues/systems/fallbacks-and-backups.gmi index 1a22db9..ccb36f3 100644 --- a/issues/systems/fallbacks-and-backups.gmi +++ b/issues/systems/fallbacks-and-backups.gmi @@ -2,6 +2,10 @@ As a hurricane is barreling towards our machine room in Memphis we are checking our fallbacks and backups for GeneNetwork. For years we have been making backups on Amazon - both S3 and a running virtual machine. The latter was expensive, so I replaced it with a bare metal server which earns itself (if it hadn't been down for months, but that is a different story). +See also + +=> restore-backups.gmi + ## Tags * type: enhancement diff --git a/topics/systems/backup-drops.gmi b/topics/systems/backup-drops.gmi index 98a2381..930ee45 100644 --- a/topics/systems/backup-drops.gmi +++ b/topics/systems/backup-drops.gmi @@ -79,7 +79,7 @@ it should give a Permission denied (publickey). On the drop server you can track messages by ``` -tail -40 /var/log/auth.log +n0tail -40 /var/log/auth.log ``` Next @@ -104,13 +104,15 @@ drwx------ 3 bacchus bacchus 4096 Oct 21 02:26 /export/backup/bacchus/drop So, as root ``` -mkdir -p /export/backup/bacchus/drop -chown bacchus.bacchus /export/backup/bacchus/drop/ -chmod 0700 /export/backup/bacchus/drop/ +cd /export +mkdir -p backup/bacchus/drop +chown bacchus.bacchus backup/bacchus/drop/ +chmod 0700 backup/bacchus/drop/ ``` -If auth.log says error: /dev/pts/11: No such file or directory on ssh we are good to go! -Next use sshfs +If auth.log says error: /dev/pts/11: No such file or directory on ssh, or received disconnect (...) disconnected by user we are good to go! + +Next try sshfs ``` su ibackup @@ -147,3 +149,7 @@ It may be useful to add the following options to sshfs: ``` sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,IdentityFile=~/.ssh/id_ecdsa_backup ... ``` + +The recent scripts can be found at + +=> https://github.com/genenetwork/gn-deploy-servers/blob/master/scripts/tux01/backup_drop.sh diff --git a/topics/systems/restore-backups.gmi b/topics/systems/restore-backups.gmi new file mode 100644 index 0000000..b9d4596 --- /dev/null +++ b/topics/systems/restore-backups.gmi @@ -0,0 +1,37 @@ +# Restore backup + +## Borg + +We use borg backups. Load a recent borg, for example from my profile + +``` +source /home/wrk/opt/borg/etc/profile +``` + + +The backup passphrase needs to be set. + +``` +source /home/ibackup/.borg-pass +``` + +Now run against a repo on 'tux02' + +``` +borg list /export3/backup/tux01/borg/borg-tux01/ +borg-backup-mariadb-20221004-03:13-Tue Mon, 2022-10-03 22:13:11 +``` + +The last backup on 'tux02' is from October 2022 - after I did a reinstall. That is not great. + +According to sheepdog the drops are happening to 'space' and 'epysode', but 'tux02' is missing: + +=> https://rabbit.genenetwork.org/sheepdog/index.html + +## Mariadb + +The first step is to find where the current mariadb is running. This will be a symlink in /var/lib/mysgl -> some dir. + +We typically run the database on an nvme partition. Check if there is enough space(!). It may be you need to remove the old database after making a backup (if it is not in borg). + +Next, find the backup. Typically it is in /export/backup |