From 0db8a79857d18314fedcdfe045594e9d4f940501 Mon Sep 17 00:00:00 2001
From: Pjotr Prins
Date: Tue, 7 Feb 2023 13:03:36 +0100
Subject: On backups and drop services

---
 issues/systems/fallbacks-and-backups.gmi |  4 ++++
 topics/systems/backup-drops.gmi          | 18 ++++++++++------
 topics/systems/restore-backups.gmi       | 37 ++++++++++++++++++++++++++++++++
 3 files changed, 53 insertions(+), 6 deletions(-)
 create mode 100644 topics/systems/restore-backups.gmi

diff --git a/issues/systems/fallbacks-and-backups.gmi b/issues/systems/fallbacks-and-backups.gmi
index 1a22db9..ccb36f3 100644
--- a/issues/systems/fallbacks-and-backups.gmi
+++ b/issues/systems/fallbacks-and-backups.gmi
@@ -2,6 +2,10 @@
 
 As a hurricane is barreling towards our machine room in Memphis we are checking our fallbacks and backups for GeneNetwork. For years we have been making backups on Amazon - both S3 and a running virtual machine. The latter was expensive, so I replaced it with a bare metal server which earns itself (if it hadn't been down for months, but that is a different story).
 
+See also
+
+=> restore-backups.gmi
+
 ## Tags
 
 * type: enhancement
diff --git a/topics/systems/backup-drops.gmi b/topics/systems/backup-drops.gmi
index 98a2381..930ee45 100644
--- a/topics/systems/backup-drops.gmi
+++ b/topics/systems/backup-drops.gmi
@@ -79,7 +79,7 @@ it should give a Permission denied (publickey).
 On the drop server you can track messages by
 
 ```
-tail -40 /var/log/auth.log
+n0tail -40 /var/log/auth.log
 ```
 
 Next
@@ -104,13 +104,15 @@ drwx------ 3 bacchus bacchus 4096 Oct 21 02:26 /export/backup/bacchus/drop
 So, as root
 
 ```
-mkdir -p /export/backup/bacchus/drop
-chown bacchus.bacchus /export/backup/bacchus/drop/
-chmod 0700 /export/backup/bacchus/drop/
+cd /export
+mkdir -p backup/bacchus/drop
+chown bacchus.bacchus backup/bacchus/drop/
+chmod 0700 backup/bacchus/drop/
 ```
 
-If auth.log says error: /dev/pts/11: No such file or directory on ssh we are good to go!
-Next use sshfs
+If auth.log says error: /dev/pts/11: No such file or directory on ssh, or received disconnect (...) disconnected by user we are good to go!
+
+Next try sshfs
 
 ```
 su ibackup
@@ -147,3 +149,7 @@ It may be useful to add the following options to sshfs:
 ```
 sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,IdentityFile=~/.ssh/id_ecdsa_backup ...
 ```
+
+The recent scripts can be found at
+
+=> https://github.com/genenetwork/gn-deploy-servers/blob/master/scripts/tux01/backup_drop.sh
diff --git a/topics/systems/restore-backups.gmi b/topics/systems/restore-backups.gmi
new file mode 100644
index 0000000..b9d4596
--- /dev/null
+++ b/topics/systems/restore-backups.gmi
@@ -0,0 +1,37 @@
+# Restore backup
+
+## Borg
+
+We use borg backups. Load a recent borg, for example from my profile
+
+```
+source /home/wrk/opt/borg/etc/profile
+```
+
+
+The backup passphrase needs to be set.
+
+```
+source /home/ibackup/.borg-pass
+```
+
+Now run against a repo on 'tux02'
+
+```
+borg list /export3/backup/tux01/borg/borg-tux01/
+borg-backup-mariadb-20221004-03:13-Tue Mon, 2022-10-03 22:13:11
+```
+
+The last backup on 'tux02' is from October 2022 - after I did a reinstall. That is not great.
+
+According to sheepdog the drops are happening to 'space' and 'epysode', but 'tux02' is missing:
+
+=> https://rabbit.genenetwork.org/sheepdog/index.html
+
+## Mariadb
+
+The first step is to find where the current mariadb is running. This will be a symlink in /var/lib/mysgl -> some dir.
+
+We typically run the database on an nvme partition. Check if there is enough space(!). It may be you need to remove the old database after making a backup (if it is not in borg).
+
+Next, find the backup. Typically it is in /export/backup
-- 
cgit v1.2.3