- Added task

- Add documentation on creating a backup drop service using sftp/sshfs

2 files changed, 117 insertions, 0 deletions

diff --git a/tasks/pjotrp.gmi b/tasks/pjotrp.gmi
index 68efd93..1f12a0c 100644
--- a/tasks/pjotrp.gmi
+++ b/tasks/pjotrp.gmi
@@ -30,6 +30,7 @@ The tasks here should probably be broken out into appropriately tagged issues, w
 
 => ./dana.gmi See Dan's list for machine room
 
+* [ ] research.gov submit Postdoc plan
 * [+] run sheepdog as root: redis password error; introduce SHEEPDOG_CONF
 * [ ] Complete vcflib work in Zig and release
 * [+] Get status list from sheepdog on epysode, tux02 and rabbit on GN2
diff --git a/topics/systems/backup-drops.gmi b/topics/systems/backup-drops.gmi
new file mode 100644
index 0000000..84c1612
--- /dev/null
+++ b/topics/systems/backup-drops.gmi
@@ -0,0 +1,116 @@
+# Backup Drops
+
+To make backups we use a combination of sheepdog, borg, sshfs, rsync. sheepdog is the monitor and status is tracked using a redis queue. borg does incremental backups as a local user. Next we drop the backup to a remote machine using a special user on the remote with very limited access - only one directory can be mounted with sshfs and there is no ssh login. Finally the data gets rsync'd across.
+
+This system proves pretty resilient over time. Only on the synology server I can't get it to work because of some CRON permission issue.
+
+# Tags
+
+* assigned: pjotrp
+* keywords: systems, backup, sheepdog, database
+
+# Info
+
+## Borg backups
+
+
+## Running sheepdog on rabbit
+
+=> https://github.com/pjotrp/deploy sheepdog monitor running on redis queue
+=> https://rabbit.genenetwork.org/sheepdog/index.html current web monitor for GN sheepdog on rabbit
+
+## Create a drop user on remote
+
+```
+adduser bacchus
+```
+
+Disable the password, just to be sure. Next add the following to /etc/ssh/sshd.conf:
+
+```diff
+-# override default of no subsystems
+-Subsystem      sftp    /usr/lib/openssh/sftp-server
++Subsystem      sftp    internal-sftp -f AUTH -l VERBOSE
++
++Match User bacchus
++    # ChrootDirectory /export/backup/%u
++    ChrootDirectory /export/backup/%u
++    X11Forwarding no
++    AllowTcpForwarding no
++    ForceCommand internal-sftp
++    PasswordAuthentication no
++    PubkeyAuthentication yes
+```
+
+And run
+
+```
+systemctl reload ssh
+```
+
+You may need to add to allowusers for ssh access.
+
+Next create a special key on the backup machine's ibackup user (just hit enter):
+
+```
+su ibackup
+ssh-keygen -t ecdsa -f $HOME/.ssh/id_ecdsa_backup
+```
+
+and copy the public key into the remote /home/bacchus/.ssh/authorized_keys
+
+Now test it from the backup server with
+
+```
+ssh -v bacchus@dropserver
+```
+
+it should give a Permission denied (publickey).
+
+On the drop server you can track messages by
+
+```
+tail -40 /var/log/auth.log
+```
+
+Next
+
+```
+ssh -v -i ~/.ssh/id_ecdsa_backup bacchus@dropserver
+```
+
+should give a Broken pipe(!). In auth.log you may see something like
+
+fatal: bad ownership or modes for chroot directory component "/export/backup/"
+
+This is a tricky bit. This directory should be owned by root and have permissions. The inside user directory has different permissions:
+
+```
+ls -ld /export/backup/
+drwxr-xr-x 3 root root 4096 Oct 21 02:08 /export/backup/
+drwxr-xr-x 3 root root 4096 Oct 21 02:07 /export/backup/bacchus
+drwx------ 3 bacchus bacchus 4096 Oct 21 02:26 /export/backup/bacchus/drop
+```
+
+So, as root
+
+```
+mkdir -p /export/backup/bacchus/drop
+chown bacchus.bacchus /export/backup/bacchus/drop/
+chmod 0700 /export/backup/bacchus/drop/
+```
+
+If auth.log says error: /dev/pts/11: No such file or directory on ssh we are good to go!
+Next use sshfs
+
+```
+su ibackup
+mkdir -p ~/mnt/dropserver
+sshfs -o IdentityFile=~/.ssh/id_ecdsa_backup bacchus@dropserver:/ ~/mnt/dropserver/
+df -h|grep mnt
+touch ~/mnt/dropserver/drop/HELLO
+```
+
+And the remote directory should be ready for dropping files!
+
+IMPORTANT: it is important to try ssh and read /var/log/auth.log to deal with permission issues. sshfs and the underlying sftp protocol are fussy.