# Backup Drops

To make backups we use a combination of sheepdog, borg, sshfs, rsync. sheepdog is the monitor and status is tracked using a redis queue. borg does incremental backups as a local user. Next we drop the backup to a remote machine using a special user on the remote with very limited access - only one directory can be mounted with sshfs and there is no ssh login. Finally the data gets rsync'd across.

This system proves pretty resilient over time. Only on the synology server I can't get it to work because of some CRON permission issue.

# Tags

* assigned: pjotrp
* keywords: systems, backup, sheepdog, database

# Info

## Borg backups

It is advised to use a backup password and not store that on the remote.

## Running sheepdog on rabbit

=> https://github.com/pjotrp/deploy sheepdog monitor running on redis queue
=> https://rabbit.genenetwork.org/sheepdog/index.html current web monitor for GN sheepdog on rabbit

## Create a drop user on remote

```
adduser bacchus
```

Disable the password, just to be sure. Next add the following to /etc/ssh/sshd.conf:

```diff
-# override default of no subsystems
-Subsystem      sftp    /usr/lib/openssh/sftp-server
+Subsystem      sftp    internal-sftp -f AUTH -l VERBOSE
+
+Match User bacchus
+    # ChrootDirectory /export/backup/%u
+    ChrootDirectory /export/backup/%u
+    X11Forwarding no
+    AllowTcpForwarding no
+    ForceCommand internal-sftp
+    PasswordAuthentication no
+    PubkeyAuthentication yes
```

And run

```
systemctl reload ssh
```

You may need to add to allowusers for ssh access. If you use allowusers (recommended) you can even specify the sending host with

```
AllowUsers bacchus@remote
```

where remote can be an IP address.

Warning: if you introduce this `AllowUsers` command all users should be listed or people may get locked out of the machine.

Next create a special key on the backup machine's ibackup user (just hit enter):

```
su ibackup
ssh-keygen -t ecdsa -f $HOME/.ssh/id_ecdsa_backup
```

and copy the public key into the remote /home/bacchus/.ssh/authorized_keys

Now test it from the backup server with

```
ssh -v bacchus@dropserver
```

it should give a Permission denied (publickey).

On the drop server you can track messages by

```
n0tail -40 /var/log/auth.log
```

Next

```
ssh -v -i ~/.ssh/id_ecdsa_backup bacchus@dropserver
```

should give a Broken pipe(!). In auth.log you may see something like

fatal: bad ownership or modes for chroot directory component "/export/backup/"

This is a tricky bit. This directory should be owned by root and have permissions. The inside user directory has different permissions:

```
ls -ld /export/backup/
drwxr-xr-x 3 root root 4096 Oct 21 02:08 /export/backup/
drwxr-xr-x 3 root root 4096 Oct 21 02:07 /export/backup/bacchus
drwx------ 3 bacchus bacchus 4096 Oct 21 02:26 /export/backup/bacchus/drop
```

So, as root

```
cd /export
mkdir -p backup/bacchus/drop
chown bacchus.bacchus backup/bacchus/drop/
chmod 0700 backup/bacchus/drop/
```

If auth.log says error: /dev/pts/11: No such file or directory on ssh, or received disconnect (...) disconnected by user we are good to go!

Next try sshfs

```
su ibackup
mkdir -p ~/mnt/dropserver
sshfs -o IdentityFile=~/.ssh/id_ecdsa_backup bacchus@dropserver:/ ~/mnt/dropserver/
df -h|grep mnt
touch ~/mnt/dropserver/drop/HELLO
```

And the remote directory should be ready for dropping files!

To unmount the dir

```
fusermount -u ~/mnt/dropserver
```

IMPORTANT: it is important to try ssh and read /var/log/auth.log to deal with permission issues. sshfs and the underlying sftp protocol are fussy.

## Using rsync over sshfs with sheepdog

A backup script with sheepdog may look like

```
sheepdog_run.rb -h rabbit --always -v --tag "drop-mount-dropserver" -c "sshfs -o IdentityFile=~/.ssh/id_ecdsa_backup bacchus@dropserver:/ ~/mnt/this"

sheepdog_run.rb -h rabbit --always -v --tag "drop-rsync-dropserver" -c "rsync -vrltDP this/* borg/* ~/mnt/this/drop/this/ --delete"

sheepdog_run.rb -h rabbit --always -v --tag "drop-unmount-dropserver" -c "fusermount -u ~/mnt/this"
```

It may be useful to add the following options to sshfs:

```
sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,IdentityFile=~/.ssh/id_ecdsa_backup ...
```

The recent scripts can be found at

=> https://github.com/genenetwork/gn-deploy-servers/blob/master/scripts/tux01/backup_drop.sh