6 files changed, 654 insertions, 7 deletions

diff --git a/issues/systems/apps.gmi b/issues/systems/apps.gmi
index 51c9d24..b9d4155 100644
--- a/issues/systems/apps.gmi
+++ b/issues/systems/apps.gmi
@@ -153,7 +153,7 @@ downloading from http://cran.r-project.org/src/contrib/Archive/KernSmooth/KernSm
 - 'configure' phasesha256 hash mismatch for /gnu/store/n05zjfhxl0iqx1jbw8i6vv1174zkj7ja-KernSmooth_2.23-17.tar.gz:
   expected hash: 11g6b0q67vasxag6v9m4px33qqxpmnx47c73yv1dninv2pz76g9b
   actual hash:   1ciaycyp79l5aj78gpmwsyx164zi5jc60mh84vxxzq4j7vlcdb5p
-hash mismatch for store item '/gnu/store/n05zjfhxl0iqx1jbw8i6vv1174zkj7ja-KernSmooth_2.23-17.tar.gz'
+  hash mismatch for store item '/gnu/store/n05zjfhxl0iqx1jbw8i6vv1174zkj7ja-KernSmooth_2.23-17.tar.gz'
 ```
 
 Guix checks and it is not great CRAN allows for changing tarballs with the same version number!! Luckily building with a more recent version of Guix just worked (TM). Now we create a root too:
@@ -184,12 +184,24 @@ and it looks like lines like these need to be updated:
 
 => https://github.com/genenetwork/singleCellRshiny/blob/6b2a344dd0d02f65228ad8c350bac0ced5850d05/app.R#L167
 
-Let me ask the author Siamak Yousefi.
+Let me ask the author Siamak Yousefi. I think we'll drop it.
+
+## longevity
+
+Package definition is at
+
+=> https://git.genenetwork.org/guix-bioinformatics/tree/gn/packages/mouse-longevity.scm
+
+Container is at
+
+=> https://git.genenetwork.org/guix-bioinformatics/tree/gn/services/bxd-power-container.scm
 
 ## jumpshiny
 
+Jumpshiny is hosted on balg01. Scripts are in tux02 git.
+
 ```
-balg01:~/gn-machines$ guix system container --network -L . -L ../guix-bioinformatics/ -L ../guix-past/modules/ --substitute-urls='https:
-//ci.guix.gnu.org https://bordeaux.guix.gnu.org https://cuirass.genenetwork.org' test-r-container.scm -L ../guix-forge/guix/
-/gnu/store/xyks73sf6pk78rvrwf45ik181v0zw8rx-run-container
+root@balg01:/home/j*/gn-machines# . /usr/local/guix-profiles/guix-pull/etc/profile
+guix system container --network -L . -L ../guix-forge/guix/ -L ../guix-bioinformatics/ -L ../guix-past/modules/ --substitute-urls='https://ci.guix.gnu.org https://bordeaux.guix.gnu.org https://cuirass.genenetwork.org' test-r-container.scm -L ../guix-forge/guix/gnu/store/xyks73sf6pk78rvrwf45ik181v0zw8rx-run-container
+/gnu/store/6y65x5jk3lxy4yckssnl32yayjx9nwl5-run-container
 ```
diff --git a/issues/systems/octoraid-storage.gmi b/issues/systems/octoraid-storage.gmi
new file mode 100644
index 0000000..97e0e55
--- /dev/null
+++ b/issues/systems/octoraid-storage.gmi
@@ -0,0 +1,18 @@
+# OctoRAID
+
+We are building machines that can handle cheap drives.
+
+# octoraid01
+
+This is a jetson with 4 22TB seagate-ironwolf-pro-st22000nt001-22tb-enterprise-nas-hard-drives-7200-rpm.
+
+Unfortunately the stock kernel has no RAID support, so we simple mount the 4 drives (hosted on a USB-SATA bridge).
+
+Stress testing:
+
+```
+cd /export/nfs/lair01
+stress -v -d 1
+```
+
+Running on multiple disks the jetson is holding up well!
diff --git a/issues/systems/penguin2-raid5.gmi b/issues/systems/penguin2-raid5.gmi
new file mode 100644
index 0000000..f03075d
--- /dev/null
+++ b/issues/systems/penguin2-raid5.gmi
@@ -0,0 +1,61 @@
+# Penguin2 RAID 5
+
+# Tags
+
+* assigned: @fredm, @pjotrp
+* status: in progress
+
+# Description
+
+The current RAID contains 3 disks:
+
+```
+root@penguin2:~# cat /proc/mdstat
+md0 : active raid5 sdb1[1] sda1[0] sdg1[4]
+/dev/md0         33T   27T  4.2T  87% /export
+```
+
+using /dev/sda,sdb,sdg
+
+The current root and swap is on
+
+```
+# root
+/dev/sdd1       393G  121G  252G  33% /
+# swap
+/dev/sdd5 partition 976M 76.5M   -2
+```
+
+We can therefore add four new disks in slots /dev/sdc,sde,sdf,sdh
+
+penguin2 has no out-of-band and no serial connector right now. That means any work needs to be done on the terminal.
+
+Boot loader menu:
+
+```
+menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-7ff268df-cb90-4cbc-9d76-7fd6677b4964' {
+        load_video
+        insmod gzio
+        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
+        insmod part_msdos
+        insmod ext2
+        set root='hd2,msdos1'
+        if [ x$feature_platform_search_hint = xy ]; then
+          search --no-floppy --fs-uuid --set=root --hint-bios=hd2,msdos1 --hint-efi=hd2,msdos1 --hint-baremetal=ahci2,msdos1  7ff268df-cb90-4cbc-9d76-7fd6677b4964
+        else
+          search --no-floppy --fs-uuid --set=root 7ff268df-cb90-4cbc-9d76-7fd6677b4964
+        fi
+        echo    'Loading Linux 5.10.0-18-amd64 ...'
+        linux   /boot/vmlinuz-5.10.0-18-amd64 root=UUID=7ff268df-cb90-4cbc-9d76-7fd6677b4964 ro  quiet
+        echo    'Loading initial ramdisk ...'
+        initrd  /boot/initrd.img-5.10.0-18-amd64
+}
+```
+
+Added to sdd MBR
+
+```
+root@penguin2:~# grub-install /dev/sdd
+Installing for i386-pc platform.
+Installation finished. No error reported.
+```
diff --git a/issues/systems/tux02-production.gmi b/issues/systems/tux02-production.gmi
index 7de911f..d811c5e 100644
--- a/issues/systems/tux02-production.gmi
+++ b/issues/systems/tux02-production.gmi
@@ -14,9 +14,9 @@ We are going to move production to tux02 - tux01 will be the staging machine. Th
 
 * [X] update guix guix-1.3.0-9.f743f20
 * [X] set up nginx (Debian)
-* [X] test ipmi console (172.23.30.40)
+* [X] test ipmi console
 * [X] test ports (nginx)
-* [?] set up network for external tux02e.uthsc.edu (128.169.4.52)
+* [?] set up network for external tux02
 * [X] set up deployment evironment
 * [X] sheepdog copy database backup from tux01 on a daily basis using ibackup user
 * [X] same for GN2 production environment
diff --git a/issues/systems/tux04-disk-issues.gmi b/issues/systems/tux04-disk-issues.gmi
index 9bba105..bc6e1db 100644
--- a/issues/systems/tux04-disk-issues.gmi
+++ b/issues/systems/tux04-disk-issues.gmi
@@ -101,3 +101,280 @@ and nothing ;). Megacli is actually the tool to use
 ```
 megacli -AdpAllInfo -aAll
 ```
+
+# Database
+
+During a backup the DB shows this error:
+
+```
+2025-03-02 06:28:33 Database page corruption detected at page 1079428, retrying...\n[01] 2025-03-02 06:29:33 Database page corruption detected at page 1103108, retrying...
+```
+
+
+Interestingly the DB recovered on a second backup.
+
+The database is hosted on a solid /dev/sde Dell Ent NVMe FI. The log says
+
+```
+kernel: I/O error, dev sde, sector 2136655448 op 0x0:(READ) flags 0x80700 phys_seg 40 prio class 2
+```
+
+Suggests:
+
+=> https://stackoverflow.com/questions/50312219/blk-update-request-i-o-error-dev-sda-sector-xxxxxxxxxxx
+
+> The errors that you see are interface errors, they are not coming from the disk itself but rather from the connection to it. It can be the cable or any of the ports in the connection.
+> Since the CRC errors on the drive do not increase I can only assume that the problem is on the receive side of the machine you use. You should check the cable and try a different SATA port on the server.
+
+and someone wrote
+
+> analyzed that most of the reasons are caused by intensive reading and writing.  This is a CDN cache node. Type reading NVME temperature is relatively high, if it continues, it will start to throttle and then slowly collapse.
+
+and temperature on that drive has been 70 C.
+
+Mariabd log is showing errors:
+
+```
+2025-03-02  6:54:47 0 [ERROR] InnoDB: Failed to read page 449925 from file './db_webqtl/SnpAll.ibd': Page read from tablespace is corrupted.
+2025-03-02  7:01:43 489015 [ERROR] Got error 180 when reading table './db_webqtl/ProbeSetXRef'
+2025-03-02  8:10:32 489143 [ERROR] Got error 180 when reading table './db_webqtl/ProbeSetXRef'
+```
+
+Let's try and dump those tables when the backup is done.
+
+```
+mariadb-dump -uwebqtlout db_webqtl SnpAll
+mariadb-dump: Error 1030: Got error 1877 "Unknown error 1877" from storage engine InnoDB when dumping table `SnpAll` at row: 0
+mariadb-dump -uwebqtlout db_webqtl ProbeSetXRef > ProbeSetXRef.sql
+```
+
+Eeep:
+
+```
+tux04:/etc$ mariadb-check -uwebqtlout -c db_webqtl ProbeSetXRef
+db_webqtl.ProbeSetXRef
+Warning  : InnoDB: Index ProbeSetFreezeId is marked as corrupted
+Warning  : InnoDB: Index ProbeSetId is marked as corrupted
+error    : Corrupt
+tux04:/etc$ mariadb-check -uwebqtlout -c db_webqtl SnpAll
+db_webqtl.SnpAll
+Warning  : InnoDB: Index PRIMARY is marked as corrupted
+Warning  : InnoDB: Index SnpName is marked as corrupted
+Warning  : InnoDB: Index Rs is marked as corrupted
+Warning  : InnoDB: Index Position is marked as corrupted
+Warning  : InnoDB: Index Source is marked as corrupted
+error    : Corrupt
+```
+
+On tux01 we have a working database, we can test with
+
+```
+mysqldump --no-data --all-databases > table_schema.sql
+mysqldump -uwebqtlout db_webqtl SnpAll > SnpAll.sql
+```
+
+Running the backup with rate limiting from:
+
+```
+Mar 02 17:09:59 tux04 sudo[548058]: pam_unix(sudo:session): session opened for user root(uid=0) by wrk(uid=1000)
+Mar 02 17:09:59 tux04 sudo[548058]:      wrk : TTY=pts/3 ; PWD=/export3/local/home/wrk/iwrk/deploy/gn-deploy-servers/scripts/tux04 ; USER=roo>
+Mar 02 17:09:55 tux04 sudo[548058]: pam_unix(sudo:auth): authentication failure; logname=wrk uid=1000 euid=0 tty=/dev/pts/3 ruser=wrk rhost= >
+Mar 02 17:04:26 tux04 su[548006]: pam_unix(su:session): session opened for user ibackup(uid=1003) by wrk(uid=0)
+```
+
+Oh oh
+
+Tux04 is showing errors on all disks. We have to bail out. I am copying the potentially corrupted files to tux01 right now. We have backups, so nothing serious I hope. I am only worried about the myisam files we have because they have no strong internal validation:
+
+```
+2025-03-04  8:32:45 502 [ERROR] db_webqtl.ProbeSetData: Record-count is not ok; is 5264578601   Should be: 5264580806
+2025-03-04  8:32:45 502 [Warning] db_webqtl.ProbeSetData: Found 28665 deleted space.   Should be 0
+2025-03-04  8:32:45 502 [Warning] db_webqtl.ProbeSetData: Found       2205 deleted blocks       Should be: 0
+2025-03-04  8:32:45 502 [ERROR] Got an error from thread_id=502, ./storage/myisam/ha_myisam.cc:1120
+2025-03-04  8:32:45 502 [ERROR] MariaDB thread id 502, OS thread handle 139625162532544, query id 837999 localhost webqtlout Checking table
+CHECK TABLE ProbeSetData
+2025-03-04  8:34:02 79695 [ERROR] mariadbd: Table './db_webqtl/ProbeSetData' is marked as crashed and should be repaired
+```
+
+See also
+
+=> https://dev.mysql.com/doc/refman/8.4/en/myisam-check.html
+
+Tux04 will require open heart 'disk controller' surgery and some severe testing before we move back. We'll also look at tux05-8 to see if they have similar problems.
+
+## Recovery
+
+According to the logs tux04 started showing serious errors on March 2nd - when I introduced sanitizing the mariadb backup:
+
+```
+Mar 02 05:00:42 tux04 kernel: I/O error, dev sde, sector 2071078320 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 2
+Mar 02 05:00:58 tux04 kernel: I/O error, dev sde, sector 2083650928 op 0x0:(READ) flags 0x80700 phys_seg 59 prio class 2
+...
+```
+
+The log started on Feb 23 when we had our last reboot. It probably is a good idea to turn on persistent logging! Anyway, it is likely files were fine until March 2nd. Similarly the mariadb logs also show
+
+```
+2025-03-02  6:53:52 489007 [ERROR] mariadbd: Index for table './db_webqtl/ProbeSetData.MYI' is corrupt; try to repair it
+2025-03-02  6:53:52 489007 [ERROR] db_webqtl.ProbeSetData: Can't read key from filepos: 2269659136
+```
+
+So, if we can restore a backup from March 1st we should be reasonably confident it is sane.
+
+First is to backup the existing database(!) Next restore the new DB by changing the DB location (symlink in /var/lib/mysql as well as check /etc/mysql/mariadb.cnf).
+
+When upgrading it is an idea to switch on these in mariadb.cnf
+
+```
+# forcing recovery with these two lines:
+innodb_force_recovery=3
+innodb_purge_threads=0
+```
+
+Make sure to disable (and restart) once it is up and running!
+
+So the steps are:
+
+* [X] install updated guix version of mariadb in /usr/local/guix-profiles (don't use Debian!!)
+* [X] repair borg backup
+* [X] Stop old mariadb (on new host tux02)
+* [X] backup old mariadb database
+* [X] restore 'sane' version of DB from borg March 1st
+* [X] point to new DB in /var/lib/mysql and cnf file
+* [X] update systemd settings
+* [X] start mariadb new version with recovery setting in cnf
+* [X] check logs
+* [X] once running revert on recovery setting in cnf and restart
+
+OK, looks like we are in business again. In the next phase we need to validate files. Normal files can be checked with
+
+```
+find -type f \( -not -name "md5sum.txt" \) -exec md5sum '{}' \; > md5sum.txt
+```
+
+and compared with another set on a different server with
+
+```
+md5sum -c md5sum.txt
+```
+
+* [X] check genotype file directory - some MAGIC files missing on tux01
+
+gn-docs is a git repo, so that is easily checked
+
+* [X] check gn-docs and sync with master repo
+
+
+## Other servers
+
+```
+journalctl -r|grep -i "I/O error"|less
+# tux05
+Nov 18 02:19:55 tux05 kernel: XFS (sdc2): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x78 len 8 error 74
+Nov 05 14:36:32 tux05 kernel: blk_update_request: I/O error, dev sdb, sector 1993616 op 0x1:(WRITE) flags
+0x0 phys_seg 35 prio class 0
+Jul 27 11:56:22 tux05 kernel: blk_update_request: I/O error, dev sdc, sector 55676616 op 0x0:(READ) flags
+0x80700 phys_seg 26 prio class 0
+Jul 27 11:56:22 tux05 kernel: blk_update_request: I/O error, dev sdc, sector 55676616 op 0x0:(READ) flags
+0x80700 phys_seg 26 prio class 0
+# tux06
+Apr 15 08:10:57 tux06 kernel: I/O error, dev sda, sector 21740352 op 0x1:(WRITE) flags 0x1000 phys_seg 4 prio class 2
+Dec 13 12:56:14 tux06 kernel: I/O error, dev sdb, sector 3910157327 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 2
+# tux07
+Mar 27 08:00:11 tux07 mfschunkserver[1927469]: replication error: failed to create chunk (No space left)
+# tux08
+Mar 27 08:12:11 tux08 mfschunkserver[464794]: replication error: failed to create chunk (No space left)
+```
+
+Tux04, 05 and 06 show disk errors. Tux07 and Tux08 are  overloaded with a full disk, but no other errors. We need to babysit Lizard more!
+
+```
+stress -v -d 1
+```
+
+Write test:
+
+```
+dd if=/dev/zero of=./test bs=512k count=2048 oflag=direct
+```
+
+Read test:
+
+```
+/sbin/sysctl -w vm.drop_caches=3
+dd if=./test of=/dev/zero bs=512k count=2048
+```
+
+
+smartctl -a /dev/sdd -d megaraid,0
+
+RAID Controller in SL 3: Dell PERC H755N Front
+
+# The story continues
+
+I don't know what happened but the server gave a hard
+error in the logs:
+
+```
+racadm getsel # get system log
+Record:      340
+Date/Time:   05/31/2025 09:25:17
+Source:      system
+Severity:    Critical
+Description: A high-severity issue has occurred at the Power-On
+Self-Test (POST) phase which has resulted in the system BIOS to
+abruptly stop functioning.
+```
+
+Woops! I fixed it by resetting idrac and rebooting remotely. Nasty.
+
+Looking around I found this link
+
+=>
+https://tomaskalabis.com/wordpress/a-high-severity-issue-has-occurred-at-the-power-on-self-te
+st-post-phase-which-has-resulted-in-the-system-bios-to-abruptly-stop-functioning/
+
+suggesting we should upgrade idrac firmware. I am not going to do that
+without backups and a fully up-to-date fallback online. It may fix the
+other hardware issues we have been seeing (who knows?).
+
+Fred, the boot sequence is not perfect yet. Turned out the network
+interfaces do not come up in the right order and nginx failed because
+of a missing /var/run/nginx. The container would not restart because -
+missing above - it could not check the certificates.
+
+## A week later
+
+```
+[SMM] APIC 0x00 S00:C00:T00 > ASSERT [AmdPlatformRasRsSmm] u:\EDK2\MdePkg\Library\BasePciSegmentLibPci\PciSegmentLib.c(766): ((Address) & (0xfffffffff0000000ULL | (3))) == 0                !!!! X64 Exception Type - 03(#BP - Breakpoint)  CPU Apic ID - 00000000 !!!!
+RIP  - 0000000076DA4343, CS  - 0000000000000038, RFLAGS - 0000000000000002
+RAX  - 0000000000000010, RCX - 00000000770D5B58, RDX - 00000000000002F8
+RBX  - 0000000000000000, RSP - 0000000077773278, RBP - 0000000000000000
+RSI  - 0000000000000087, RDI - 00000000777733E0                                              R8   - 00000000777731F8, R9  - 0000000000000000, R10 - 0000000000000000
+R11  - 00000000000000A0, R12 - 0000000000000000, R13 - 0000000000000000
+R14  - FFFFFFFFA0C1A118, R15 - 000000000005B000
+DS   - 0000000000000020, ES  - 0000000000000020, FS  - 0000000000000020
+GS   - 0000000000000020, SS  - 0000000000000020
+CR0  - 0000000080010033, CR2 - 0000000015502000, CR3 - 0000000077749000
+CR4  - 0000000000001668, CR8 - 0000000000000001
+DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000                      DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
+GDTR - 000000007773C000 000000000000004F, LDTR - 0000000000000000                            IDTR - 0000000077761000 00000000000001FF,   TR - 0000000000000040
+FXSAVE_STATE - 0000000077772ED0
+!!!! Find image based on IP(0x76DA4343) u:\Build_Genoa\DellBrazosPkg\DEBUG_MYTOOLS\X64\DellPkgs\DellChipsetPkgs\AmdGenoaModulePkg\Override\AmdCpmPkg\Features\PlatformRas\Rs\Smm\AmdPlatformRasRsSmm\DEBUG\AmdPlatformRasRsSmm.pdb (ImageBase=0000000076D3E000, EntryPoint=0000000076D3E6C0) !!!!
+```
+
+New error in system log:
+
+```
+Record:      341                                                                             Date/Time:   06/04/2025 19:47:08
+Source:      system
+Severity:    Critical                                                                        Description: A high-severity issue has occurred at the Power-On Self-Test (POST) phase which has resulted in the system BIOS to abruptly stop functioning.
+```
+
+The error appears to relate to AMD Brazos which is probably part of the on board APU/GPU.
+
+The code where it segfaulted is online at:
+
+=> https://github.com/tianocore/edk2/blame/master/MdePkg/Library/BasePciSegmentLibPci/PciSegmentLib.c
+
+and has to do with PCI registers and that can actually be caused by the new PCIe card we hosted.
diff --git a/issues/systems/tux04-production.gmi b/issues/systems/tux04-production.gmi
new file mode 100644
index 0000000..58ff8c1
--- /dev/null
+++ b/issues/systems/tux04-production.gmi
@@ -0,0 +1,279 @@
+# Production on tux04
+
+Lately we have been running production on tux04. Unfortunately Debian got broken and I don't see a way to fix it (something with python versions that break apt!). Also mariadb is giving problems:
+
+=> issues/production-container-mechanical-rob-failure.gmi
+
+and that is alarming. We might as well try an upgrade. I created a new partition on /dev/sda4 using debootstrap.
+
+The hardware RAID has proven unreliable on this machine (and perhaps others).
+
+We added a drive on a PCIe raiser outside the RAID. Use this for bulk data copying. We still bootstrap from the RAID.
+
+Luckily not too much is running on this machine and if we mount things again, most should work.
+
+# Tasks
+
+* [X] cleanly shut down mariadb
+* [X] reboot into new partition /dev/sda4
+* [X] git in /etc
+* [X] make sure serial boot works (/etc/default/grub)
+* [X] fix groups and users
+* [X] get guix going
+* [X] get mariadb going
+* [X] fire up GN2 service
+* [X] fire up SPARQL service
+* [X] sheepdog
+* [ ] fix CRON jobs and backups
+* [ ] test full reboots
+
+
+# Boot in new partition
+
+```
+blkid /dev/sda4
+/dev/sda4: UUID="4aca24fe-3ece-485c-b04b-e2451e226bf7" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="2e3d569f-6024-46ea-8ef6-15b26725f811"
+```
+
+After debootstrap there are two things to take care of: the /dev directory and grub. For good measure
+I also capture some state
+
+```
+cd ~
+ps xau > cron.log
+systemctl > systemctl.txt
+cp /etc/network/interfaces .
+cp /boot/grub/grub.cfg .
+```
+
+we should still have access to the old root partition, so I don't need to capture everything.
+
+## /dev
+
+I ran MAKEDEV and that may not be needed with udev.
+
+## grub
+
+We need to tell grub to boot into the new partition. The old root is on
+UUID=8e874576-a167-4fa1-948f-2031e8c3809f /dev/sda2.
+
+Next I ran
+
+```
+tux04:~$ update-grub2 /dev/sda
+Generating grub configuration file ...
+Found linux image: /boot/vmlinuz-5.10.0-32-amd64
+Found initrd image: /boot/initrd.img-5.10.0-32-amd64
+Found linux image: /boot/vmlinuz-5.10.0-22-amd64
+Found initrd image: /boot/initrd.img-5.10.0-22-amd64
+Warning: os-prober will be executed to detect other bootable partitions.
+Its output will be used to detect bootable binaries on them and create new boot entries.
+Found Debian GNU/Linux 12 (bookworm) on /dev/sda4
+Found Windows Boot Manager on /dev/sdd1@/efi/Microsoft/Boot/bootmgfw.efi
+Found Debian GNU/Linux 11 (bullseye) on /dev/sdf2
+```
+
+Very good. Do a diff on grub.cfg and you see it even picked up the serial configuration. It only shows it added menu entries for the new boot. Very nice.
+
+At this point I feel safe to boot as we should be able to get back into the old partition.
+
+# /etc/fstab
+
+The old fstab looked like
+
+```
+UUID=8e874576-a167-4fa1-948f-2031e8c3809f /               ext4    errors=remount-ro 0       1
+# /boot/efi was on /dev/sdc1 during installation
+UUID=998E-68AF  /boot/efi       vfat    umask=0077      0       1
+# swap was on /dev/sdc3 during installation
+UUID=cbfcd84e-73f8-4cec-98ee-40cad404735f none            swap    sw              0       0
+UUID="783e3bd6-5610-47be-be82-ac92fdd8c8b8"   /export2    ext4   auto     0   2
+UUID="9e6a9d88-66e7-4a2e-a12c-f80705c16f4f"   /export     ext4   auto     0   2
+UUID="f006dd4a-2365-454d-a3a2-9a42518d6286"   /export3    auto   auto     0   2
+/export2/gnu /gnu none defaults,bind 0 0
+# /dev/sdd1: PARTLABEL="bulk" PARTUUID="b1a820fe-cb1f-425e-b984-914ee648097e"
+# /dev/sdb4   /export   ext4    auto   0       2
+# /dev/sdd1   /export2   ext4    auto   0       2
+```
+
+# reboot
+
+Next we are going to reboot, and we need a serial connector to the Dell out-of-band using racadm:
+
+```
+ssh IP
+console com2
+racadm getsel
+racadm serveraction powercycle
+racadm serveraction powerstatus
+
+```
+
+Main trick it so hit ESC, wait 2 sec and 2 when you want the bios boot menu. Ctrl-\ to escape console. Otherwise ESC (wait) ! to get to the boot menu.
+
+# First boot
+
+It still boots by default into the old root. That gave an error:
+
+[FAILED] Failed to start File Syste…a-2365-454d-a3a2-9a42518d6286
+
+This is /export3. We can fix that later.
+
+When I booted into the proper partition the console clapped out. Also the racadm password did not work on tmux -- I had to switch to a standard console to log in again. Not sure why that is, but next I got:
+
+```
+Give root password for maintenance
+(or press Control-D to continue):
+```
+
+and giving the root password I was in maintenance mode on the correct partition!
+
+To rerun grup I had to add `GRUB_DISABLE_OS_PROBER=false`.
+
+Once booting up it is a matter of mounting partitions and tick the check boxes above.
+
+The following contained errors:
+
+```
+/dev/sdd1       3.6T  1.8T  1.7T  52% /export2
+```
+
+# Guix
+
+Getting guix going is a bit tricky because we want to keep the store!
+
+```
+cp -vau /mnt/old-root/var/guix/ /var/
+cp -vau /mnt/old-root/usr/local/guix-profiles /usr/local/
+cp -vau /mnt/old-root/usr/local/bin/* /usr/local/bin/
+cp -vau /mnt/old-root/etc/systemd/system/guix-daemon.service* /etc/systemd/system/
+cp -vau /mnt/old-root/etc/systemd/system/gnu-store.mount* /etc/systemd/system/
+```
+
+Also had to add guixbuild users and group by hand.
+
+# nginx
+
+We use the streaming facility. Check that
+
+```
+nginx -V
+```
+
+lists --with-stream=static, see
+
+=> https://serverfault.com/questions/858067/unknown-directive-stream-in-etc-nginx-nginx-conf86/858074#858074
+
+and load at the start of nginx.conf:
+
+```
+load_module /usr/lib/nginx/modules/ngx_stream_module.so;
+```
+
+and
+
+```
+nginx -t
+```
+
+passes
+
+Now the container responds to the browser with `Internal Server Error`.
+
+# container web server
+
+Visit the container with something like
+
+```
+nsenter -at 2838 /run/current-system/profile/bin/bash --login
+```
+
+The nginx log in the container has many
+
+```
+2025/02/22 17:23:48 [error] 136#0: *166916 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: genenetwork.org, request: "GET /gn3/gene/aliases/st%2029:1;o;s HTTP/1.1", upstream: "http://127.0.0.1:9800/gene/aliases/st%2029:1;o;s", host: "genenetwork.org"
+```
+
+that is interesting. Acme/https is working because GN2 is working:
+
+```
+curl https://genenetwork.org/api3/version
+"1.0"
+```
+
+Looking at the logs it appears it is a redis problem first for GN2.
+
+Fred builds the container with `/home/fredm/opt/guix-production/bin/guix`. Machines are defined in
+
+```
+fredm@tux04:/export3/local/home/fredm/gn-machines
+```
+
+The shared dir for redis is at
+
+--share=/export2/guix-containers/genenetwork/var/lib/redis=/var/lib/redis
+
+with
+
+```
+root@genenetwork-production /var# ls lib/redis/ -l
+-rw-r--r-- 1 redis redis 629328484 Feb 22 17:25 dump.rdb
+```
+
+In production.scm it is defined as
+
+```
+(service redis-service-type
+         (redis-configuration
+          (bind "127.0.0.1")
+          (port 6379)
+          (working-directory "/var/lib/redis")))
+```
+
+The defaults are the same as the definition of redis-service-type (in guix). Not sure why we are duplicating.
+
+After starting redis by hand I get another error `500 DatabaseError: The following exception was raised while attempting to access http://auth.genenetwork.org/auth/data/authorisation: database disk image is malformed`. The problem is it created
+a DB in the wrong place. Alright, the logs in the container say:
+
+```
+Feb 23 14:04:31 genenetwork-production shepherd[1]: [redis-server] 3977:C 23 Feb 2025 14:04:31.040 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
+Feb 23 14:04:31 genenetwork-production shepherd[1]: [redis-server] 3977:C 23 Feb 2025 14:04:31.040 # Redis version=7.0.12, bits=64, commit=00000000, modified=0, pid=3977, just started
+Feb 23 14:04:31 genenetwork-production shepherd[1]: [redis-server] 3977:C 23 Feb 2025 14:04:31.040 # Configuration loaded
+Feb 23 14:04:31 genenetwork-production shepherd[1]: [redis-server] 3977:M 23 Feb 2025 14:04:31.041 * Increased maximum number of open files to 10032 (it was originally set to 1024).
+Feb 23 14:04:31 genenetwork-production shepherd[1]: [redis-server] 3977:M 23 Feb 2025 14:04:31.041 * monotonic clock: POSIX clock_gettime
+Feb 23 14:04:31 genenetwork-production shepherd[1]: [redis-server] 3977:M 23 Feb 2025 14:04:31.041 * Running mode=standalone, port=6379.
+Feb 23 14:04:31 genenetwork-production shepherd[1]: [redis-server] 3977:M 23 Feb 2025 14:04:31.042 # Server initialized
+Feb 23 14:04:31 genenetwork-production shepherd[1]: [redis-server] 3977:M 23 Feb 2025 14:04:31.042 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
+Feb 23 14:04:31 genenetwork-production shepherd[1]: [redis-server] 3977:M 23 Feb 2025 14:04:31.042 # Wrong signature trying to load DB from file
+Feb 23 14:04:31 genenetwork-production shepherd[1]: [redis-server] 3977:M 23 Feb 2025 14:04:31.042 # Fatal error loading the DB: Invalid argument. Exiting.
+Feb 23 14:04:31 genenetwork-production shepherd[1]: Service redis (PID 3977) exited with 1.
+```
+
+This is caused by a newer version of redis. This is odd because we are using the same version from the container?!
+
+Actually it turned out the redis DB was corrupted on the SSD! Same for some other databases (ugh).
+
+Fred copied all data to an enterprise level storage, and we rolled back to some older DBs, so hopefully we'll be OK for now.
+
+# Reinstating backups
+
+In the next step we need to restore backups as described in
+
+=> /topics/systems/backups-with-borg
+
+I already created an ibackup user. Next we test the backup script for mariadb.
+
+One important step is to check the database:
+
+```
+/usr/bin/mariadb-check -c -u user -p* db_webqtl
+```
+
+A successful mariadb backup consists of multiple steps
+
+```
+2025-02-27 11:48:28 +0000 (ibackup@tux04) SUCCESS 0 <32m43s> mariabackup-dump
+2025-02-27 11:48:29 +0000 (ibackup@tux04) SUCCESS 0 <00m00s> mariabackup-make-consistent
+2025-02-27 12:16:37 +0000 (ibackup@tux04) SUCCESS 0 <28m08s> borg-tux04-sql-backup
+2025-02-27 12:16:46 +0000 (ibackup@tux04) SUCCESS 0 <00m07s> drop-rsync-balg01
+```