Octopus cluster updates

1 files changed, 43 insertions, 0 deletions

diff --git a/topics/octopus/octopussy-needs-love.gmi b/topics/octopus/octopussy-needs-love.gmi
new file mode 100644
index 0000000..ee10240
--- /dev/null
+++ b/topics/octopus/octopussy-needs-love.gmi
@@ -0,0 +1,43 @@
+# Octopussy needs love
+
+At UTHSC, Memphis, TN, around October 2020 Efraim and I installed Octopus on Debian+Guix with lizard as a distributed network storage system and slurm for job control. Around October 2023 we added 5 genoa tux05-09 machines, doubling the cluster in size. See
+
+=> https://genenetwork.org/gn-docs/facilities
+
+Octopus made a lot of work possible we can't really do on larger HPCs and led to a bunch of high impact studies and publications, particularly on pangenomics.
+
+In the coming period we want te replace lizard with moosefs. Lizard is no longer maintained and as it was a fork of Moose, it is only logical to go forward on that one. We also looked at Ceph, but apparently Ceph is not great for systems that carry no redundancy. So far, lizard has been using redundancy, but we figure we can do without if the occassional (cheap) SSD goes bad.
+We'll run moose on Debian.
+
+We also need to look at upgrading some of the Dell BIOS - particularly tux05-09 - as they can be occassionally problematic with non-OEM SSDs.
+
+On the worker nodes it may be wise to upgrade Debian. Followed by an upgrade to the head nodes and other supporting machines. Even though we rely on Guix for latest and greatest, there may be good upgrades in the underlying Linux kernel and drivers.
+
+Our PBS we are up-to-date because we run that completely on Guix and Arun supports the latest and greatest.
+
+Another thing we ought to fix is introduce centralized user management. So far we have had few users and just got by. But sometimes it bites us that users have different UIDs on the nodes.
+
+
+# Tasks
+
+* [ ] Upgrade bios (tuxes)
+* [ ] Install moosefs
+* [ ] Migrate lizardfs nodes to moosefs (one at a time)
+
+# Progress
+
+## Lizardfs and Moosefs
+
+Our Lizard documention lives at
+
+=> lizardfs/README
+
+Efraim wrote a lizardfs for Guix at the time in guix-bioinformatics, but we ended up deploying with Debian. Going back now, the package does not look too taxing (I think we dropped it because the Guix system configuration did not play well).
+
+=> https://git.genenetwork.org/guix-bioinformatics/tree/gn/packages/file-systems.scm
+
+Looking at the Debian package
+
+=> https://salsa.debian.org/debian/moosefs
+
+It carries no special patches, but a few nice hints in *.README.debian. I think it is worth trying to write a Guix package so we can easily upgrade (even on an aging Debian). Future proofing is key.