diff options
| -rw-r--r-- | topics/systems/migrate-p2.gmi | 12 | ||||
| -rw-r--r-- | topics/systems/orchestration.gmi | 31 |
2 files changed, 43 insertions, 0 deletions
diff --git a/topics/systems/migrate-p2.gmi b/topics/systems/migrate-p2.gmi new file mode 100644 index 0000000..c7fcb90 --- /dev/null +++ b/topics/systems/migrate-p2.gmi @@ -0,0 +1,12 @@ +* Penguin2 crash + +This week the boot partition of P2 crashed. We have a few lessons here, not least having a fallback for all services ;) + +* Tasks + +- [ ] setup space.uthsc.edu for GN2 development +- [ ] update DNS to tux02 128.169.4.52 and space 128.169.5.175 +- [ ] move CI/CD to tux02 + + +* Notes diff --git a/topics/systems/orchestration.gmi b/topics/systems/orchestration.gmi new file mode 100644 index 0000000..336dbbd --- /dev/null +++ b/topics/systems/orchestration.gmi @@ -0,0 +1,31 @@ +* Orchestration and fallbacks + +After the Penguin2 crash in Aug. 2022 it has become increasingly clear how hard it is to deploy GeneNetwork. GNU Guix helps a great deal with dependencies, but it does not handle orchestration between machines/services well. Also we need to look at the future. + +What is GN today in terms of services + + 1. Main GN2 server (Python, 20+ processes, 3+ instances: depends on all below) + 2. Matching GN3 server and REST endpoint (Python: less dependencies) + 3. Mariadb + 4. redis + 5. virtuoso + 6. GN-proxy (Racket, authentication handler: redis, mariadb) + 7. Alias proxy (Racket, gene aliases wikidata) + 8. Jupyter R and Julia notebooks + 9. BNW server (Octave) +10. UCSC browser +11. GN1 instances (older python, 12 instances in principle, 2 running today) +12. Access to HPC for GEMMA (coming) +13. Backup services +14. monitoring services + +I am still missing a few! All run by a man and his diligent dog. + +For the future the orchestration needs to be more robust and resilient. This means: + + 1. A fallback for every service on a separate machine + 2. Improved privacy protection for (future) human data + 3. Separate servers serving different data sources + 4. Partial synchronization between data sources + +The only way we *can* scale is by adding machines. But the system is not yet ready for that. Also getting rid of monolithic primary databases in favor of files helps synchronization. |