summaryrefslogtreecommitdiff
path: root/topics
diff options
context:
space:
mode:
Diffstat (limited to 'topics')
-rw-r--r--topics/ai/aider.gmi6
-rw-r--r--topics/ai/ontogpt.gmi7
-rw-r--r--topics/database/mariadb-database-architecture.gmi66
-rw-r--r--topics/deploy/installation.gmi2
-rw-r--r--topics/genenetwork-releases.gmi77
-rw-r--r--topics/systems/backup-drops.gmi21
-rw-r--r--topics/systems/backups-with-borg.gmi202
-rw-r--r--topics/systems/ci-cd.gmi48
-rw-r--r--topics/systems/screenshot-github-webhook.pngbin0 -> 177112 bytes
9 files changed, 390 insertions, 39 deletions
diff --git a/topics/ai/aider.gmi b/topics/ai/aider.gmi
index 71dfa9e..aa88e71 100644
--- a/topics/ai/aider.gmi
+++ b/topics/ai/aider.gmi
@@ -1,12 +1,16 @@
# Aider
-https://aider.chat/
+=> https://aider.chat/
+```
python3 -m venv ~/opt/python-aider
~/opt/python-aider/bin/python3 -m pip install aider-install
~/opt/python-aider/bin/aider-install
+```
Installed 1 executable: aider
Executable directory /home/wrk/.local/bin is already in PATH
+```
aider --model gpt-4o --openai-api-key aa...
+```
diff --git a/topics/ai/ontogpt.gmi b/topics/ai/ontogpt.gmi
new file mode 100644
index 0000000..94bd165
--- /dev/null
+++ b/topics/ai/ontogpt.gmi
@@ -0,0 +1,7 @@
+# OntoGPT
+
+python3 -m venv ~/opt/ontogpt
+~/opt/ontogpt/bin/python3 -m pip install ontogpt
+
+
+runoak set-apikey -e openai
diff --git a/topics/database/mariadb-database-architecture.gmi b/topics/database/mariadb-database-architecture.gmi
index 5c9b0c5..0454d71 100644
--- a/topics/database/mariadb-database-architecture.gmi
+++ b/topics/database/mariadb-database-architecture.gmi
@@ -28,6 +28,12 @@ Naming convention-wise there is a confusing use of id and data-id in particular.
The default install comes with a smaller database which includes a
number of the BXDs and the Human liver dataset (GSE9588).
+It can be downloaded from:
+
+=> https://files.genenetwork.org/database/
+
+Try the latest one first.
+
# GeneNetwork database
Estimated table sizes with metadata comment for the important tables
@@ -536,8 +542,8 @@ select * from ProbeSetSE limit 5;
For the other tables, you may check the GN2/doc/database.org document (the starting point for this document).
-# Contributions regarding data upload to the GeneNetwork webserver
-* Ideas shared by the GeneNetwork team to facilitate the process of uploading data to production
+# Contributions regarding data upload to the GeneNetwork webserver
+* Ideas shared by the GeneNetwork team to facilitate the process of uploading data to production
## Quality check and integrity of the data to be uploaded to gn2
@@ -556,7 +562,7 @@ For the other tables, you may check the GN2/doc/database.org document (the start
* Unique identifiers solve the hurdles that come with having duplicate genes. So, the QA tools in place should ensure the uploaded dataset adheres to the requirements mentioned
* However, newer RNA-seq data sets generated by sequencing do not usually have an official vendor identifier. The identifier is usually based on the NCBI mRNA model (NM_XXXXXX) that was used to evaluate an expression and on the sequence that is involved, usually the start and stop nucleotide positions based on a specific genome assembly or just a suffix to make sure it is unique. In this case, you are looking at mRNA assays for a single transcript, but different parts of the transcript that have different genome coordinates. We now typically use ENSEMBL identifiers.
* The mouse version of the sonic hedgehog gene as an example: `ENSMUST00000002708` or `ENSMUSG00000002633` sources should be fine. The important thing is to know the provenance of the ID—who is in charge of that ID type?
-* When a mRNA assay is super precise (one exon only or a part of the 5' UTR), then we should use exon identifiers from ENSEMBL probably.
+* When a mRNA assay is super precise (one exon only or a part of the 5' UTR), then we should use exon identifiers from ENSEMBL probably.
* Ideally, we should enter the sequence's first and last 100 nt in GeneNetwork for verification and alignment. We did this religiously for arrays, but have started to get lazy now. The sequence is the ultimate identifier
* For methylation arrays and CpG assays, we can use this format `cg14050475` as seen in MBD UTHSC Ben's data
* For metabolites like isoleucine—the ID we have been using is the mass-to-charge (MZ) ratio such as `130.0874220_MZ`
@@ -579,16 +585,16 @@ abcb10_q9ji39_t312
## BXD individuals
-* Basically groups (represented by the InbredSet tables) are primarily defined by their list of samples/strains (represented by the Strain tables). When we create a new group, it's because we have data with a distinct set of samples/strains from any existing groups.
-* So when we receive data for BXD individuals, as far as the database is concerned they are a completely separate group (since the list of samples is new/distinct from any other existing groups). We can choose to also enter it as part of the "generic" BXD group (by converting it to strain means/SEs using the strain of each individual, assuming it's provided like in the files Arthur was showing us).
+* Basically groups (represented by the InbredSet tables) are primarily defined by their list of samples/strains (represented by the Strain tables). When we create a new group, it's because we have data with a distinct set of samples/strains from any existing groups.
+* So when we receive data for BXD individuals, as far as the database is concerned they are a completely separate group (since the list of samples is new/distinct from any other existing groups). We can choose to also enter it as part of the "generic" BXD group (by converting it to strain means/SEs using the strain of each individual, assuming it's provided like in the files Arthur was showing us).
* This same logic could apply to other groups as well - we could choose to make one group the "strain mean" group for another set of groups that contain sample data for individuals. But the database doesn't reflect the relationship between these groups*
* As far as the database is concerned, there is no distinction between strain means and individual sample data - they're all rows in the ProbeSetData/PublishData tables. The only difference is that strain mean data will probably also have an SE value in the ProbeSetSE/PublishSE tables and/or an N (number of individuals per strain) value in the NStrain table
* As for what this means for the uploader - I think it depends on whether Rob/Arthur/etc wants to give users the ability to simultaneously upload both strain mean and individual data. For example, if someone uploads some BXD individuals' data, do we want the uploader to both create a new group for this (or add to an existing BXD individuals group) and calculate the strain means/SE and enter it into the "main" BXD group? My personal feeling is that it's probably best to postpone that for later and only upload the data with the specific set of samples indicated in the file since it would insert some extra complexity to the uploading process that could always be added later (since the user would need to select "the group the strains are from" as a separate option)
* The relationship is sorta captured in the CaseAttribute and CaseAttributeXRefNew tables (which contain sample metadata), but only in the form of the metadata that is sometimes displayed as extra columns in the trait page table - this data isn't used in any queries/analyses currently (outside of some JS filters run on the table itself) and isn't that important as part of the uploading process (or at least can be postponed)
-## Individual Datasets and Derivatives datasets in gn2
-* Individual dataset reflects the actual data provided or submitted by the investigator (user). Derivative datasets include the processed information from the individual dataset, as in the case of the average datasets.
-* An example of an individual dataset would look something like; (MBD dataset)
+## Individual Datasets and Derivatives datasets in gn2
+* Individual dataset reflects the actual data provided or submitted by the investigator (user). Derivative datasets include the processed information from the individual dataset, as in the case of the average datasets.
+* An example of an individual dataset would look something like; (MBD dataset)
```
#+begin_example
sample, strain, Sex, Age,…
@@ -600,13 +606,13 @@ FEB0005,BXD16,F,14,…
⋮
#+end_example
```
-* The strain column above has repetitive values. Each value has a one-to-many relationship with values on sample column. From this dataset, there can be several derivatives. For example;
-- Sex-based categories
-- Average data (3 sample values averaged to one strain value)
-- Standard error table computed for the averages
+* The strain column above has repetitive values. Each value has a one-to-many relationship with values on sample column. From this dataset, there can be several derivatives. For example;
+- Sex-based categories
+- Average data (3 sample values averaged to one strain value)
+- Standard error table computed for the averages
-## Saving data to database
-* Strain table schema
+## Saving data to database
+* Strain table schema
```
#+begin_src sql
MariaDB [db_webqtl]> DESC Strain;
@@ -639,21 +645,21 @@ FEB0005,BXD16,F,14,…
5 rows in set (0.00 sec)
#+end_src
```
-* Where the =InbredSetId= comes from the =InbredSet= table and the =StrainId= comes from the =Strain= table. The *individual data* would be linked to an inbredset group that is for individuals
+* Where the =InbredSetId= comes from the =InbredSet= table and the =StrainId= comes from the =Strain= table. The *individual data* would be linked to an inbredset group that is for individuals
* For the *average data*, the only value to save would be the =strain= field, which would be saved as =Name= in the =Strain= table and linked to an InbredSet group that is for averages
*Question 01*: How do we distinguish the inbredset groups?
*Answer*: The =Family= field is useful for this.
*Question 02*: If you have more derived "datasets", e.g. males-only, females-only, under-10-years, 10-to-25-years, etc. How would the =Strains= table handle all those differences?
-## Metadata
+## Metadata
* The data we looked at had =gene id= and =gene symbol= fields. These fields were used to fetch the *Ensembl ID* and *descriptions* from [[https://www.ncbi.nlm.nih.gov/][NCBI]] and the [[https://useast.ensembl.org/][Ensembl Genome Browser]]
-## Files for mapping
+## Files for mapping
* Files used for mapping need to be in =bimbam= or =.geno= formats. We would need to do conversions to at least one of these formats where necessary
-## Annotation files
-* Consider the following schema of DB tables
+## Annotation files
+* Consider the following schema of DB tables
#+begin_src sql
MariaDB [db_webqtl]> DESC InbredSet;
+-----------------+----------------------+------+-----+---------+----------------+
@@ -718,10 +724,10 @@ FEB0005,BXD16,F,14,…
- The =used_for_mapping= field should be set to ~Y~ unless otherwise informed
- The =PedigreeStatus= field is unknown to us for now: set to ~NULL~
-* Annotation file format
+* Annotation file format
The important fields are:
- =ChipId=: The platform that the data was collected from/with
-Consider the following table;
+Consider the following table;
#+begin_src sql
MariaDB [db_webqtl]> DESC GeneChip;
+---------------+----------------------+------+-----+---------+----------------+
@@ -744,7 +750,7 @@ Consider the following table;
- =Probe_set_Blat_Mb_start=/=Probe_set_Blat_Mb_end=: In Byron's and Beni's data, these correspond to the =geneStart= and =geneEnd= fields respectively. These are the positions, in megabasepairs, that the gene begins and ends at, respectively.
- =Mb=: This is the =geneStart=/=Probe_set_Blat_Mb_start= value divided by *1000000*. (*Note to self*: Maybe the Probe_set_Blat_Mb_* fields above might not be in megabase pairs — please confirm)
- =Strand_Probe= and =Strand_Gene=: These fields' values are simply ~+~ or ~-~. If these values are missing, you can [[https://ftp.ncbi.nih.gov/gene/README][retrieve them from NCBI]], specifically from the =orientation= field of seemingly any text file with the field
- - =Chr=: This is the chromosome on which the gene is found
+ - =Chr=: This is the chromosome on which the gene is found
* The final annotation file will have (at minimum) the following fields (or their
analogs):
@@ -765,8 +771,8 @@ analogs):
* =.geno= Files
- The =.geno= files have sample names, not the strain/symbol. The =Locus= field in the =.geno= file corresponds to the **marker**. =.geno= files are used with =QTLReaper=
- The sample names in the ~.geno~ files *MUST* be in the same order as the
-strains/symbols for that species. For example;
-Data format is as follows;
+strains/symbols for that species. For example;
+Data format is as follows;
```
#+begin_example
SampleName,Strain,…
@@ -779,7 +785,7 @@ BJCWI0005,BXD50,…
⋮
#+end_example
```
-and the order of strains is as follows;
+and the order of strains is as follows;
```
#+begin_example
…,BXD33,…,BXD40,…,BXD50,…
@@ -806,9 +812,9 @@ The order of samples that belong to the same strain is irrelevant - they share t
- Treatment
- Sex (Really? Isn't sex an expression of genes?)
- batch
- - Case ID, etc
+ - Case ID, etc
-* Summary steps to load data to the database
+* Summary steps to load data to the database
- [x] Create *InbredSet* group (think population)
- [x] Load the strains/samples data
- [x] Load the sample cross-reference data to link the samples to their
@@ -821,8 +827,4 @@ The order of samples that belong to the same strain is irrelevant - they share t
- [x] Load the *Log2* data (ProbeSetData and ProbeSetXRef tables)
- [x] Compute means (an SQL query was used — this could be pre-computed in code
and entered along with the data)
-- [x] Run QTLReaper
-
-
-
-
+- [x] Run QTLReaper
diff --git a/topics/deploy/installation.gmi b/topics/deploy/installation.gmi
index 757d848..d6baa79 100644
--- a/topics/deploy/installation.gmi
+++ b/topics/deploy/installation.gmi
@@ -319,7 +319,7 @@ Currently we have two databases for deployment,
from BXD mice and 'db_webqtl_plant' which contains all plant related
material.
-Download one database from
+Download a recent database from
=> https://files.genenetwork.org/database/
diff --git a/topics/genenetwork-releases.gmi b/topics/genenetwork-releases.gmi
new file mode 100644
index 0000000..e179629
--- /dev/null
+++ b/topics/genenetwork-releases.gmi
@@ -0,0 +1,77 @@
+# GeneNetwork Releases
+
+## Tags
+
+* status: open
+* priority:
+* assigned:
+* type: documentation
+* keywords: documentation, docs, release, releases, genenetwork
+
+## Introduction
+
+The sections that follow will be note down the commits used for various stable (and stable-ish) releases of genenetwork.
+
+The tagging of the commits will need to distinguish repository-specific tags from overall system tags.
+
+In this document, we only concern ourselves with the overall system tags, that shall have the template:
+
+```
+genenetwork-system-v<major>.<minor>.<patch>[-<commit>]
+```
+
+the portions in angle brackets will be replaced with the actual version numbers.
+
+## genenetwork-system-v1.0.0
+
+This is the first, guix-system-container-based, stable release of the entire genenetwork system.
+The commits involved are:
+
+=> https://github.com/genenetwork/genenetwork2/commit/314c6d597a96ac903071fcb6e50df3d9e88935e9 GN2: 314c6d5
+=> https://github.com/genenetwork/genenetwork3/commit/0d902ec267d96b87648669a7a43b699c8a22a3de GN3: 0d902ec
+=> https://git.genenetwork.org/gn-auth/commit/?id=8e64f7f8a392b8743a4f36c497cd2ec339fcfebc: gn-auth: 8e64f7f
+=> https://git.genenetwork.org/gn-libs/commit/?id=72a95f8ffa5401649f70978e863dd3f21900a611: gn-libs: 72a95f8
+
+The guix channels used for deployment of the system above are as follows:
+
+```
+(list (channel
+ (name 'guix-bioinformatics)
+ (url "https://git.genenetwork.org/guix-bioinformatics/")
+ (branch "master")
+ (commit
+ "039a3dd72c32d26b9c5d2cc99986fd7c968a90a5"))
+ (channel
+ (name 'guix-forge)
+ (url "https://git.systemreboot.net/guix-forge/")
+ (branch "main")
+ (commit
+ "bcb3e2353b9f6b5ac7bc89d639e630c12049fc42")
+ (introduction
+ (make-channel-introduction
+ "0432e37b20dd678a02efee21adf0b9525a670310"
+ (openpgp-fingerprint
+ "7F73 0343 F2F0 9F3C 77BF 79D3 2E25 EE8B 6180 2BB3"))))
+ (channel
+ (name 'guix-past)
+ (url "https://gitlab.inria.fr/guix-hpc/guix-past")
+ (branch "master")
+ (commit
+ "5fb77cce01f21a03b8f5a9c873067691cf09d057")
+ (introduction
+ (make-channel-introduction
+ "0c119db2ea86a389769f4d2b9c6f5c41c027e336"
+ (openpgp-fingerprint
+ "3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5"))))
+ (channel
+ (name 'guix)
+ (url "https://git.savannah.gnu.org/git/guix.git")
+ (branch "master")
+ (commit
+ "2394a7f5fbf60dd6adc0a870366adb57166b6d8b")
+ (introduction
+ (make-channel-introduction
+ "9edb3f66fd807b096b48283debdcddccfea34bad"
+ (openpgp-fingerprint
+ "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA")))))
+```
diff --git a/topics/systems/backup-drops.gmi b/topics/systems/backup-drops.gmi
index 191b185..a4677f7 100644
--- a/topics/systems/backup-drops.gmi
+++ b/topics/systems/backup-drops.gmi
@@ -4,6 +4,10 @@ To make backups we use a combination of sheepdog, borg, sshfs, rsync. sheepdog i
This system proves pretty resilient over time. Only on the synology server I can't get it to work because of some CRON permission issue.
+For doing the actual backups see
+
+=> ./backups-with-borg.gmi
+
# Tags
* assigned: pjotrp
@@ -13,7 +17,7 @@ This system proves pretty resilient over time. Only on the synology server I can
## Borg backups
-It is advised to use a backup password and not store that on the remote.
+Despite our precautions it is advised to use a backup password and *not* store that on the remote.
## Running sheepdog on rabbit
@@ -59,14 +63,14 @@ where remote can be an IP address.
Warning: if you introduce this `AllowUsers` command all users should be listed or people may get locked out of the machine.
-Next create a special key on the backup machine's ibackup user (just hit enter):
+Next create a special password-less key on the backup machine's ibackup user (just hit enter):
```
su ibackup
ssh-keygen -t ecdsa -f $HOME/.ssh/id_ecdsa_backup
```
-and copy the public key into the remote /home/bacchus/.ssh/authorized_keys
+and copy the public key into the remote /home/bacchus/.ssh/authorized_keys.
Now test it from the backup server with
@@ -82,13 +86,20 @@ On the drop server you can track messages by
tail -40 /var/log/auth.log
```
+or on recent linux with systemd
+
+```
+journalctl -r
+```
+
Next
```
ssh -v -i ~/.ssh/id_ecdsa_backup bacchus@dropserver
```
-should give a Broken pipe(!). In auth.log you may see something like
+should give a Broken pipe(!) or it says `This service allows sftp connections only`.
+When running sshd with a verbose switch you may see something like
fatal: bad ownership or modes for chroot directory component "/export/backup/"
@@ -171,3 +182,5 @@ sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,IdentityFile=~/.
The recent scripts can be found at
=> https://github.com/genenetwork/gn-deploy-servers/blob/master/scripts/tux01/backup_drop.sh
+
+# borg-borg
diff --git a/topics/systems/backups-with-borg.gmi b/topics/systems/backups-with-borg.gmi
new file mode 100644
index 0000000..5cdb2a3
--- /dev/null
+++ b/topics/systems/backups-with-borg.gmi
@@ -0,0 +1,202 @@
+# Borg backups
+
+We use borg for backups. Borg is an amazing tool and after 25+ years of making backups it just feels right.
+With the new tux04 production install we need to organize backups off-site. The first step is to create a
+borg runner using sheepdog -- sheepdog we use for monitoring success/failure.
+Sheepdog essentially wraps a Unix command and sends a report to a local or remote redis instance.
+Sheepdog also includes a web server for output:
+
+=> http://sheepdog.genenetwork.org/sheepdog/status.html
+
+# Tags
+
+* assigned: pjotrp
+* keywords: systems, backup, sheepdog, database
+
+# Install borg
+
+Usually I use a version of borg from guix. This should really be done as the borg user (ibackup).
+
+```
+mkdir ~/opt
+guix package -i borg ~/opt/borg
+tux04:~$ ~/opt/borg/bin/borg --version
+ 1.2.2
+```
+
+# Create a new backup dir and user
+
+The backup should live on a different disk from the things we backup, so when that disk fails we have another.
+
+The SQL database lives on /export and the containers live on /export2. /export3 is a largish slow drive, so perfect.
+
+By convention I point /export/backup to the real backup dir on /export3/backup/borg/ Another convention is that we use an ibackup user which has the backup passphrase in ~/.borg-pass. As root:
+
+```
+mkdir /export/backup/borg
+chown ibackup:ibackup /export/backup/borg
+chown ibackup:ibackup /home/ibackup/.borg-pass
+su ibackup
+```
+
+Now you should be able to load the passphrase and create the backup dir
+
+```
+id
+ uid=1003(ibackup)
+. ~/.borg-pass
+cd /export/backup/borg
+~/opt/borg/bin/borg init --encryption=repokey-blake2 genenetwork
+```
+
+Now we can run our first backup. Note that ibackup should be a member of the mysql and gn groups
+
+```
+mysql:x:116:ibackup
+```
+
+# First backup
+
+Run the backup the first time:
+
+```
+id
+ uid=1003(ibackup) groups=1003(ibackup),116(mysql)
+~/opt/borg/bin/borg create --progress --stats genenetwork::first-backup /export/mysql/database/*
+```
+
+You may first need to update permissions to give group access
+
+```
+chmod g+rx -R /var/lib/mysql/*
+```
+
+When that works borg reports:
+
+```
+Archive name: first-backup
+Archive fingerprint: 376d32fda9738daa97078fe4ca6d084c3fa9be8013dc4d359f951f594f24184d
+Time (start): Sat, 2025-02-08 04:46:48
+Time (end): Sat, 2025-02-08 05:30:01
+Duration: 43 minutes 12.87 seconds
+Number of files: 799
+Utilization of max. archive size: 0%
+------------------------------------------------------------------------------
+ Original size Compressed size Deduplicated size
+This archive: 534.24 GB 238.43 GB 237.85 GB
+All archives: 534.24 GB 238.43 GB 238.38 GB
+ Unique chunks Total chunks
+Chunk index: 200049 227228
+------------------------------------------------------------------------------
+```
+
+50% compression is not bad. borg is incremental so it will only backup differences next round.
+
+Once borg works we could run a CRON job. But we should use the sheepdog monitor to make sure backups keep going without failure going unnoticed.
+
+# Using the sheepdog
+
+=> https://github.com/pjotrp/deploy sheepdog code
+
+## Clone sheepdog
+
+=> https://github.com/pjotrp/deploy#install sheepdog install
+
+Essentially clone the repo so it shows up in ~/deploy
+
+```
+cd /home/ibackup
+git clone https://github.com/pjotrp/deploy.git
+/export/backup/scripts/tux04/backup-tux04.sh
+```
+
+## Setup redis
+
+All sheepdog messages get pushed to redis. You can run it locally or remotely.
+
+By default we use redis, but syslog and others may also be used. The advantage of redis is that it is not bound to the same host, can cross firewalls using an ssh reverse tunnel, and is easy to query.
+
+=> https://github.com/pjotrp/deploy#install sheepdog install
+
+In our case we use redis on a remote host and the results get displayed by a webserver. Also some people get E-mail updates on failure. The configuration is in
+
+```
+/home/ibackup# cat .config/sheepdog/sheepdog.conf .
+{
+ "redis": {
+ "host" : "remote-host",
+ "password": "something"
+ }
+}
+```
+
+If you see localhost with port 6377 it is probably a reverse tunnel setup:
+
+=> https://github.com/pjotrp/deploy#redis-reverse-tunnel
+
+Update the fields according to what we use. Main thing is that is the definition of the sheepdog->redis connector. If you also use sheepdog as another user you'll need to add a config.
+
+Sheepdog should show a warning when you configure redis and it is not connecting.
+
+## Scripts
+
+Typically I run the cron job from root CRON so people can find it. Still it is probably a better idea to use an ibackup CRON. In my version a script is run that also captures output:
+
+```cron root
+0 6 * * * /bin/su ibackup -c /export/backup/scripts/tux04/backup-tux04.sh >> ~/cron.log 2>&1
+```
+
+The script contains something like
+
+```bash
+#! /bin/bash
+if [ "$EUID" -eq 0 ]
+ then echo "Please do not run as root. Run as: su ibackup -c $0"
+ exit
+fi
+rundir=$(dirname "$0")
+# ---- for sheepdog
+source $rundir/sheepdog_env.sh
+cd $rundir
+sheepdog_borg.rb -t borg-tux04-sql --group ibackup -v -b /export/backup/borg/genenetwork /export/mysql/database/*
+```
+
+and the accompanying sheepdov_env.sh
+
+```
+export GEM_PATH=/home/ibackup/opt/deploy/lib/ruby/vendor_ruby
+export PATH=/home/ibackup/opt/deploy/deploy/bin:/home/wrk/opt/deploy/bin:$PATH
+```
+
+If it reports
+
+```
+/export/backup/scripts/tux04/backup-tux04.sh: line 11: /export/backup/scripts/tux04/sheepdog_env.sh: No such file or directory
+```
+
+you need to install sheepdog first.
+
+If all shows green (and takes some time) we made a backup. Check the backup with
+
+```
+ibackup@tux04:/export/backup/borg$ borg list genenetwork/
+first-backup Sat, 2025-02-08 04:39:50 [58715b883c080996ab86630b3ae3db9bedb65e6dd2e83977b72c8a9eaa257cdf]
+borg-tux04-sql-20250209-01:43-Sun Sun, 2025-02-09 01:43:23 [5e9698a032143bd6c625cdfa12ec4462f67218aa3cedc4233c176e8ffb92e16a]
+```
+and you should see the latest. The contents with all files should be visible with
+
+```
+borg list genenetwork::borg-tux04-sql-20250209-01:43-Sun
+```
+
+Make sure you not only see just a symlink.
+
+# More backups
+
+Our production server runs databases and file stores that need to be backed up too.
+
+# Drop backups
+
+Once backups work it is useful to copy them to a remote server, so when the machine stops functioning we have another chance at recovery. See
+
+=> ./backup-drops.gmi
diff --git a/topics/systems/ci-cd.gmi b/topics/systems/ci-cd.gmi
index 6aa17f2..a1ff2e3 100644
--- a/topics/systems/ci-cd.gmi
+++ b/topics/systems/ci-cd.gmi
@@ -31,7 +31,7 @@ Arun has figured out the CI part. It runs a suitably configured laminar CI servi
CD hasn't been figured out. Normally, Guix VMs and containers created by `guix system` can only access the store read-only. Since containers don't have write access to the store, you cannot `guix build' from within a container or deploy new containers from within a container. This is a problem for CD. How do you make Guix containers have write access to the store?
-Another alternative for CI/ CID were to have the quick running tests, e.g unit tests, run on each commit to branch "main". Once those are successful, the CI/CD system we choose should automatically pick the latest commit that passed the quick running tests for for further testing and deployment, maybe once an hour or so. Once the next battery of tests is passed, the CI/CD system will create a build/artifact to be deployed to staging and have the next battery of tests runs against it. If that passes, then that artifact could be deployed to production, and details on the commit and
+Another alternative for CI/ CD were to have the quick running tests, e.g unit tests, run on each commit to branch "main". Once those are successful, the CI/CD system we choose should automatically pick the latest commit that passed the quick running tests for for further testing and deployment, maybe once an hour or so. Once the next battery of tests is passed, the CI/CD system will create a build/artifact to be deployed to staging and have the next battery of tests runs against it. If that passes, then that artifact could be deployed to production, and details on the commit and
#### Possible Steps
@@ -90,3 +90,49 @@ This contains a check-list of things that need to be done:
=> /topics/systems/orchestration Orchestration
=> /issues/broken-cd Broken-cd (Resolved)
+
+## Adding a web-hook
+
+### Github hooks
+
+IIRC actions run artifacts inside github's infrastracture. We use webhooks: e.g.
+
+Update the hook at
+
+=> https://github.com/genenetwork/genenetwork3/settings/hooks
+
+=> ./screenshot-github-webhook.png
+
+To trigger CI manually, run this with the project name:
+
+```
+curl https://ci.genenetwork.org/hooks/example-gn3
+```
+
+For gemtext we have a github hook that adds a forge-project and looks like
+
+```lisp
+(define gn-gemtext-threads-project
+ (forge-project
+ (name "gn-gemtext-threads")
+ (repository "https://github.com/genenetwork/gn-gemtext-threads/")
+ (ci-jobs (list (forge-laminar-job
+ (name "gn-gemtext-threads")
+ (run (with-packages (list nss-certs openssl)
+ (with-imported-modules '((guix build utils))
+ #~(begin
+ (use-modules (guix build utils))
+
+ (setenv "LC_ALL" "en_US.UTF-8")
+ (invoke #$(file-append tissue "/bin/tissue")
+ "pull" "issues.genenetwork.org"))))))))
+ (ci-jobs-trigger 'webhook)))
+```
+
+Guix forge can be found at
+
+=> https://git.systemreboot.net/guix-forge/
+
+### git.genenetwork.org hooks
+
+TBD
diff --git a/topics/systems/screenshot-github-webhook.png b/topics/systems/screenshot-github-webhook.png
new file mode 100644
index 0000000..08feed3
--- /dev/null
+++ b/topics/systems/screenshot-github-webhook.png
Binary files differ
generated by cgit v1.2.3 (git 2.47.1) at 2025-04-16 09:11:09 +0000