diff options
Diffstat (limited to 'topics/systems/ci-cd.gmi')
| -rw-r--r-- | topics/systems/ci-cd.gmi | 130 |
1 files changed, 78 insertions, 52 deletions
diff --git a/topics/systems/ci-cd.gmi b/topics/systems/ci-cd.gmi index a1ff2e3..e20a37a 100644 --- a/topics/systems/ci-cd.gmi +++ b/topics/systems/ci-cd.gmi @@ -1,4 +1,6 @@ -# CI/ CD for genetwork projects +# CI/CD for genetwork projects + +Continuous intergration (CI) and continuous deployment (CD) are critical parts of making sure software development does not mess up our deployment(s). We need various levels of tests to be run, from unit tests to the more complicated ones like integration, performance, regression, etc tests, and of course, they cannot all be run for each and every commit, and will thus need to be staggered across the entire deployment cycle to help with quick iteration of the code. @@ -22,16 +24,86 @@ As part of the CI/CD effort, it is necessary that there is GeneNetwork is interested in doing two things on every commit (or periodically, say, once an hour/day): -* CI: run unit tests +* CI: run unit tests on git push * CD: rebuild and redeploy a container running GN3 -Arun has figured out the CI part. It runs a suitably configured laminar CI service in a Guix container created with `guix system container'. A cron job periodically triggers the laminar CI job (note: this is no longer true). +Arun has figured out the CI part. It runs a suitably configured laminar CI service in a Guix container created with `guix system container'. => https://git.systemreboot.net/guix-forge/about/ -CD hasn't been figured out. Normally, Guix VMs and containers created by `guix system` can only access the store read-only. Since containers don't have write access to the store, you cannot `guix build' from within a container or deploy new containers from within a container. This is a problem for CD. How do you make Guix containers have write access to the store? +We have the quick running tests, e.g unit tests, run on each commit to branch "main". Once those are successful, the CI/CD system we choose should automatically pick the latest commit that passed the quick running tests for for further testing and deployment. +Once the next battery of tests is passed, the CI/CD system will create a build/artifact to be deployed to staging and have the next battery of tests runs against it. If that passes, then that artifact could be deployed to production with details of the commit and deployment dependencies. + +## Adding a web-hook + +### Github hooks + +IIRC actions run artifacts inside github's infrastracture. We use webhooks: e.g. + +Update the hook at + +=> https://github.com/genenetwork/genenetwork3/settings/hooks + +A web hook basically calls an endpoint on a git push event. The webhook for genenetwork3 has recently not been called (ever? it says: This hook has never been triggered. ). The webhook for genenetwork2, however, has been called. + +=> ./screenshot-github-webhook.png + +To trigger CI manually, run this with the project name: + +``` +curl https://ci.genenetwork.org/hooks/example-gn3 +``` + +I just tested and it appeared this triggered a redeploy of gn2: + +``` +curl -XGET "https://ci.genenetwork.org/hooks/genenetwork2 +``` + +For gemtext we have a github hook that adds a forge-project and looks like + +```lisp +(define gn-gemtext-threads-project + (forge-project + (name "gn-gemtext-threads") + (repository "https://github.com/genenetwork/gn-gemtext-threads/") + (ci-jobs (list (forge-laminar-job + (name "gn-gemtext-threads") + (run (with-packages (list nss-certs openssl) + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils)) + + (setenv "LC_ALL" "en_US.UTF-8") + (invoke #$(file-append tissue "/bin/tissue") + "pull" "issues.genenetwork.org")))))))) + (ci-jobs-trigger 'webhook))) +``` + +The normal trigger is automatic, you push code in any of the two repos (three? I'll verify), GN2 and GN3 and the laminar runs the jobs and updates the code in the container restarts services, as appropriate. + +If you want to trigger the CI manually, there are webhooks available for that that can be triggered manually with something like: + +``` +curl -XGET "https://ci.genenetwork.org/hooks/genenetwork2" +``` + +for GN2. Change the part after /hooks/ for each of the different repos as follows: + +``` +GN2: /genenetwork2 +GN3: /genenetwork3 +gn-auth: /gn-auth (I need to verify this) +gn-uploader: Does not exist right now +``` + +Guix forge can be found at + +=> https://git.systemreboot.net/guix-forge/ -Another alternative for CI/ CD were to have the quick running tests, e.g unit tests, run on each commit to branch "main". Once those are successful, the CI/CD system we choose should automatically pick the latest commit that passed the quick running tests for for further testing and deployment, maybe once an hour or so. Once the next battery of tests is passed, the CI/CD system will create a build/artifact to be deployed to staging and have the next battery of tests runs against it. If that passes, then that artifact could be deployed to production, and details on the commit and +### git.genenetwork.org hooks + +TBD #### Possible Steps @@ -81,7 +153,7 @@ Below are some possible steps (and tasks) to undertake for automated deployment * Generate guix declaration for re-generating the release * Archive container image, documentation and guix declaration for possible rollback -#### Some Work Done +#### See also => /topics/systems/gn-services GN Services @@ -90,49 +162,3 @@ This contains a check-list of things that need to be done: => /topics/systems/orchestration Orchestration => /issues/broken-cd Broken-cd (Resolved) - -## Adding a web-hook - -### Github hooks - -IIRC actions run artifacts inside github's infrastracture. We use webhooks: e.g. - -Update the hook at - -=> https://github.com/genenetwork/genenetwork3/settings/hooks - -=> ./screenshot-github-webhook.png - -To trigger CI manually, run this with the project name: - -``` -curl https://ci.genenetwork.org/hooks/example-gn3 -``` - -For gemtext we have a github hook that adds a forge-project and looks like - -```lisp -(define gn-gemtext-threads-project - (forge-project - (name "gn-gemtext-threads") - (repository "https://github.com/genenetwork/gn-gemtext-threads/") - (ci-jobs (list (forge-laminar-job - (name "gn-gemtext-threads") - (run (with-packages (list nss-certs openssl) - (with-imported-modules '((guix build utils)) - #~(begin - (use-modules (guix build utils)) - - (setenv "LC_ALL" "en_US.UTF-8") - (invoke #$(file-append tissue "/bin/tissue") - "pull" "issues.genenetwork.org")))))))) - (ci-jobs-trigger 'webhook))) -``` - -Guix forge can be found at - -=> https://git.systemreboot.net/guix-forge/ - -### git.genenetwork.org hooks - -TBD |