about summary refs log tree commit diff
path: root/genenetwork
AgeCommit message (Collapse)Author
3 dayscontainers: write the banner to /var/log/container-banner.log as wellPjotr
The boot banner was displayed to stdout, which at boot is shepherd's own port and reaches nobody: measured, the container's /var/log/messages gains "Service container-banner has been started" but not the banner text, and `herd start container-banner` prints it only because the client is attached to the socket. So the text is now also written to a file the container owns, /var/log/container-banner.log, which the daemon's LOG verb reads back through /proc/<pid>/root (run-guix-container a18ac68). Verified on a rebuilt cgit container: the file appears four seconds after start with the full text.
4 dayscontainers: every container now says which one it isPjotr
cgit showed the shape, this applies it to the rest: bnw, genecup, gn-guile-webserver, julia-notebook, jbrowse2, slurm and gn4ai. The boot line moved into container-ssh-service, so a container opts in with three things -- one banner string, `(issue %x-banner)` and `#:banner %x-banner` -- and the helper does the rest: `Banner /etc/issue.net` in sshd_config (so the name is shown BEFORE authentication), /etc/issue.net and /etc/motd for after login, and a one-shot container-banner shepherd service that prints the same text as the first line of the boot log. cgit's own copy of that service is gone, because two copies would be two texts to keep right. slurm does not use the helper (it configures sshd itself, with password authentication for its cluster), so it gets the same four things by hand: issue, extra-content "Banner /etc/issue.net", the two banner files, and the boot service. Each text names the container, what it serves and its ports, and says that it is a container so a shell that behaves differently from the host is explained rather than mysterious: bnw-container BNW web app :8881, ssh :2222 genecup-container genecup web :4201, ssh :2223 gn-guile-webserver nginx :8091 -> :8092, ssh :2225 julia-notebook Pluto :8890, ssh :2225 jbrowse2-container JBrowse 2 :8080, ssh :2225 slurm-container slurmrestd :6820, ssh :2222 gn4ai-container GN4 API :3003, ssh :2223 cgit-container cgit :80, ssh :2224 Verified per file with guix repl -- issue set and sshd carrying the Banner directive -- and with `guix system container -d --network' for all eight, then by building cgit: its /etc/issue, /etc/motd and the compiled shepherd-container-banner in shepherd.conf all carry the text. One paren bug worth noting for whoever edits these next: my first pass at slurm closed the services list one paren early, and Guix answered with "operating-system: invalid field s" -- a record field error, nowhere near the bracket that was wrong.
4 dayscontainers: name themselves, on boot, over ssh and in the logPjotr
A store path is a hash, so /gnu/store/…-run-container cannot say which container it builds, and neither can a pid from the daemon's --list. cgit is the worked example of the three places that fix it, all fed from one Scheme string: * `(issue %cgit-banner)' in the operating-system: /etc/issue, which is what a console and nsenter show. * `#:banner %cgit-banner' to container-ssh-service: sshd_config gains `Banner /etc/issue.net', so the name appears BEFORE authentication -- you learn which container you are entering before you are in it -- and /etc/issue.net and /etc/motd are written with the same text for after login. The keyword is optional and defaults to off, so the other containers keep their current behaviour until they opt in. * a one-shot container-banner shepherd service that displays the same string at boot, so the daemon's log starts by naming the container instead of beginning with raw boot noise. Verified on the built container rather than by reading the source: the system's /etc/issue, /etc/issue.net and /etc/motd all carry the text, sshd's config for port 2224 has `extra-content: "Banner /etc/issue.net\n"', and the compiled shepherd.conf lists a shepherd-container-banner.go whose payload contains the banner string. Also documented in skills/guix-containers/SKILL.md, together with the fourth half of the same problem: name the store item itself with a GC root (guix build --root=$HOME/containers/<name>), which both pins it against a gc and gives a human a name for the hash. --root refuses to replace an existing root, so the symlink has to go first.
5 dayscgit: Move the /plain/ exemption into the Anubis policy.Pjotr
The nginx-side bypass added for raw downloads answered 403 to any user agent outside the allowlist, which also broke cgit's "plain" link for browsers, and it put bot policy in two places (an nginx location plus the Anubis policy). Serve /plain/ through Anubis like everything else and let the policy allow it: %anubis-challenge-all-policy now has a `plain-download-tools' rule (CEL, since user_agent_regex and path_regex cannot be combined) that ALLOWs curl/wget on /<repo>/plain/ when the request carries no Sec-Fetch-* header. Tools fetch raw files and cannot solve a proof-of-work challenge; browsers do send Sec-Fetch-*, so they still get the challenge -- and can solve it, so the plain link works again. A browser claiming a curl user agent is challenged too. Header names in the expression must be in canonical Go form (Sec-Fetch-Mode): the X-Forwarded-For style lookup is case-sensitive. Drops the plain-anubis-bypass? and plain-user-agents fields, the cgit-plain-location, and the now unused nginx-map serializer. Tests t23-t26 cover the policy: curl/wget are served, browsers and spoofed curl user agents are challenged (the latter two are Anubis-only, like t17/t18).
5 daysanubis: Honour the configuration other services contribute.Pjotr
fold-services only applies extensions when a service type defines `extend' -- without it the instance keeps its own value and the contributed values are dropped. anubis-service-type defined neither compose nor extend, so cgit-nginx-service-type's contribution (the Anubis target, policy and difficulty derived from the cgit configuration) never took effect: Anubis always ran with the default configuration. It went unnoticed because the default target happened to be the internal listener address. Compose the contributions and prefer a contributed configuration over the default one, and export the target accessor so the extension can set the field.
2026-10-28cgit: Test per-client budgets and /plain/ handling.Pjotr
The rate-limit tests could not catch the shared-bucket bug: they always spoke from one address, so "per client IP" and "per proxy IP" looked identical. Give each test its own X-Forwarded-For bucket (the listener trusts 127.0.0.1, see trusted-proxies) and add the regression: * t19/t20 keep covering the expensive and general limits, now with the 60r/m burst 20 and a bucket nobody else spends; * t21 exhausting one forwarded client's expensive budget must leave the next one alone; * t22 the same for the general (server-level) budget; * t23/t24 /plain/ serves curl and wget and answers 403 to anything else; * t25 /plain/ is not challenged by Anubis. t10 identifies as curl now, because /plain/ is an allowlist.
2026-10-28cgit: Rate limit per forwarded client and serve /plain/ without Anubis.Pjotr
Every limit was keyed on $binary_remote_addr, but the deployment terminates TLS on the host and forwards from 127.0.0.1 without X-Forwarded-For, so the four limits collapsed into one bucket shared by the whole internet: 5 r/s, 20 connections and 60 r/min for /log|/commit|/diff for everybody. Ordinary browsing got 429s (9 of 12 requests at one request every two seconds), and `git clone' failed on its first request. Recover the client address instead: new trusted-proxies and real-ip-header fields feed set_real_ip_from/real_ip_header/ real_ip_recursive, emitted by the new cgit-http-directives (the OS passes it as nginx extra-content; %cgit-rate-limit-http-directives is kept as the default-config value). The trusted-proxies default of 127.0.0.1 matches the host proxy, and set to '() when an instance is exposed directly so a client cannot pick its own bucket. Also for anonymous reads: * the expensive zone goes from 5r/m burst 3 to 60r/m burst 20 -- one request per second per client still stops history walks but leaves log -> commit -> diff chains alone; * /plain/ (raw file content: curl, wget, scripts) can skip the Anubis challenge (plain-anubis-bypass?), which a tool cannot solve, and can be restricted to user-agent prefixes (plain-user-agents) so it cannot be used from a browser; * internal-listen makes the Anubis-mode internal cgit address configurable, since a --network container shares the host netns and port 8081 may be taken. The container uses nginx-realip (guix-bioinformatics, gn packages web): the stock nginx is built without the real_ip module and refuses the directives with `unknown directive "set_real_ip_from"'.
2026-10-28nginx: Add real_ip and map serializers.Pjotr
set_real_ip_from / real_ip_header / real_ip_recursive let nginx recover the client address from X-Forwarded-For when it runs behind a reverse proxy, so directives keyed on $binary_remote_addr stay per-client instead of per-proxy. nginx-map emits a map block, used for the user-agent allowlists that need a variable rather than a hand-built string.
2026-10-22Update gn-uploader WSGI module pathpi coding agent
gn-uploader's `scripts` package moved to `uploader/scripts` (now `uploader.scripts`), so the gunicorn app module for the gn-uploader service is now `uploader.scripts.qcapp_wsgi:app`.
2026-10-22cgit: port 8011Pjotr Prins
2026-10-19genecup: run the Flask app unbufferedPjotr
Shepherd redirects the app's stdout to #:log-file ("/var/log/genecup.log"), so Python block-buffers it and the diagnostics that explain a failed request ("Gemini batch attempt 1/3 failed: ...", "Gemini response: ...") can trail the page by kilobytes -- the log looks empty exactly when it is needed. Set PYTHONUNBUFFERED=1 in the service environment.
2026-10-19genecup: boot check calls the model, not just the models cataloguePjotr
A key can list gemini-3-flash-preview in /v1beta/models and still not be allowed to call it (preview gating, project/quota/region limits). In that case the app's stress view prints "All related sentences (Gemini classification not available or no specific stress types found)" after three failed attempts, which reads like "nothing to classify" and hides the API's own error -- exactly the report from a second deployment where the very same key/container works elsewhere. Make gemini-check do what the app does: POST /v1beta/models/<model>:generateContent with a one-sentence classification prompt (a handful of tokens), and on failure quote the API's response body so the boot log names the reason (model not found, permission denied, quota, TLS, ...). The model is a single definition (%gemini-model, gemini-3-flash-preview) that must stay in sync with server.py's hard-coded batch classifier, and the docstring says so. Verdict is now Gemini classification available: key in PATH works, MODEL answered (HTTP 200) Gemini classification not available: MODEL call failed (HTTP 404): {...} so the container tests assert "answered (HTTP 200)".
2026-10-19genecup: expose the Gemini API key and check it at bootPjotr
The app reads its key from ~/.config/gemini/credentials of the genecup user (i.e. <datadir>/.config/gemini/credentials) and exits when the mode is not 0400; when the file is missing it warns and keeps running without LLM features. Nothing brought the host's key into the container, and nothing said whether a key that was there actually worked before a user hit the Gemini path in the UI. New genecup-configuration field 'gemini-credentials': the path where the launcher exposes the host key read-only (a secret, so never in the store). The activation copies it to <datadir>/.config/gemini/credentials with owner genecup and mode 0400 -- a bind mount would keep the host owner, which the genecup user cannot read -- and complains when it is not exposed. New one-shot shepherd service 'gemini-check' reports at boot, in the app's own wording, whether the key exists, has mode 0400, is non-empty and is *accepted by the API* (GET /v1beta/models costs no tokens: 200 usable, 400/403 rejected). Verdict goes to the boot log and /var/log/gemini-check.log. It is a program-file rather than a start gexp because use-modules inside a shepherd start value does not take effect (it compiles to an unbound-variable failure at runtime), and it sets SSL_CERT_FILE since bare curl in a minimal container fails verification with "curl: (60) ... (CAfile: none)". genecup-container.scm exposes the host key and documents the flag; it also documents the -L layouts and the trap found on a second host: a wrong -L makes (forge anubis) unresolvable, so (genenetwork services genecup) compiles to nothing and guix only reports "genecup-service-type: unbound variable". New test genenetwork/services/test/run_genecup_container_tests.sh builds with the key exposed, starts through the daemon and asserts the file owner/mode, the check verdict, GEMINI_API_KEY=set in the app log and the app serving on 4202; --expect-missing inverts it for the unkeyed container. Checkout locations are auto-detected, so it also runs on flat layouts (balg01 ~/services/genecup-service). Both modes pass here.
2026-10-19julia-notebook: drop JULIA_PKG_OFFLINE for the modern stackPjotr
The modern pluto-notebook installs its Pluto/PlutoSliderServer environment with Pkg on first run, so Pkg must be online (the nss-certs bundle already lets it reach the registry over HTTPS). A mounted/persistent depot makes subsequent runs offline.
2026-10-14julia-notebook tests: retry while the slider server restartsPjotr
PlutoSliderServer on this stack can abort and be restarted by the container's wrapper, so the server is briefly unavailable in cycles. Make the HTTP client retry connection errors (and poll) until the server answers, so the tests are robust to that.
2026-10-14julia-notebook: run Pkg offline; add the container testsPjotr
Set JULIA_PKG_OFFLINE=true for the notebook server: the packaged dependencies are already on the read-only JULIA_LOAD_PATH, and letting Pkg hit the registry crashed the server mid-run (a libcurl "curlmultiassign" error). Notebooks whose packages are not packaged now fail cleanly instead of taking the server down. Add julia_notebook_http_tests.py -- the index is served, the notebooks are listed, and the power notebook runs and is served as power.html -- and run_julia_notebook_container_tests.sh to drive the build/start/test/log-harvest cycle.
2026-10-13julia-notebook: add the Pluto notebook container and servicePjotr
genenetwork/services/julia-notebook.scm: a julia-notebook service-type running pluto-notebook's runsliderserver as an unprivileged 'julia' user, with a configurable port, notebooks directory and Julia depot (JULIA_DEPOT_PATH, a single dir or a colon-separated stack), and the nss-certs CA bundle (SSL_CERT_FILE/CURL_CA_BUNDLE) so Julia's HTTP stack can reach the package registry. genenetwork/containers/julia-notebook-container.scm: the system container around that service -- PlutoSliderServer on port 8890, ssh on 2225 -- following the other containers' conventions (container-ssh, no-op sysctl/udev, no guix service). Host mappings (the Julia depot, real notebooks) are documented as --share/--expose at build time. The copied specials/pluto.scm it supersedes is removed.
2026-10-06cgit: rate limit by client IP, serve on port 80Pjotr
Introduce a general per-client-IP rate limit on the public cgit listener: 5 requests/second with burst 30 through limit_req zone=perip, plus at most 20 concurrent connections per IP through limit_conn connperip. Both answer 429 Too Many Requests and log at warn level (limit_req_log_level/limit_conn_log_level) for fail2ban. The existing, stricter limit on the expensive endpoints (/diff/, /commit/, /log/) is kept. A bot that keeps one connection open is not caught by the connection cap, so limit_req is the one that matters most. Serve the container on plain port 80 (we only use port 80). A container shares the host network namespace from a child user namespace, so its root has no CAP_NET_BIND_SERVICE over that namespace and nginx cannot bind port 80; lower net.ipv4.ip_unprivileged_port_start in the sysctl no-op service instead. Run nginx at warn log level so the rejection lines actually reach /var/log/nginx/error.log. Add nginx serializers for the limit_conn* and limit_*_log_level directives, and a t20 regression test. Update the port-80 defaults in the test scripts.
2026-10-05gn-guile: challenge every client on aging.genenetwork.orgPjotr
Add the same Anubis front-end the other containers run to the gn-guile branding webserver: * services/anubis.scm: cgit's anubis-configuration, anubis-activation, anubis-shepherd-service, anubis-service-type and %anubis-unix-socket move here from cgit.scm, so any front-end can instantiate the service without importing the cgit module. The record keeps the challenge-all policy and difficulty-4 defaults. * services/cgit.scm: import the moved names from the shared module; no behavior change. * containers/gn-guile-webserver-container.scm: the guile webserver moves to the internal port 8092; the public port 8091 is the stock nginx-service-type proxying everything through Anubis (typed proxy headers via (genenetwork services nginx)), which challenges every client and forwards verified requests to the guile server. The system's shepherd is the self-pipe variant (shepherd-for-containers), required by the stock nginx service. Verified live: curl and browser user agents get the challenge on 8091; the guile server serves the aging site on 8092; Anubis runs the challenge-all policy at difficulty 4 with target http://127.0.0.1:8092; nginx-init completes (the patched shepherd works); ssh on 2225. All four containers rebuild.
2026-10-05anubis: challenge every client on all front-ends (cgit, BNW, GeneCup)Pjotr
Bring the challenge-all policy and the difficulty knob to every containerized front-end, via a shared module: * services/anubis.scm: new shared module -- %anubis-challenge-all-policy (challenge every client that reaches Anubis; browsers solve the proof-of-work and get a cookie, every other client stays blocked; Anubis's built-in default passes non-browser user agents through unchallenged) and anubis-command, building the full Anubis command line (bind socket, policy file, difficulty, target and any extra arguments such as -cookie-secure=false) for splicing into the shepherd gexp. * services/cgit.scm: use the shared policy and anubis-command. * services/genecup.scm: genecup-nginx-configuration gains anubis-policy (default: challenge-all) and anubis-difficulty (default 4); the bundled Anubis shepherd passes them through. * services/bnw.scm: anubis-configuration and bnw-nginx-configuration gain the same policy and difficulty fields. The GeneNetwork workflow endpoints and /bnw-data/ never reach Anubis (nginx serves them directly), so GN2's server-to-server calls keep working for non-browser clients. * test/genecup_smoke.py: BASE configurable via the environment, so the content suite can run against the internal listener of an anubis-enabled deployment. Fixes two bugs found during live testing: the policy YAML was double-escaped (the catchall user-agent regex matched literal backslash-quote sequences and never fired, letting everything through), and the first splice of anubis-command passed its elements to append as bare arguments. Verified live: GeneCup -- challenge for curl and browser user agents on 4201, internal listener serves the app, smoke suite 11/11 against the internal listener; cgit -- challenge on 8883, git clone stays direct, internal content + rate limiting (429) work, policy file and difficulty 4 logged; BNW -- challenge on 8881, bn_genenet.php and /bnw-data/ stay direct (GN2 keeps working), /version.php is challenged, policy file and difficulty 4 logged. Note: BNW's /version.php deployment-validation endpoint is now behind the challenge -- check it from a browser, the internal listener, or exempt it alongside the workflow endpoints.
2026-10-05cgit: expose the Anubis proof-of-work difficultyPjotr
anubis-configuration: new 'difficulty' field (integer, default 4, matching the Anubis binary), passed to the shepherd service via -difficulty. Each increment doubles the compute a client must burn to solve the challenge before it gets a week-long cookie; raise it when headless browsers start solving their way through the challenge-all policy (10-12 costs a real browser a few seconds once a week and bots an infeasible amount per IP). Verified live in the container (anubis-enabled, TEMP internal port 8085 on this test host): Anubis logs difficulty 4, the challenge is served, git clone stays direct. The flag was also verified standalone (-difficulty 12 accepted and logged by the binary).
2026-10-05cgit: Anubis challenges every client (challenge-all policy)Pjotr
git.genenetwork.org must not serve any client that cannot solve a challenge -- only the git clone endpoints keep working for non-browser clients: * services/cgit.scm (%anubis-challenge-all-policy): new Anubis policy document -- deny the embedded pathological-scraper and aggressive-AI-bot rules outright, then CHALLENGE everything else. Anubis's built-in default policy, by contrast, passes non-browser user agents (curl, wget, feed readers) through unchallenged -- they carry zero request weight in its threshold system. * anubis-configuration: new 'policy' field (YAML file-like, defaulting to the challenge-all policy; #f selects Anubis's built-in default). The shepherd service passes it via -policy-fname. * The git smart-HTTP endpoints (info/refs, git-upload-pack, git-receive-pack) never reach Anubis -- nginx serves them directly -- so git clone/ls-remote/push keep working for every client. Note cgit atom feeds and tarball snapshot downloads DO go through the challenge now. * test/cgit_http_tests.py (t18): non-browser user agents are challenged too; t17/t18 apply to anubis-enabled deployments. Verified live in the container (anubis-enabled, TEMP internal port 8085 since a foreign process squats 8081 on this test host): curl and browser user agents both get the challenge through the public port; git ls-remote and a full git clone work directly through the public port; the internal listener serves cgit content; rate limiting still answers 429 after the burst; Anubis logs loading the policy from the store. The policy was also verified standalone against the binary (challenge for curl/ browser/wget UAs vs. pass-through under the built-in default).
2026-10-05nginx: typed serializers for every directive; genecup joins the stock ↵Pjotr
nginx-service-type Complete the repository-wide removal of hand-built nginx directives (proxy_set_header and friends were still plain strings): * services/nginx.scm: new shared module with typed serializers -- <fastcgi-parameter> and <proxy-header> records (name/value, like the environment-variable records of the standard fcgiwrap service) serialized by fastcgi-parameter->nginx / proxy-header->nginx with proper quoting, plus fastcgi-pass, proxy-pass, nginx-unix-socket-url, unix-socket-proxy-pass, try-files, nginx-include, limit-request, limit-request-status, limit-request-zone and client-max-body-size procedures. No nginx directive is built by hand anywhere anymore; values come from configuration fields, and quoting handles empty values (httpoxy), spaces in paths and nginx variables alike. * services/cgit.scm: use the shared module for everything -- the Anubis proxy location (unix-socket-proxy-pass + proxy-header records), the rate-limiting directives (limit-request-zone / limit-request / limit-request-status from %cgit-expensive-*), the try_files fallbacks and the fastcgi_params includes. * services/genecup.scm: replace the hand-rolled nginx.conf and the custom foreground-nginx shepherd with the stock nginx-service-type receiving nginx-server-configuration records (pure reverse proxy in front of Flask; the Anubis variant's public/internal server split included). The bundled Anubis shepherd service (with the cookie-secure knob) stays; nginx accounts, activation and conf rendering come from upstream now. * containers/genecup-container.scm: instantiate nginx-service-type (client_max_body_size via the typed serializer) and run PID 1 with shepherd-for-containers. * packages.scm: new shared module with shepherd-for-containers (the signalfd->self-pipe patch), moved out of cgit-container.scm so all containers can use it. Verified live: cgit suite 18/19 (t17 Anubis challenge not applicable in the plain-variant test deployment), including git clone/ls-remote/snapshot and the rate-limit test; genecup smoke suite 11/11 with the Anubis challenge verified for browser user agents, through the records-rendered confs. bnw.scm still carries the old hand-rolled nginx.conf and is next.
2026-10-05cgit: typed fastcgi parameters and the standard fastcgi_params includePjotr
Replace the remaining hand-written fastcgi_param directive strings in the location bodies with typed configuration, as with the standard services: * services/cgit.scm (<fastcgi-parameter>): new name/value record, like the environment-variable records of the standard fcgiwrap service; fastcgi-parameter->nginx serializes it into a fastcgi_param directive, quoting string values (so the empty httpoxy-mitigation value and repository paths with spaces work, and nginx variables such as $uri keep expanding) and splicing file-like values as unquoted store paths. fastcgi-pass serializes the fastcgi_pass directive from the socket. * The parameters every FastCGI program expects (QUERY_STRING, REQUEST_METHOD, CONTENT_TYPE, REMOTE_ADDR, ...) are no longer listed by hand: they come from the standard fastcgi_params file the nginx service installs into /etc/nginx, included the standard way. * cgit-web-configuration: new 'socket' parameter replaces the hard-coded %cgit-fcgiwrap-socket constant; used by the activation (socket directory ownership), the fcgiwrap shepherd service and the fastcgi_pass directives. Verified live (anubis? #f test deployment): nginx-init completes, suite 18/19 (t17 Anubis challenge not applicable in the plain variant), including git clone/ls-remote/snapshot/about-filter through the CGI; both anubis variants pass nginx -t with the fastcgi_params include resolving against /etc/nginx.
2026-10-04cgit: use the stock nginx-service-type with nginx-server-configuration recordsPjotr
Replace the hand-rolled nginx.conf string builder and the custom foreground-nginx shepherd service with upstream machinery, as requested upstream: * services/cgit.scm: drop cgit-nginx-conf-file (string concatenation) and the nginx shepherd/activation/accounts services. The cgit sites are now nginx-server-configuration records (locations as nginx-location-configuration / nginx-named-location-configuration) contributed to the stock nginx-service-type via a service extension; the rate-limit zone travels as %cgit-rate-limit-http-directives (http-context extra-content of the OS's nginx-configuration). The fcgiwrap shepherd service stays (the stock fcgiwrap-service-type cannot pass CGIT_CONFIG or a worker count). * containers/cgit-container.scm: instantiate nginx-service-type; shepherd/container-signals: patch shepherd 1.0.9 to prefer its self-pipe signal mechanism (SHEPHERD_DISABLE_SIGNALFD) and run it as PID 1 via the essential-services shepherd-configuration. In this daemon's nested-container sandbox the signalfd wake-up never fires, so every start method that waits for a child's exit status (the stock nginx 'nginx -t' one-shot, invoke-based starts) blocks forever; the self-pipe fallback works. Tests are disabled for the patched build (the upstream suite does not survive the build sandbox); the container's own suite covers the deployment. * services/container-ssh.scm: keep /var/empty root-owned/read-only in activation -- the stock nginx-service-type gives its system user /var/empty as the home directory and on a minimal container that would break sshd's privilege separation. Verified live (anubis? #f test deployment): stock nginx shepherd starts (nginx-init + nginx), suite 18/19 (t17 Anubis challenge not applicable in the plain-variant test deployment), including t19 rate limiting through the records-rendered conf. The anubis? #t conf (public/internal server split) verified statically through the service-extension machinery and by nginx -t on the rendered file.
2026-10-04cgit: rate-limit expensive endpoints (diff/commit/log) to 5r/m per client IPPjotr
Crawlers (including ones that pass the Anubis challenge) hammer the history-walking cgit endpoints. Add nginx limit_req protection: * services/cgit.scm (cgit-expensive-location, cgit-expensive-locations): new helpers rendering the three rate-limited location blocks; %cgit-expensive-rate 5r/m, %cgit-expensive-burst 3 (nodelay), status 429, shared zone cgit-expensive (10m, keyed on binary_remote_addr). * The blocks sit before the generic @cgit fallback in both variants: the plain server and the Anubis-internal server (127.0.0.1:8081), so challenge-passing clients are limited too while the public Anubis proxy itself stays unlimited. * Export cgit-nginx-conf-file for deployment-validation tooling. * test/cgit_http_tests.py (t19): rapid-fire /demo/log must exhaust the burst (429) while the index page stays reachable; runs last so its budget consumption cannot affect the single-hit tests. Verified live on the container (anubis? #f test deployment): 4 rapid requests pass (1+burst), the 5th returns 429, single requests after a refill window pass, cheap endpoints unaffected; suite 18/19 (t17 Anubis challenge N/A in the plain-variant test deployment).
2026-10-04gn-guile: rename the container to gn-guile-webserver-containerPjotr
The name now says what it serves (the stripped branding webserver for aging.genenetwork.org) rather than just the package it derives from. Module name and container host-name updated to match.
2026-10-04gn-guile container: use the webserver shipped by gn-guilePjotr
gn-guile now ships web/webserver-aging.scm and bin/gn-guile-webserver (the stripped branding webserver for aging.genenetwork.org) in its own tree, so the container no longer embeds them: the strip phase only deletes the REST API files and the install phase copies/substitutes the shipped script. The brand-page footers also render the current copyright year (see gn-guile f2d53ea), and the pages now serialize through the shipped (web sxml) -- the htmlprag serializer rendered (doctype "html") as a literal <doctype> element, showing stray HTML in the top-left corner.
2026-10-04gn-guile: add the stripped branding webserver containerPjotr
genenetwork/containers/gn-guile-container.scm serves ONLY aging.genenetwork.org. The gn-guile-webserver package (defined in the container file; container-specific stripped variant of guix-bioinformatics' gn-guile) inherits the gn-guile source (git.genenetwork.org/gn-guile @454244b) and PHYSICALLY DELETES everything that is not the branding webserver at build time: the REST API modules (gn/db, gn/data, gn/runner, gn/cache), the API webserver (web/webserver.scm) with its config and gn-uri, the precompute scripts, the test runner and the development guix files. What ships is one guile script (bin/gn-guile-webserver, wrapped with the profile CA bundle for the GitHub markdown fetches) plus the stripped (web webserver-aging) module: /home/aging/... brand pages, /css/* and /static/images/*, everything else 404. The REST API dependencies (guile3-dbi, guile3-dbd-mysql, redis, lmdb, gemma) are gone from the closure. The web server listens on 8091 (the gn4 port), container SSH on 2225 (bnw 2222, genecup 2223, cgit 2224). Packaging notes in the file document the two non-obvious gexp traps hit on the way: embedding #$phases drags in the parent package's whole input closure even for replaced phases, and this-package-input looks up by input LABEL (package name), not variable name. Verified live: brand pages render (markdown fetched from GitHub over TLS), css/images served with correct MIME types, 404s work, all API routes disabled.
2026-10-04genecup: add the end-to-end smoke testPjotr
test/genecup_smoke.py drives the full anonymous GeneCup flow against a running container: front page, about (version), search kickoff (/progress), the /search SSE stream (read to EOF -- the generator writes nodes.json/edges.json only after the final yield, so closing the connection at data:100 kills it mid-flight), the cytoscape graph page, the table view, a /sentences page via an edgeID from the rendered data, and the static assets. 11/11 against the container. Run it with the container up: python3 genenetwork/services/test/genecup_smoke.py
2026-10-03genecup: add Flask services and a system containerPjotr
genenetwork/services/genecup.scm, modelled on the BNW services (bnw.scm): a genecup-service-type that runs the Flask app as a dedicated 'genecup' user with writable state under GENECUP_DATADIR=/var/lib/genecup, and a genecup-nginx-service-type that reverse-proxies to it -- nginx in the foreground via make-forkexec-constructor and Anubis (when enabled) bundled into the nginx shepherd service, both lessons learned in bnw.scm. The pubmed-archive field exports EDIRECT_LOCAL_ARCHIVE for a shared PubMed archive (otherwise edirect goes over the network). genenetwork/containers/genecup-container.scm: the system container for the run-guix-container daemon -- nginx on 4201, Flask on 4202, Anubis on (bundled) 8083, container SSH on 2223 (bnw uses 2222, cgit 2224), same pinned keys via container-ssh-service. Bug found while testing: the nginx user's home must not be /var/empty -- guix's account activation creates and chowns system users' homes, and sshd requires a root-owned privilege-separation directory /var/empty; with an nginx-owned /var/empty every ssh connection died before the banner. bnw.scm's accounts only get away with it through activation ordering; genecup's nginx user uses /var/run/genecup-nginx instead (commented in the service). Verified: guix system container --dry-run; container boots with nginx (4201, Anubis challenges browser user agents), Flask (4202) and sshd (2223) all serving; /var/lib/genecup populated with userspub.sqlite and user/.
2026-10-03bnw: bundle Anubis into the nginx shepherd services; cookie-secure knobPjotr
Routing the Anubis configuration through a service extension never worked: anubis-service-type has no compose/extend procedures, so fold-services DISCARDS the values contributed by extensions, and instantiate-missing-services creates a default-valued Anubis instance instead. Consequences observed: an Anubis service always ran with the DEFAULT configuration (even when anubis? was #f), and any anubis-configuration field set through the extension was silently ignored. bnw-nginx-shepherd-service now returns the nginx shepherd service PLUS, when anubis? is set, the Anubis shepherd service built directly from the bnw-nginx-configuration; the activation and profile extensions include the Anubis parts conditionally, and the service-extension to anubis-service-type is gone (so no phantom default Anubis runs when anubis? is #f). New anubis-configuration field cookie-secure? (default #t) and bnw-nginx-configuration field anubis-cookie-secure? pass -cookie-secure=false to Anubis. Anubis marks its challenge cookies "Secure" by default and browsers DROP Secure cookies over plain HTTP, so the challenge loops forever on HTTP-only test deployments (seen while debugging production); production sits behind TLS and keeps the default.
2026-10-03bnw: exempt static assets from Anubis; ^~ on /bnw-data/Pjotr
The pages load their scripts and styles from /scripts/ and /fonts/; with Anubis enabled those requests were challenged, and the challenge response is text/html -- browsers reject it with "The script ... was loaded even though its MIME type ("text/html") is not a valid JavaScript MIME type" and the network diagram page breaks (observed on production). Replace the .htm-only block with a static-asset block: ~* \.(js|css|htm|html|png|gif|jpg|svg|ico|ttf|otf|woff|woff2| eot|txt)$ served directly via try_files. The new regex locations would have overridden the /bnw-data/ prefix for the rendered-network SVG downloads (regex beats plain prefix in nginx), so the bnw-data block becomes ^~ to keep precedence. New test t19: browser UA fetches the actual script/css/image assets and requires 200 with real javascript/css/image MIME types and no challenge body. bnw suite: 19/19.
2026-10-03bnw: exempt the whole default modeling flow from AnubisPjotr
After the prepared-data page, the default flow is a chain of php pages that forward via window.open('_self'): bn_genenet.php -> executionprogress_default.php (progress bar) -> execute_bn_gom.php (structure learning runner) -> layout_svg_no.php (rendered network). The runner pages output almost nothing by design -- so when Anubis challenges a mid-chain hop, the browser hangs on a permanently blank page (observed in production on execute_bn_gom.php?My_key=...). Replace the hand-written endpoint pairs with a single %bnw-direct-php-endpoints list and bnw-nginx-direct-endpoint-blocks, which serves each workflow endpoint DIRECT under both prefixes (root with the generic php body; /sourcecodes/ with explicit SCRIPT_FILENAME, since the generic body is relative to a web root that already IS .../sourcecodes). Endpoints: bn_genenet.php, executionprogress_default.php, execute_bn_gom.php, graphviz_structure.php (external rendering endpoint), layout_svg_no.php, layout_svg_wt.php. New tests t17 (full flow browser-UA, three hops, no challenge, real SVG at the end) and t18 (graphviz_structure.php rendering endpoint). bnw suite: 18/18.
2026-10-03bnw: serve GN2 endpoints directly on the legacy /sourcecodes/ pathPjotr
GN2's bnw_page.html posts to the LEGACY path /sourcecodes/bn_genenet.php (the upstream deployment layout). The legacy redirect from b820a0d answers that POST with a 301, and browsers turn POST into GET when following redirects -- the payload is silently dropped and bn_genenet.php renders its default form ("the Loading data from GeneNetwork hang"). Exact-match locations (= /sourcecodes/bn_genenet.php and = /sourcecodes/executionprogress_default.php) win over the ^~ /sourcecodes/ redirect and serve the endpoints directly, without Anubis. The generic php body cannot be reused there: its if(!-f $document_root$fastcgi_script_name) check and SCRIPT_FILENAME are relative to the web root, which already IS .../sourcecodes, so the legacy prefix doubles up and 404s. These locations set SCRIPT_FILENAME explicitly and use fastcgi_params (fastcgi.conf would override it). New test t16: the GN2 handshake POST to /sourcecodes/bn_genenet.php with a BROWSER user agent returns the prepared-data page (200, no redirect, no challenge). bnw suite: 16/16.
2026-10-03bnw: exempt the .htm tutorial pages from AnubisPjotr
The .htm tutorial/howto pages (BNW_workflow_*.htm etc.) are fetched server-to-server by external GeneNetwork webservices. Anubis must not sit between services: its challenge needs a javascript browser, so a plain GET hangs. A location ~ \.htm$ serving the static files is added to the public block, before the Anubis proxy; the plain config is unchanged in behavior. This exposed an nginx location-precedence subtlety: the regex .htm location overrode the plain-prefix /sourcecodes/ redirect for .htm URIs (regex locations win over prefix matches), so /sourcecodes/BNW_workflow_net1.htm 404ed instead of redirecting. The legacy redirect is now a ^~ prefix location, which suppresses regex matching for everything below /sourcecodes/. New test t15: a browser UA fetching /BNW_workflow_net1.htm gets the real page, not the challenge. bnw suite: 15/15.
2026-10-03bnw: emit relative redirects (absolute_redirect off)Pjotr
The legacy /sourcecodes/ rewrite emitted an ABSOLUTE Location built from nginx's own listen port -- browsers following the redirect from https://bnw.genenetwork.org were sent to http://bnw.genenetwork.org:8881/..., which is unreachable through the TLS-terminating host proxy. Set absolute_redirect off at http level so Location is relative (/BNW_workflow_net1.htm) and the browser resolves it against the original https origin. t14 now asserts the Location header is exactly relative, which would have caught this before deployment. bnw suite: 14/14.
2026-10-03bnw: redirect legacy /sourcecodes/ URLs to the web rootPjotr
Upstream BNW was served under /sourcecodes/... and old bookmarks and search-engine links still use that layout (e.g. https://bnw.genenetwork.org/sourcecodes/BNW_workflow_net1.htm, which 404s). The Guix nginx root is already <bnw>/sourcecodes, so the prefix must be stripped: a rewrite .../sourcecodes/(.*) -> /$1 permanent 301 in both the public (Anubis) and plain server blocks. The redirect is answered directly, before the Anubis proxy. The first attempt used "return 301 /$is_args$args", which drops the path entirely (Location: /) -- rewrite with permanent is the correct tool and preserves path and query string. New test t14 checks the 301, the Location, query-string preservation and that the followed request lands on 200. bnw suite: 14/14.
2026-10-02typo - grrrPjotr Prins
2026-10-02remarksPjotr Prins
2026-10-02docs and tests: drop hardcoded factory pathsPjotr
The header comments in bnw-container.scm, cgit-container.scm, bnw-vm.scm and cgit-vm.scm used factory/code/... paths that break on other checkouts (balg01 uses ~/guix/...). All build/run examples now use $base = a directory holding the checkouts side by side, with a note to adapt to the local layout; the stale factory/... comment in run-guix-container's container files and the remaining cgit-README reference in the test runner are fixed too. The test runners rename $FACTORY to $base and make every checkout path overridable (GUIX_BIOINF, GN_MACHINES, GUIX_FORGE, RUN_GUIX_CONTAINER default to $base/code/<checkout> for the factory layout). Verified both ways on this machine: default layout and fully explicit overrides both pass the bnw suite (13/13).
2026-10-02cgit: fold cgit-README.md into the container file headerPjotr
Move the cgit-README.md content into the header comment of genenetwork/containers/cgit-container.scm and drop the separate file, like bnw-README before it. Updated to current reality on the way in: - the container does not seed a demo repository (repos are host-side, see the bind-mount section) - the SSH knobs table now documents container-ssh-service (the shared module) instead of the removed inline ssh-host-key/openssh fields - drop the stale e716b771 fingerprint example (auto-detected today) - test count updated to 18 including the Anubis tests Cross-references to cgit-README.md in container-ssh.scm, both test runners/scripts and bnw-container.scm now point at the Keys / bind-mount sections of the container file header. bnw.service style systemd instructions do not apply to cgit (there is no cgit.service). Pure comment change: guix system container resolves to the same store path (zga020ym4y3...); cgit test cycle re-run with unchanged results (Anubis + tool checks green, t01-t16 still gated on host-side /srv/git provisioning).
2026-10-02bnw: fold bnw-README into the container file headerPjotr
Move the bnw-README content into the header comment of genenetwork/containers/bnw-container.scm and drop the separate file. The content is updated to current reality on the way in: new containers/ path and module layout, the guix-forge load path needed for (forge anubis), the ~/hello-container-key location, the Anubis exemption note, and a pointer to the test suite. bnw.service stays untouched and referenced.
2026-10-02containers: move bnw-container and cgit-container to genenetwork/containersPjotr
The container OS definitions move from genenetwork/services to genenetwork/containers to keep services (reusable service-types and procedures: bnw.scm, cgit.scm, container-ssh.scm) apart from the concrete containers that instantiate them. The modules rename to (genenetwork containers bnw-container) and (genenetwork containers cgit-container); container-ssh stays in services -- it provides services, the containers consume them. Path/module references updated: both test runners, cgit-README.md and the bnw-vm.scm header comment. Pure move: guix system container resolves to the exact same store paths as before (bnw 3jiirryqwjb..., cgit zga020ym4y3...). Both containers rebuilt and re-tested: bnw suite 13/13 green; cgit Anubis tests and tool checks green (t01-t16 still need the host-side /srv/git provisioning, unchanged).
2026-10-01container-ssh: factor the container SSH setup out of the containersPjotr
bnw-container.scm and cgit-container.scm carried identical SSH code (openssh with password-less root login via hello-container-key, the pinned ECDSA host key installed into /etc/ssh, and the /var/empty + /var/run/sshd fix), differing only in the port. Move it into the new (genanetwork services container-ssh) module as container-ssh-service, a service PROCEDURE returning the list of services, parameterized by port-number (and optionally host key / authorized key). It is a procedure rather than a service-type on purpose: openssh-service-type has no extension point that merges configurations, so a wrapping service-type would have to re-implement sshd's shepherd, activation and account services. Both containers now splice (container-ssh-service #:port-number N) into their services with append. Behavior-preserving by construction and verified by content: guix system container resolves to the exact same store paths as before the refactor (bnw 3jiirryqwjb..., cgit zga020ym4y3...), and both containers re-tested green (bnw suite 13/13 including the ssh tool check; cgit ssh login, host key, /var/empty and services all verified over ssh).
2026-10-01bnw: exempt the GeneNetwork endpoints from AnubisPjotr
GN2 calls BNW server-to-server: it posts My_Gennenet to bn_genenet.php (single-POST handshake), links straight to executionprogress_default.php?My_key=... for the computed network and serves rendered networks from /bnw-data/. Anubis must not sit between GN2 and BNW, so those endpoints stay DIRECT in the public nginx block (like the git smart-HTTP exemption in cgit.scm) -- an exact-match php location for each endpoint plus the /bnw-data/ alias, with everything else still proxied through the challenge. The fastcgi statements are factored into bnw-nginx-php-fastcgi-body and bnw-nginx-php-block, shared by the internal block and the public exemptions. bnw_http_tests.py proves the exemption with a BROWSER user agent on each GN2 endpoint (any other path with that UA gets the challenge): t10 GN2 handshake POST, t11 data-load POST, t12 executionprogress_default.php, t13 /bnw-data/ 404-without-challenge. 13/13 tests pass against the rebuilt container.
2026-10-01test: split container test runners per container, add BNW suitePjotr
Each container gets its own runner (we only work on one container at a time): run_container_tests.sh stays cgit-only and the new run_bnw_container_tests.sh drives the BNW container with its bnw_http_tests.py suite (front page, static assets, php execution via the diag page, GET+POST through Anubis, browser-UA challenge, non-browser pass-through, 404 handling). Runner fixes applied to both: - auto-detect the daemon fingerprint instead of pinning a stale value (the daemon generates a random one at startup) - default the ssh client key to $HOME/hello-container-key, the same file the container definitions bake in -- the stale copy under run-guix-container/etc/ failed with "Permission denied (publickey)" - append '; true' to the log-harvest remote loop so missing log files do not fail the whole harvest cgit tests: add the Anubis tests (t17 browser UA challenged, t18 non-browser UA passes through). t18 accepts the 404 front page that cgit serves while the host-side repositories are not provisioned. t01-t16 need /srv/git mounted from the host (see cgit-README.md).
2026-10-01bnw: enable the Anubis AI-bot firewallPjotr
Mirror the cgit.scm adaptation of (forge anubis): anubis-service-type runs anubis as the nginx user on a unix socket and proxies to an internal nginx listener on 127.0.0.1:8082 (distinct from cgit's 8081 so both front-ends can coexist). bnw-nginx-service-type instantiates it when the new anubis? field is set; the conf then splits into a PUBLIC block (everything proxied through Anubis -- BNW is a pure browser application) and an INTERNAL block with the php-fpm locations. The no-anubis? config is unchanged, so the BNW VM keeps its current behavior. The shared location blocks were factored into bnw-nginx-location-blocks to avoid duplicating the php-fpm stanza between the two server blocks. bnw-container.scm enables anubis? and documents the guix-forge load path now required for (forge anubis). Verified with test/run_bnw_container_tests.sh: 9/9 HTTP tests pass (browser UA challenged, non-browser UA passes through to php, POST works through the challenge, static assets and 404s intact), nginx + php-fpm + anubis all running in the container.
2026-10-01cgit: add repoPjotr Prins
2026-10-01cgit: enable anubisPjotr Prins