about summary refs log tree commit diff
path: root/genenetwork/services/cgit.scm
AgeCommit message (Collapse)Author
6 dayscgit: Move the /plain/ exemption into the Anubis policy.Pjotr
The nginx-side bypass added for raw downloads answered 403 to any user agent outside the allowlist, which also broke cgit's "plain" link for browsers, and it put bot policy in two places (an nginx location plus the Anubis policy). Serve /plain/ through Anubis like everything else and let the policy allow it: %anubis-challenge-all-policy now has a `plain-download-tools' rule (CEL, since user_agent_regex and path_regex cannot be combined) that ALLOWs curl/wget on /<repo>/plain/ when the request carries no Sec-Fetch-* header. Tools fetch raw files and cannot solve a proof-of-work challenge; browsers do send Sec-Fetch-*, so they still get the challenge -- and can solve it, so the plain link works again. A browser claiming a curl user agent is challenged too. Header names in the expression must be in canonical Go form (Sec-Fetch-Mode): the X-Forwarded-For style lookup is case-sensitive. Drops the plain-anubis-bypass? and plain-user-agents fields, the cgit-plain-location, and the now unused nginx-map serializer. Tests t23-t26 cover the policy: curl/wget are served, browsers and spoofed curl user agents are challenged (the latter two are Anubis-only, like t17/t18).
2026-10-28cgit: Rate limit per forwarded client and serve /plain/ without Anubis.Pjotr
Every limit was keyed on $binary_remote_addr, but the deployment terminates TLS on the host and forwards from 127.0.0.1 without X-Forwarded-For, so the four limits collapsed into one bucket shared by the whole internet: 5 r/s, 20 connections and 60 r/min for /log|/commit|/diff for everybody. Ordinary browsing got 429s (9 of 12 requests at one request every two seconds), and `git clone' failed on its first request. Recover the client address instead: new trusted-proxies and real-ip-header fields feed set_real_ip_from/real_ip_header/ real_ip_recursive, emitted by the new cgit-http-directives (the OS passes it as nginx extra-content; %cgit-rate-limit-http-directives is kept as the default-config value). The trusted-proxies default of 127.0.0.1 matches the host proxy, and set to '() when an instance is exposed directly so a client cannot pick its own bucket. Also for anonymous reads: * the expensive zone goes from 5r/m burst 3 to 60r/m burst 20 -- one request per second per client still stops history walks but leaves log -> commit -> diff chains alone; * /plain/ (raw file content: curl, wget, scripts) can skip the Anubis challenge (plain-anubis-bypass?), which a tool cannot solve, and can be restricted to user-agent prefixes (plain-user-agents) so it cannot be used from a browser; * internal-listen makes the Anubis-mode internal cgit address configurable, since a --network container shares the host netns and port 8081 may be taken. The container uses nginx-realip (guix-bioinformatics, gn packages web): the stock nginx is built without the real_ip module and refuses the directives with `unknown directive "set_real_ip_from"'.
2026-10-06cgit: rate limit by client IP, serve on port 80Pjotr
Introduce a general per-client-IP rate limit on the public cgit listener: 5 requests/second with burst 30 through limit_req zone=perip, plus at most 20 concurrent connections per IP through limit_conn connperip. Both answer 429 Too Many Requests and log at warn level (limit_req_log_level/limit_conn_log_level) for fail2ban. The existing, stricter limit on the expensive endpoints (/diff/, /commit/, /log/) is kept. A bot that keeps one connection open is not caught by the connection cap, so limit_req is the one that matters most. Serve the container on plain port 80 (we only use port 80). A container shares the host network namespace from a child user namespace, so its root has no CAP_NET_BIND_SERVICE over that namespace and nginx cannot bind port 80; lower net.ipv4.ip_unprivileged_port_start in the sysctl no-op service instead. Run nginx at warn log level so the rejection lines actually reach /var/log/nginx/error.log. Add nginx serializers for the limit_conn* and limit_*_log_level directives, and a t20 regression test. Update the port-80 defaults in the test scripts.
2026-10-05gn-guile: challenge every client on aging.genenetwork.orgPjotr
Add the same Anubis front-end the other containers run to the gn-guile branding webserver: * services/anubis.scm: cgit's anubis-configuration, anubis-activation, anubis-shepherd-service, anubis-service-type and %anubis-unix-socket move here from cgit.scm, so any front-end can instantiate the service without importing the cgit module. The record keeps the challenge-all policy and difficulty-4 defaults. * services/cgit.scm: import the moved names from the shared module; no behavior change. * containers/gn-guile-webserver-container.scm: the guile webserver moves to the internal port 8092; the public port 8091 is the stock nginx-service-type proxying everything through Anubis (typed proxy headers via (genenetwork services nginx)), which challenges every client and forwards verified requests to the guile server. The system's shepherd is the self-pipe variant (shepherd-for-containers), required by the stock nginx service. Verified live: curl and browser user agents get the challenge on 8091; the guile server serves the aging site on 8092; Anubis runs the challenge-all policy at difficulty 4 with target http://127.0.0.1:8092; nginx-init completes (the patched shepherd works); ssh on 2225. All four containers rebuild.
2026-10-05anubis: challenge every client on all front-ends (cgit, BNW, GeneCup)Pjotr
Bring the challenge-all policy and the difficulty knob to every containerized front-end, via a shared module: * services/anubis.scm: new shared module -- %anubis-challenge-all-policy (challenge every client that reaches Anubis; browsers solve the proof-of-work and get a cookie, every other client stays blocked; Anubis's built-in default passes non-browser user agents through unchallenged) and anubis-command, building the full Anubis command line (bind socket, policy file, difficulty, target and any extra arguments such as -cookie-secure=false) for splicing into the shepherd gexp. * services/cgit.scm: use the shared policy and anubis-command. * services/genecup.scm: genecup-nginx-configuration gains anubis-policy (default: challenge-all) and anubis-difficulty (default 4); the bundled Anubis shepherd passes them through. * services/bnw.scm: anubis-configuration and bnw-nginx-configuration gain the same policy and difficulty fields. The GeneNetwork workflow endpoints and /bnw-data/ never reach Anubis (nginx serves them directly), so GN2's server-to-server calls keep working for non-browser clients. * test/genecup_smoke.py: BASE configurable via the environment, so the content suite can run against the internal listener of an anubis-enabled deployment. Fixes two bugs found during live testing: the policy YAML was double-escaped (the catchall user-agent regex matched literal backslash-quote sequences and never fired, letting everything through), and the first splice of anubis-command passed its elements to append as bare arguments. Verified live: GeneCup -- challenge for curl and browser user agents on 4201, internal listener serves the app, smoke suite 11/11 against the internal listener; cgit -- challenge on 8883, git clone stays direct, internal content + rate limiting (429) work, policy file and difficulty 4 logged; BNW -- challenge on 8881, bn_genenet.php and /bnw-data/ stay direct (GN2 keeps working), /version.php is challenged, policy file and difficulty 4 logged. Note: BNW's /version.php deployment-validation endpoint is now behind the challenge -- check it from a browser, the internal listener, or exempt it alongside the workflow endpoints.
2026-10-05cgit: expose the Anubis proof-of-work difficultyPjotr
anubis-configuration: new 'difficulty' field (integer, default 4, matching the Anubis binary), passed to the shepherd service via -difficulty. Each increment doubles the compute a client must burn to solve the challenge before it gets a week-long cookie; raise it when headless browsers start solving their way through the challenge-all policy (10-12 costs a real browser a few seconds once a week and bots an infeasible amount per IP). Verified live in the container (anubis-enabled, TEMP internal port 8085 on this test host): Anubis logs difficulty 4, the challenge is served, git clone stays direct. The flag was also verified standalone (-difficulty 12 accepted and logged by the binary).
2026-10-05cgit: Anubis challenges every client (challenge-all policy)Pjotr
git.genenetwork.org must not serve any client that cannot solve a challenge -- only the git clone endpoints keep working for non-browser clients: * services/cgit.scm (%anubis-challenge-all-policy): new Anubis policy document -- deny the embedded pathological-scraper and aggressive-AI-bot rules outright, then CHALLENGE everything else. Anubis's built-in default policy, by contrast, passes non-browser user agents (curl, wget, feed readers) through unchallenged -- they carry zero request weight in its threshold system. * anubis-configuration: new 'policy' field (YAML file-like, defaulting to the challenge-all policy; #f selects Anubis's built-in default). The shepherd service passes it via -policy-fname. * The git smart-HTTP endpoints (info/refs, git-upload-pack, git-receive-pack) never reach Anubis -- nginx serves them directly -- so git clone/ls-remote/push keep working for every client. Note cgit atom feeds and tarball snapshot downloads DO go through the challenge now. * test/cgit_http_tests.py (t18): non-browser user agents are challenged too; t17/t18 apply to anubis-enabled deployments. Verified live in the container (anubis-enabled, TEMP internal port 8085 since a foreign process squats 8081 on this test host): curl and browser user agents both get the challenge through the public port; git ls-remote and a full git clone work directly through the public port; the internal listener serves cgit content; rate limiting still answers 429 after the burst; Anubis logs loading the policy from the store. The policy was also verified standalone against the binary (challenge for curl/ browser/wget UAs vs. pass-through under the built-in default).
2026-10-05nginx: typed serializers for every directive; genecup joins the stock ↵Pjotr
nginx-service-type Complete the repository-wide removal of hand-built nginx directives (proxy_set_header and friends were still plain strings): * services/nginx.scm: new shared module with typed serializers -- <fastcgi-parameter> and <proxy-header> records (name/value, like the environment-variable records of the standard fcgiwrap service) serialized by fastcgi-parameter->nginx / proxy-header->nginx with proper quoting, plus fastcgi-pass, proxy-pass, nginx-unix-socket-url, unix-socket-proxy-pass, try-files, nginx-include, limit-request, limit-request-status, limit-request-zone and client-max-body-size procedures. No nginx directive is built by hand anywhere anymore; values come from configuration fields, and quoting handles empty values (httpoxy), spaces in paths and nginx variables alike. * services/cgit.scm: use the shared module for everything -- the Anubis proxy location (unix-socket-proxy-pass + proxy-header records), the rate-limiting directives (limit-request-zone / limit-request / limit-request-status from %cgit-expensive-*), the try_files fallbacks and the fastcgi_params includes. * services/genecup.scm: replace the hand-rolled nginx.conf and the custom foreground-nginx shepherd with the stock nginx-service-type receiving nginx-server-configuration records (pure reverse proxy in front of Flask; the Anubis variant's public/internal server split included). The bundled Anubis shepherd service (with the cookie-secure knob) stays; nginx accounts, activation and conf rendering come from upstream now. * containers/genecup-container.scm: instantiate nginx-service-type (client_max_body_size via the typed serializer) and run PID 1 with shepherd-for-containers. * packages.scm: new shared module with shepherd-for-containers (the signalfd->self-pipe patch), moved out of cgit-container.scm so all containers can use it. Verified live: cgit suite 18/19 (t17 Anubis challenge not applicable in the plain-variant test deployment), including git clone/ls-remote/snapshot and the rate-limit test; genecup smoke suite 11/11 with the Anubis challenge verified for browser user agents, through the records-rendered confs. bnw.scm still carries the old hand-rolled nginx.conf and is next.
2026-10-05cgit: typed fastcgi parameters and the standard fastcgi_params includePjotr
Replace the remaining hand-written fastcgi_param directive strings in the location bodies with typed configuration, as with the standard services: * services/cgit.scm (<fastcgi-parameter>): new name/value record, like the environment-variable records of the standard fcgiwrap service; fastcgi-parameter->nginx serializes it into a fastcgi_param directive, quoting string values (so the empty httpoxy-mitigation value and repository paths with spaces work, and nginx variables such as $uri keep expanding) and splicing file-like values as unquoted store paths. fastcgi-pass serializes the fastcgi_pass directive from the socket. * The parameters every FastCGI program expects (QUERY_STRING, REQUEST_METHOD, CONTENT_TYPE, REMOTE_ADDR, ...) are no longer listed by hand: they come from the standard fastcgi_params file the nginx service installs into /etc/nginx, included the standard way. * cgit-web-configuration: new 'socket' parameter replaces the hard-coded %cgit-fcgiwrap-socket constant; used by the activation (socket directory ownership), the fcgiwrap shepherd service and the fastcgi_pass directives. Verified live (anubis? #f test deployment): nginx-init completes, suite 18/19 (t17 Anubis challenge not applicable in the plain variant), including git clone/ls-remote/snapshot/about-filter through the CGI; both anubis variants pass nginx -t with the fastcgi_params include resolving against /etc/nginx.
2026-10-04cgit: use the stock nginx-service-type with nginx-server-configuration recordsPjotr
Replace the hand-rolled nginx.conf string builder and the custom foreground-nginx shepherd service with upstream machinery, as requested upstream: * services/cgit.scm: drop cgit-nginx-conf-file (string concatenation) and the nginx shepherd/activation/accounts services. The cgit sites are now nginx-server-configuration records (locations as nginx-location-configuration / nginx-named-location-configuration) contributed to the stock nginx-service-type via a service extension; the rate-limit zone travels as %cgit-rate-limit-http-directives (http-context extra-content of the OS's nginx-configuration). The fcgiwrap shepherd service stays (the stock fcgiwrap-service-type cannot pass CGIT_CONFIG or a worker count). * containers/cgit-container.scm: instantiate nginx-service-type; shepherd/container-signals: patch shepherd 1.0.9 to prefer its self-pipe signal mechanism (SHEPHERD_DISABLE_SIGNALFD) and run it as PID 1 via the essential-services shepherd-configuration. In this daemon's nested-container sandbox the signalfd wake-up never fires, so every start method that waits for a child's exit status (the stock nginx 'nginx -t' one-shot, invoke-based starts) blocks forever; the self-pipe fallback works. Tests are disabled for the patched build (the upstream suite does not survive the build sandbox); the container's own suite covers the deployment. * services/container-ssh.scm: keep /var/empty root-owned/read-only in activation -- the stock nginx-service-type gives its system user /var/empty as the home directory and on a minimal container that would break sshd's privilege separation. Verified live (anubis? #f test deployment): stock nginx shepherd starts (nginx-init + nginx), suite 18/19 (t17 Anubis challenge not applicable in the plain-variant test deployment), including t19 rate limiting through the records-rendered conf. The anubis? #t conf (public/internal server split) verified statically through the service-extension machinery and by nginx -t on the rendered file.
2026-10-04cgit: rate-limit expensive endpoints (diff/commit/log) to 5r/m per client IPPjotr
Crawlers (including ones that pass the Anubis challenge) hammer the history-walking cgit endpoints. Add nginx limit_req protection: * services/cgit.scm (cgit-expensive-location, cgit-expensive-locations): new helpers rendering the three rate-limited location blocks; %cgit-expensive-rate 5r/m, %cgit-expensive-burst 3 (nodelay), status 429, shared zone cgit-expensive (10m, keyed on binary_remote_addr). * The blocks sit before the generic @cgit fallback in both variants: the plain server and the Anubis-internal server (127.0.0.1:8081), so challenge-passing clients are limited too while the public Anubis proxy itself stays unlimited. * Export cgit-nginx-conf-file for deployment-validation tooling. * test/cgit_http_tests.py (t19): rapid-fire /demo/log must exhaust the burst (429) while the index page stays reachable; runs last so its budget consumption cannot affect the single-hit tests. Verified live on the container (anubis? #f test deployment): 4 rapid requests pass (1+burst), the 5th returns 429, single requests after a refill window pass, cheap endpoints unaffected; suite 18/19 (t17 Anubis challenge N/A in the plain-variant test deployment).
2026-09-30Maintain cgit root pathPjotr Prins
2026-09-30Adding cgit as a container+vm based on Arun's guix-forge.Pjotr Prins