about summary refs log tree commit diff
path: root/genenetwork/services/test/cgit_http_tests.py
AgeCommit message (Collapse)Author
4 dayscgit: Move the /plain/ exemption into the Anubis policy.Pjotr
The nginx-side bypass added for raw downloads answered 403 to any user agent outside the allowlist, which also broke cgit's "plain" link for browsers, and it put bot policy in two places (an nginx location plus the Anubis policy). Serve /plain/ through Anubis like everything else and let the policy allow it: %anubis-challenge-all-policy now has a `plain-download-tools' rule (CEL, since user_agent_regex and path_regex cannot be combined) that ALLOWs curl/wget on /<repo>/plain/ when the request carries no Sec-Fetch-* header. Tools fetch raw files and cannot solve a proof-of-work challenge; browsers do send Sec-Fetch-*, so they still get the challenge -- and can solve it, so the plain link works again. A browser claiming a curl user agent is challenged too. Header names in the expression must be in canonical Go form (Sec-Fetch-Mode): the X-Forwarded-For style lookup is case-sensitive. Drops the plain-anubis-bypass? and plain-user-agents fields, the cgit-plain-location, and the now unused nginx-map serializer. Tests t23-t26 cover the policy: curl/wget are served, browsers and spoofed curl user agents are challenged (the latter two are Anubis-only, like t17/t18).
2026-10-28cgit: Test per-client budgets and /plain/ handling.Pjotr
The rate-limit tests could not catch the shared-bucket bug: they always spoke from one address, so "per client IP" and "per proxy IP" looked identical. Give each test its own X-Forwarded-For bucket (the listener trusts 127.0.0.1, see trusted-proxies) and add the regression: * t19/t20 keep covering the expensive and general limits, now with the 60r/m burst 20 and a bucket nobody else spends; * t21 exhausting one forwarded client's expensive budget must leave the next one alone; * t22 the same for the general (server-level) budget; * t23/t24 /plain/ serves curl and wget and answers 403 to anything else; * t25 /plain/ is not challenged by Anubis. t10 identifies as curl now, because /plain/ is an allowlist.
2026-10-06cgit: rate limit by client IP, serve on port 80Pjotr
Introduce a general per-client-IP rate limit on the public cgit listener: 5 requests/second with burst 30 through limit_req zone=perip, plus at most 20 concurrent connections per IP through limit_conn connperip. Both answer 429 Too Many Requests and log at warn level (limit_req_log_level/limit_conn_log_level) for fail2ban. The existing, stricter limit on the expensive endpoints (/diff/, /commit/, /log/) is kept. A bot that keeps one connection open is not caught by the connection cap, so limit_req is the one that matters most. Serve the container on plain port 80 (we only use port 80). A container shares the host network namespace from a child user namespace, so its root has no CAP_NET_BIND_SERVICE over that namespace and nginx cannot bind port 80; lower net.ipv4.ip_unprivileged_port_start in the sysctl no-op service instead. Run nginx at warn log level so the rejection lines actually reach /var/log/nginx/error.log. Add nginx serializers for the limit_conn* and limit_*_log_level directives, and a t20 regression test. Update the port-80 defaults in the test scripts.
2026-10-05cgit: Anubis challenges every client (challenge-all policy)Pjotr
git.genenetwork.org must not serve any client that cannot solve a challenge -- only the git clone endpoints keep working for non-browser clients: * services/cgit.scm (%anubis-challenge-all-policy): new Anubis policy document -- deny the embedded pathological-scraper and aggressive-AI-bot rules outright, then CHALLENGE everything else. Anubis's built-in default policy, by contrast, passes non-browser user agents (curl, wget, feed readers) through unchallenged -- they carry zero request weight in its threshold system. * anubis-configuration: new 'policy' field (YAML file-like, defaulting to the challenge-all policy; #f selects Anubis's built-in default). The shepherd service passes it via -policy-fname. * The git smart-HTTP endpoints (info/refs, git-upload-pack, git-receive-pack) never reach Anubis -- nginx serves them directly -- so git clone/ls-remote/push keep working for every client. Note cgit atom feeds and tarball snapshot downloads DO go through the challenge now. * test/cgit_http_tests.py (t18): non-browser user agents are challenged too; t17/t18 apply to anubis-enabled deployments. Verified live in the container (anubis-enabled, TEMP internal port 8085 since a foreign process squats 8081 on this test host): curl and browser user agents both get the challenge through the public port; git ls-remote and a full git clone work directly through the public port; the internal listener serves cgit content; rate limiting still answers 429 after the burst; Anubis logs loading the policy from the store. The policy was also verified standalone against the binary (challenge for curl/ browser/wget UAs vs. pass-through under the built-in default).
2026-10-04cgit: rate-limit expensive endpoints (diff/commit/log) to 5r/m per client IPPjotr
Crawlers (including ones that pass the Anubis challenge) hammer the history-walking cgit endpoints. Add nginx limit_req protection: * services/cgit.scm (cgit-expensive-location, cgit-expensive-locations): new helpers rendering the three rate-limited location blocks; %cgit-expensive-rate 5r/m, %cgit-expensive-burst 3 (nodelay), status 429, shared zone cgit-expensive (10m, keyed on binary_remote_addr). * The blocks sit before the generic @cgit fallback in both variants: the plain server and the Anubis-internal server (127.0.0.1:8081), so challenge-passing clients are limited too while the public Anubis proxy itself stays unlimited. * Export cgit-nginx-conf-file for deployment-validation tooling. * test/cgit_http_tests.py (t19): rapid-fire /demo/log must exhaust the burst (429) while the index page stays reachable; runs last so its budget consumption cannot affect the single-hit tests. Verified live on the container (anubis? #f test deployment): 4 rapid requests pass (1+burst), the 5th returns 429, single requests after a refill window pass, cheap endpoints unaffected; suite 18/19 (t17 Anubis challenge N/A in the plain-variant test deployment).
2026-10-02cgit: fold cgit-README.md into the container file headerPjotr
Move the cgit-README.md content into the header comment of genenetwork/containers/cgit-container.scm and drop the separate file, like bnw-README before it. Updated to current reality on the way in: - the container does not seed a demo repository (repos are host-side, see the bind-mount section) - the SSH knobs table now documents container-ssh-service (the shared module) instead of the removed inline ssh-host-key/openssh fields - drop the stale e716b771 fingerprint example (auto-detected today) - test count updated to 18 including the Anubis tests Cross-references to cgit-README.md in container-ssh.scm, both test runners/scripts and bnw-container.scm now point at the Keys / bind-mount sections of the container file header. bnw.service style systemd instructions do not apply to cgit (there is no cgit.service). Pure comment change: guix system container resolves to the same store path (zga020ym4y3...); cgit test cycle re-run with unchanged results (Anubis + tool checks green, t01-t16 still gated on host-side /srv/git provisioning).
2026-10-01test: split container test runners per container, add BNW suitePjotr
Each container gets its own runner (we only work on one container at a time): run_container_tests.sh stays cgit-only and the new run_bnw_container_tests.sh drives the BNW container with its bnw_http_tests.py suite (front page, static assets, php execution via the diag page, GET+POST through Anubis, browser-UA challenge, non-browser pass-through, 404 handling). Runner fixes applied to both: - auto-detect the daemon fingerprint instead of pinning a stale value (the daemon generates a random one at startup) - default the ssh client key to $HOME/hello-container-key, the same file the container definitions bake in -- the stale copy under run-guix-container/etc/ failed with "Permission denied (publickey)" - append '; true' to the log-harvest remote loop so missing log files do not fail the whole harvest cgit tests: add the Anubis tests (t17 browser UA challenged, t18 non-browser UA passes through). t18 accepts the 404 front page that cgit serves while the host-side repositories are not provisioned. t01-t16 need /srv/git mounted from the host (see cgit-README.md).
2026-09-30Adding cgit as a container+vm based on Arun's guix-forge.Pjotr Prins