| Age | Commit message (Collapse) | Author |
|
The nginx-side bypass added for raw downloads answered 403 to any user
agent outside the allowlist, which also broke cgit's "plain" link for
browsers, and it put bot policy in two places (an nginx location plus
the Anubis policy).
Serve /plain/ through Anubis like everything else and let the policy
allow it: %anubis-challenge-all-policy now has a `plain-download-tools'
rule (CEL, since user_agent_regex and path_regex cannot be combined)
that ALLOWs curl/wget on /<repo>/plain/ when the request carries no
Sec-Fetch-* header. Tools fetch raw files and cannot solve a
proof-of-work challenge; browsers do send Sec-Fetch-*, so they still get
the challenge -- and can solve it, so the plain link works again. A
browser claiming a curl user agent is challenged too.
Header names in the expression must be in canonical Go form
(Sec-Fetch-Mode): the X-Forwarded-For style lookup is case-sensitive.
Drops the plain-anubis-bypass? and plain-user-agents fields, the
cgit-plain-location, and the now unused nginx-map serializer. Tests
t23-t26 cover the policy: curl/wget are served, browsers and spoofed
curl user agents are challenged (the latter two are Anubis-only, like
t17/t18).
|
|
The rate-limit tests could not catch the shared-bucket bug: they always
spoke from one address, so "per client IP" and "per proxy IP" looked
identical. Give each test its own X-Forwarded-For bucket (the listener
trusts 127.0.0.1, see trusted-proxies) and add the regression:
* t19/t20 keep covering the expensive and general limits, now with the
60r/m burst 20 and a bucket nobody else spends;
* t21 exhausting one forwarded client's expensive budget must leave the
next one alone;
* t22 the same for the general (server-level) budget;
* t23/t24 /plain/ serves curl and wget and answers 403 to anything else;
* t25 /plain/ is not challenged by Anubis.
t10 identifies as curl now, because /plain/ is an allowlist.
|
|
Introduce a general per-client-IP rate limit on the public cgit
listener: 5 requests/second with burst 30 through limit_req
zone=perip, plus at most 20 concurrent connections per IP through
limit_conn connperip. Both answer 429 Too Many Requests and log at
warn level (limit_req_log_level/limit_conn_log_level) for fail2ban.
The existing, stricter limit on the expensive endpoints (/diff/,
/commit/, /log/) is kept. A bot that keeps one connection open is
not caught by the connection cap, so limit_req is the one that
matters most.
Serve the container on plain port 80 (we only use port 80). A
container shares the host network namespace from a child user
namespace, so its root has no CAP_NET_BIND_SERVICE over that
namespace and nginx cannot bind port 80; lower
net.ipv4.ip_unprivileged_port_start in the sysctl no-op service
instead. Run nginx at warn log level so the rejection lines actually
reach /var/log/nginx/error.log.
Add nginx serializers for the limit_conn* and limit_*_log_level
directives, and a t20 regression test. Update the port-80 defaults
in the test scripts.
|
|
git.genenetwork.org must not serve any client that cannot solve a
challenge -- only the git clone endpoints keep working for
non-browser clients:
* services/cgit.scm (%anubis-challenge-all-policy): new Anubis
policy document -- deny the embedded pathological-scraper and
aggressive-AI-bot rules outright, then CHALLENGE everything else.
Anubis's built-in default policy, by contrast, passes non-browser
user agents (curl, wget, feed readers) through unchallenged --
they carry zero request weight in its threshold system.
* anubis-configuration: new 'policy' field (YAML file-like,
defaulting to the challenge-all policy; #f selects Anubis's
built-in default). The shepherd service passes it via
-policy-fname.
* The git smart-HTTP endpoints (info/refs, git-upload-pack,
git-receive-pack) never reach Anubis -- nginx serves them
directly -- so git clone/ls-remote/push keep working for every
client. Note cgit atom feeds and tarball snapshot downloads DO
go through the challenge now.
* test/cgit_http_tests.py (t18): non-browser user agents are
challenged too; t17/t18 apply to anubis-enabled deployments.
Verified live in the container (anubis-enabled, TEMP internal
port 8085 since a foreign process squats 8081 on this test host):
curl and browser user agents both get the challenge through the
public port; git ls-remote and a full git clone work directly
through the public port; the internal listener serves cgit
content; rate limiting still answers 429 after the burst; Anubis
logs loading the policy from the store. The policy was also
verified standalone against the binary (challenge for curl/
browser/wget UAs vs. pass-through under the built-in default).
|
|
Crawlers (including ones that pass the Anubis challenge) hammer the
history-walking cgit endpoints. Add nginx limit_req protection:
* services/cgit.scm (cgit-expensive-location, cgit-expensive-locations):
new helpers rendering the three rate-limited location blocks;
%cgit-expensive-rate 5r/m, %cgit-expensive-burst 3 (nodelay), status
429, shared zone cgit-expensive (10m, keyed on binary_remote_addr).
* The blocks sit before the generic @cgit fallback in both variants:
the plain server and the Anubis-internal server (127.0.0.1:8081), so
challenge-passing clients are limited too while the public Anubis
proxy itself stays unlimited.
* Export cgit-nginx-conf-file for deployment-validation tooling.
* test/cgit_http_tests.py (t19): rapid-fire /demo/log must exhaust the
burst (429) while the index page stays reachable; runs last so its
budget consumption cannot affect the single-hit tests.
Verified live on the container (anubis? #f test deployment): 4 rapid
requests pass (1+burst), the 5th returns 429, single requests after a
refill window pass, cheap endpoints unaffected; suite 18/19 (t17 Anubis
challenge N/A in the plain-variant test deployment).
|
|
Move the cgit-README.md content into the header comment of
genenetwork/containers/cgit-container.scm and drop the separate file,
like bnw-README before it. Updated to current reality on the way in:
- the container does not seed a demo repository (repos are host-side,
see the bind-mount section)
- the SSH knobs table now documents container-ssh-service (the shared
module) instead of the removed inline ssh-host-key/openssh fields
- drop the stale e716b771 fingerprint example (auto-detected today)
- test count updated to 18 including the Anubis tests
Cross-references to cgit-README.md in container-ssh.scm, both test
runners/scripts and bnw-container.scm now point at the Keys /
bind-mount sections of the container file header. bnw.service style
systemd instructions do not apply to cgit (there is no cgit.service).
Pure comment change: guix system container resolves to the same store
path (zga020ym4y3...); cgit test cycle re-run with unchanged results
(Anubis + tool checks green, t01-t16 still gated on host-side
/srv/git provisioning).
|
|
Each container gets its own runner (we only work on one container at
a time): run_container_tests.sh stays cgit-only and the new
run_bnw_container_tests.sh drives the BNW container with its
bnw_http_tests.py suite (front page, static assets, php execution via
the diag page, GET+POST through Anubis, browser-UA challenge,
non-browser pass-through, 404 handling).
Runner fixes applied to both:
- auto-detect the daemon fingerprint instead of pinning a stale value
(the daemon generates a random one at startup)
- default the ssh client key to $HOME/hello-container-key, the same
file the container definitions bake in -- the stale copy under
run-guix-container/etc/ failed with "Permission denied (publickey)"
- append '; true' to the log-harvest remote loop so missing log files
do not fail the whole harvest
cgit tests: add the Anubis tests (t17 browser UA challenged, t18
non-browser UA passes through). t18 accepts the 404 front page that
cgit serves while the host-side repositories are not provisioned.
t01-t16 need /srv/git mounted from the host (see cgit-README.md).
|
|
|