| Age | Commit message (Collapse) | Author |
|
The nginx-side bypass added for raw downloads answered 403 to any user
agent outside the allowlist, which also broke cgit's "plain" link for
browsers, and it put bot policy in two places (an nginx location plus
the Anubis policy).
Serve /plain/ through Anubis like everything else and let the policy
allow it: %anubis-challenge-all-policy now has a `plain-download-tools'
rule (CEL, since user_agent_regex and path_regex cannot be combined)
that ALLOWs curl/wget on /<repo>/plain/ when the request carries no
Sec-Fetch-* header. Tools fetch raw files and cannot solve a
proof-of-work challenge; browsers do send Sec-Fetch-*, so they still get
the challenge -- and can solve it, so the plain link works again. A
browser claiming a curl user agent is challenged too.
Header names in the expression must be in canonical Go form
(Sec-Fetch-Mode): the X-Forwarded-For style lookup is case-sensitive.
Drops the plain-anubis-bypass? and plain-user-agents fields, the
cgit-plain-location, and the now unused nginx-map serializer. Tests
t23-t26 cover the policy: curl/wget are served, browsers and spoofed
curl user agents are challenged (the latter two are Anubis-only, like
t17/t18).
|
|
The rate-limit tests could not catch the shared-bucket bug: they always
spoke from one address, so "per client IP" and "per proxy IP" looked
identical. Give each test its own X-Forwarded-For bucket (the listener
trusts 127.0.0.1, see trusted-proxies) and add the regression:
* t19/t20 keep covering the expensive and general limits, now with the
60r/m burst 20 and a bucket nobody else spends;
* t21 exhausting one forwarded client's expensive budget must leave the
next one alone;
* t22 the same for the general (server-level) budget;
* t23/t24 /plain/ serves curl and wget and answers 403 to anything else;
* t25 /plain/ is not challenged by Anubis.
t10 identifies as curl now, because /plain/ is an allowlist.
|
|
A key can list gemini-3-flash-preview in /v1beta/models and still not be
allowed to call it (preview gating, project/quota/region limits). In
that case the app's stress view prints "All related sentences (Gemini
classification not available or no specific stress types found)" after
three failed attempts, which reads like "nothing to classify" and hides
the API's own error -- exactly the report from a second deployment
where the very same key/container works elsewhere.
Make gemini-check do what the app does: POST
/v1beta/models/<model>:generateContent with a one-sentence
classification prompt (a handful of tokens), and on failure quote the
API's response body so the boot log names the reason (model not found,
permission denied, quota, TLS, ...). The model is a single definition
(%gemini-model, gemini-3-flash-preview) that must stay in sync with
server.py's hard-coded batch classifier, and the docstring says so.
Verdict is now
Gemini classification available: key in PATH works, MODEL answered (HTTP 200)
Gemini classification not available: MODEL call failed (HTTP 404): {...}
so the container tests assert "answered (HTTP 200)".
|
|
The app reads its key from ~/.config/gemini/credentials of the genecup
user (i.e. <datadir>/.config/gemini/credentials) and exits when the
mode is not 0400; when the file is missing it warns and keeps running
without LLM features. Nothing brought the host's key into the
container, and nothing said whether a key that was there actually
worked before a user hit the Gemini path in the UI.
New genecup-configuration field 'gemini-credentials': the path where
the launcher exposes the host key read-only (a secret, so never in the
store). The activation copies it to <datadir>/.config/gemini/credentials
with owner genecup and mode 0400 -- a bind mount would keep the host
owner, which the genecup user cannot read -- and complains when it is
not exposed.
New one-shot shepherd service 'gemini-check' reports at boot, in the
app's own wording, whether the key exists, has mode 0400, is non-empty
and is *accepted by the API* (GET /v1beta/models costs no tokens: 200
usable, 400/403 rejected). Verdict goes to the boot log and
/var/log/gemini-check.log. It is a program-file rather than a start
gexp because use-modules inside a shepherd start value does not take
effect (it compiles to an unbound-variable failure at runtime), and it
sets SSL_CERT_FILE since bare curl in a minimal container fails
verification with "curl: (60) ... (CAfile: none)".
genecup-container.scm exposes the host key and documents the flag; it
also documents the -L layouts and the trap found on a second host: a
wrong -L makes (forge anubis) unresolvable, so (genenetwork services
genecup) compiles to nothing and guix only reports
"genecup-service-type: unbound variable".
New test genenetwork/services/test/run_genecup_container_tests.sh
builds with the key exposed, starts through the daemon and asserts the
file owner/mode, the check verdict, GEMINI_API_KEY=set in the app log
and the app serving on 4202; --expect-missing inverts it for the
unkeyed container. Checkout locations are auto-detected, so it also
runs on flat layouts (balg01 ~/services/genecup-service). Both modes
pass here.
|
|
PlutoSliderServer on this stack can abort and be restarted by the
container's wrapper, so the server is briefly unavailable in cycles.
Make the HTTP client retry connection errors (and poll) until the
server answers, so the tests are robust to that.
|
|
Set JULIA_PKG_OFFLINE=true for the notebook server: the packaged
dependencies are already on the read-only JULIA_LOAD_PATH, and letting
Pkg hit the registry crashed the server mid-run (a libcurl
"curlmultiassign" error). Notebooks whose packages are not packaged now
fail cleanly instead of taking the server down.
Add julia_notebook_http_tests.py -- the index is served, the notebooks
are listed, and the power notebook runs and is served as power.html --
and run_julia_notebook_container_tests.sh to drive the
build/start/test/log-harvest cycle.
|
|
Introduce a general per-client-IP rate limit on the public cgit
listener: 5 requests/second with burst 30 through limit_req
zone=perip, plus at most 20 concurrent connections per IP through
limit_conn connperip. Both answer 429 Too Many Requests and log at
warn level (limit_req_log_level/limit_conn_log_level) for fail2ban.
The existing, stricter limit on the expensive endpoints (/diff/,
/commit/, /log/) is kept. A bot that keeps one connection open is
not caught by the connection cap, so limit_req is the one that
matters most.
Serve the container on plain port 80 (we only use port 80). A
container shares the host network namespace from a child user
namespace, so its root has no CAP_NET_BIND_SERVICE over that
namespace and nginx cannot bind port 80; lower
net.ipv4.ip_unprivileged_port_start in the sysctl no-op service
instead. Run nginx at warn log level so the rejection lines actually
reach /var/log/nginx/error.log.
Add nginx serializers for the limit_conn* and limit_*_log_level
directives, and a t20 regression test. Update the port-80 defaults
in the test scripts.
|
|
Bring the challenge-all policy and the difficulty knob to every
containerized front-end, via a shared module:
* services/anubis.scm: new shared module -- %anubis-challenge-all-policy
(challenge every client that reaches Anubis; browsers solve the
proof-of-work and get a cookie, every other client stays blocked;
Anubis's built-in default passes non-browser user agents through
unchallenged) and anubis-command, building the full Anubis
command line (bind socket, policy file, difficulty, target and
any extra arguments such as -cookie-secure=false) for splicing
into the shepherd gexp.
* services/cgit.scm: use the shared policy and anubis-command.
* services/genecup.scm: genecup-nginx-configuration gains
anubis-policy (default: challenge-all) and anubis-difficulty
(default 4); the bundled Anubis shepherd passes them through.
* services/bnw.scm: anubis-configuration and bnw-nginx-configuration
gain the same policy and difficulty fields. The GeneNetwork
workflow endpoints and /bnw-data/ never reach Anubis (nginx
serves them directly), so GN2's server-to-server calls keep
working for non-browser clients.
* test/genecup_smoke.py: BASE configurable via the environment,
so the content suite can run against the internal listener of an
anubis-enabled deployment.
Fixes two bugs found during live testing: the policy YAML was
double-escaped (the catchall user-agent regex matched literal
backslash-quote sequences and never fired, letting everything
through), and the first splice of anubis-command passed its
elements to append as bare arguments.
Verified live: GeneCup -- challenge for curl and browser user
agents on 4201, internal listener serves the app, smoke suite
11/11 against the internal listener; cgit -- challenge on 8883,
git clone stays direct, internal content + rate limiting (429)
work, policy file and difficulty 4 logged; BNW -- challenge on
8881, bn_genenet.php and /bnw-data/ stay direct (GN2 keeps
working), /version.php is challenged, policy file and difficulty 4
logged. Note: BNW's /version.php deployment-validation endpoint
is now behind the challenge -- check it from a browser, the
internal listener, or exempt it alongside the workflow endpoints.
|
|
git.genenetwork.org must not serve any client that cannot solve a
challenge -- only the git clone endpoints keep working for
non-browser clients:
* services/cgit.scm (%anubis-challenge-all-policy): new Anubis
policy document -- deny the embedded pathological-scraper and
aggressive-AI-bot rules outright, then CHALLENGE everything else.
Anubis's built-in default policy, by contrast, passes non-browser
user agents (curl, wget, feed readers) through unchallenged --
they carry zero request weight in its threshold system.
* anubis-configuration: new 'policy' field (YAML file-like,
defaulting to the challenge-all policy; #f selects Anubis's
built-in default). The shepherd service passes it via
-policy-fname.
* The git smart-HTTP endpoints (info/refs, git-upload-pack,
git-receive-pack) never reach Anubis -- nginx serves them
directly -- so git clone/ls-remote/push keep working for every
client. Note cgit atom feeds and tarball snapshot downloads DO
go through the challenge now.
* test/cgit_http_tests.py (t18): non-browser user agents are
challenged too; t17/t18 apply to anubis-enabled deployments.
Verified live in the container (anubis-enabled, TEMP internal
port 8085 since a foreign process squats 8081 on this test host):
curl and browser user agents both get the challenge through the
public port; git ls-remote and a full git clone work directly
through the public port; the internal listener serves cgit
content; rate limiting still answers 429 after the burst; Anubis
logs loading the policy from the store. The policy was also
verified standalone against the binary (challenge for curl/
browser/wget UAs vs. pass-through under the built-in default).
|
|
Crawlers (including ones that pass the Anubis challenge) hammer the
history-walking cgit endpoints. Add nginx limit_req protection:
* services/cgit.scm (cgit-expensive-location, cgit-expensive-locations):
new helpers rendering the three rate-limited location blocks;
%cgit-expensive-rate 5r/m, %cgit-expensive-burst 3 (nodelay), status
429, shared zone cgit-expensive (10m, keyed on binary_remote_addr).
* The blocks sit before the generic @cgit fallback in both variants:
the plain server and the Anubis-internal server (127.0.0.1:8081), so
challenge-passing clients are limited too while the public Anubis
proxy itself stays unlimited.
* Export cgit-nginx-conf-file for deployment-validation tooling.
* test/cgit_http_tests.py (t19): rapid-fire /demo/log must exhaust the
burst (429) while the index page stays reachable; runs last so its
budget consumption cannot affect the single-hit tests.
Verified live on the container (anubis? #f test deployment): 4 rapid
requests pass (1+burst), the 5th returns 429, single requests after a
refill window pass, cheap endpoints unaffected; suite 18/19 (t17 Anubis
challenge N/A in the plain-variant test deployment).
|
|
test/genecup_smoke.py drives the full anonymous GeneCup flow against
a running container: front page, about (version), search kickoff
(/progress), the /search SSE stream (read to EOF -- the generator
writes nodes.json/edges.json only after the final yield, so closing
the connection at data:100 kills it mid-flight), the cytoscape graph
page, the table view, a /sentences page via an edgeID from the
rendered data, and the static assets. 11/11 against the container.
Run it with the container up:
python3 genenetwork/services/test/genecup_smoke.py
|
|
The pages load their scripts and styles from /scripts/ and /fonts/;
with Anubis enabled those requests were challenged, and the challenge
response is text/html -- browsers reject it with "The script ... was
loaded even though its MIME type ("text/html") is not a valid
JavaScript MIME type" and the network diagram page breaks (observed
on production). Replace the .htm-only block with a static-asset
block: ~* \.(js|css|htm|html|png|gif|jpg|svg|ico|ttf|otf|woff|woff2|
eot|txt)$ served directly via try_files.
The new regex locations would have overridden the /bnw-data/ prefix
for the rendered-network SVG downloads (regex beats plain prefix in
nginx), so the bnw-data block becomes ^~ to keep precedence.
New test t19: browser UA fetches the actual script/css/image assets
and requires 200 with real javascript/css/image MIME types and no
challenge body. bnw suite: 19/19.
|
|
After the prepared-data page, the default flow is a chain of php
pages that forward via window.open('_self'): bn_genenet.php ->
executionprogress_default.php (progress bar) -> execute_bn_gom.php
(structure learning runner) -> layout_svg_no.php (rendered network).
The runner pages output almost nothing by design -- so when Anubis
challenges a mid-chain hop, the browser hangs on a permanently blank
page (observed in production on execute_bn_gom.php?My_key=...).
Replace the hand-written endpoint pairs with a single
%bnw-direct-php-endpoints list and bnw-nginx-direct-endpoint-blocks,
which serves each workflow endpoint DIRECT under both prefixes (root
with the generic php body; /sourcecodes/ with explicit
SCRIPT_FILENAME, since the generic body is relative to a web root
that already IS .../sourcecodes). Endpoints: bn_genenet.php,
executionprogress_default.php, execute_bn_gom.php,
graphviz_structure.php (external rendering endpoint),
layout_svg_no.php, layout_svg_wt.php.
New tests t17 (full flow browser-UA, three hops, no challenge, real
SVG at the end) and t18 (graphviz_structure.php rendering endpoint).
bnw suite: 18/18.
|
|
GN2's bnw_page.html posts to the LEGACY path
/sourcecodes/bn_genenet.php (the upstream deployment layout). The
legacy redirect from b820a0d answers that POST with a 301, and
browsers turn POST into GET when following redirects -- the payload
is silently dropped and bn_genenet.php renders its default form ("the
Loading data from GeneNetwork hang").
Exact-match locations (= /sourcecodes/bn_genenet.php and
= /sourcecodes/executionprogress_default.php) win over the
^~ /sourcecodes/ redirect and serve the endpoints directly, without
Anubis. The generic php body cannot be reused there: its
if(!-f $document_root$fastcgi_script_name) check and SCRIPT_FILENAME
are relative to the web root, which already IS .../sourcecodes, so
the legacy prefix doubles up and 404s. These locations set
SCRIPT_FILENAME explicitly and use fastcgi_params (fastcgi.conf would
override it).
New test t16: the GN2 handshake POST to /sourcecodes/bn_genenet.php
with a BROWSER user agent returns the prepared-data page (200, no
redirect, no challenge). bnw suite: 16/16.
|
|
The .htm tutorial/howto pages (BNW_workflow_*.htm etc.) are fetched
server-to-server by external GeneNetwork webservices. Anubis must not
sit between services: its challenge needs a javascript browser, so a
plain GET hangs. A location ~ \.htm$ serving the static files is
added to the public block, before the Anubis proxy; the plain config
is unchanged in behavior.
This exposed an nginx location-precedence subtlety: the regex .htm
location overrode the plain-prefix /sourcecodes/ redirect for .htm
URIs (regex locations win over prefix matches), so
/sourcecodes/BNW_workflow_net1.htm 404ed instead of redirecting. The
legacy redirect is now a ^~ prefix location, which suppresses regex
matching for everything below /sourcecodes/.
New test t15: a browser UA fetching /BNW_workflow_net1.htm gets the
real page, not the challenge. bnw suite: 15/15.
|
|
The legacy /sourcecodes/ rewrite emitted an ABSOLUTE Location built
from nginx's own listen port -- browsers following the redirect from
https://bnw.genenetwork.org were sent to http://bnw.genenetwork.org:8881/...,
which is unreachable through the TLS-terminating host proxy. Set
absolute_redirect off at http level so Location is relative
(/BNW_workflow_net1.htm) and the browser resolves it against the
original https origin.
t14 now asserts the Location header is exactly relative, which would
have caught this before deployment. bnw suite: 14/14.
|
|
Upstream BNW was served under /sourcecodes/... and old bookmarks and
search-engine links still use that layout (e.g.
https://bnw.genenetwork.org/sourcecodes/BNW_workflow_net1.htm, which
404s). The Guix nginx root is already <bnw>/sourcecodes, so the
prefix must be stripped: a rewrite .../sourcecodes/(.*) -> /$1
permanent 301 in both the public (Anubis) and plain server blocks.
The redirect is answered directly, before the Anubis proxy.
The first attempt used "return 301 /$is_args$args", which drops the
path entirely (Location: /) -- rewrite with permanent is the correct
tool and preserves path and query string.
New test t14 checks the 301, the Location, query-string preservation
and that the followed request lands on 200. bnw suite: 14/14.
|
|
The header comments in bnw-container.scm, cgit-container.scm,
bnw-vm.scm and cgit-vm.scm used factory/code/... paths that break on
other checkouts (balg01 uses ~/guix/...). All build/run examples now
use $base = a directory holding the checkouts side by side, with a
note to adapt to the local layout; the stale factory/... comment in
run-guix-container's container files and the remaining cgit-README
reference in the test runner are fixed too.
The test runners rename $FACTORY to $base and make every checkout
path overridable (GUIX_BIOINF, GN_MACHINES, GUIX_FORGE,
RUN_GUIX_CONTAINER default to $base/code/<checkout> for the factory
layout). Verified both ways on this machine: default layout and
fully explicit overrides both pass the bnw suite (13/13).
|
|
Move the cgit-README.md content into the header comment of
genenetwork/containers/cgit-container.scm and drop the separate file,
like bnw-README before it. Updated to current reality on the way in:
- the container does not seed a demo repository (repos are host-side,
see the bind-mount section)
- the SSH knobs table now documents container-ssh-service (the shared
module) instead of the removed inline ssh-host-key/openssh fields
- drop the stale e716b771 fingerprint example (auto-detected today)
- test count updated to 18 including the Anubis tests
Cross-references to cgit-README.md in container-ssh.scm, both test
runners/scripts and bnw-container.scm now point at the Keys /
bind-mount sections of the container file header. bnw.service style
systemd instructions do not apply to cgit (there is no cgit.service).
Pure comment change: guix system container resolves to the same store
path (zga020ym4y3...); cgit test cycle re-run with unchanged results
(Anubis + tool checks green, t01-t16 still gated on host-side
/srv/git provisioning).
|
|
The container OS definitions move from genenetwork/services to
genenetwork/containers to keep services (reusable service-types and
procedures: bnw.scm, cgit.scm, container-ssh.scm) apart from the
concrete containers that instantiate them. The modules rename to
(genenetwork containers bnw-container) and (genenetwork containers
cgit-container); container-ssh stays in services -- it provides
services, the containers consume them.
Path/module references updated: both test runners, cgit-README.md and
the bnw-vm.scm header comment.
Pure move: guix system container resolves to the exact same store
paths as before (bnw 3jiirryqwjb..., cgit zga020ym4y3...). Both
containers rebuilt and re-tested: bnw suite 13/13 green; cgit Anubis
tests and tool checks green (t01-t16 still need the host-side
/srv/git provisioning, unchanged).
|
|
GN2 calls BNW server-to-server: it posts My_Gennenet to bn_genenet.php
(single-POST handshake), links straight to
executionprogress_default.php?My_key=... for the computed network and
serves rendered networks from /bnw-data/. Anubis must not sit between
GN2 and BNW, so those endpoints stay DIRECT in the public nginx block
(like the git smart-HTTP exemption in cgit.scm) -- an exact-match php
location for each endpoint plus the /bnw-data/ alias, with everything
else still proxied through the challenge.
The fastcgi statements are factored into bnw-nginx-php-fastcgi-body
and bnw-nginx-php-block, shared by the internal block and the public
exemptions.
bnw_http_tests.py proves the exemption with a BROWSER user agent on
each GN2 endpoint (any other path with that UA gets the challenge):
t10 GN2 handshake POST, t11 data-load POST, t12
executionprogress_default.php, t13 /bnw-data/ 404-without-challenge.
13/13 tests pass against the rebuilt container.
|
|
Each container gets its own runner (we only work on one container at
a time): run_container_tests.sh stays cgit-only and the new
run_bnw_container_tests.sh drives the BNW container with its
bnw_http_tests.py suite (front page, static assets, php execution via
the diag page, GET+POST through Anubis, browser-UA challenge,
non-browser pass-through, 404 handling).
Runner fixes applied to both:
- auto-detect the daemon fingerprint instead of pinning a stale value
(the daemon generates a random one at startup)
- default the ssh client key to $HOME/hello-container-key, the same
file the container definitions bake in -- the stale copy under
run-guix-container/etc/ failed with "Permission denied (publickey)"
- append '; true' to the log-harvest remote loop so missing log files
do not fail the whole harvest
cgit tests: add the Anubis tests (t17 browser UA challenged, t18
non-browser UA passes through). t18 accepts the 404 front page that
cgit serves while the host-side repositories are not provisioned.
t01-t16 need /srv/git mounted from the host (see cgit-README.md).
|
|
|