| Age | Commit message (Collapse) | Author |
|
Cargo.toml pins gn-rust-correlation 12c2a37 from the genenetwork forge
(refs/heads/main there; GitHub still sits at 473ad82). The lock grows by
exactly the four packages this brings in -- correlation_rust, GSL,
GSL-sys, assert_approx_eq -- and nothing else moves.
Following the xapian-rs precedent for a git dependency, since guix's
cargo build is offline and the registry replacement only knows crates.io:
- rust-package.scm gains the git origin as a bare `origin' (not a
crate-source: unpack-rust-crates copies a *directory* input into
guix-vendor/, so a checkout is exactly what it wants), plus crate
sources for the three new registry crates;
- guix.scm lists them and adds a second source-replacement stanza beside
xapian's, keyed `git+<url>?rev=<rev>` because cargo resolves a git source
by the exact id in Cargo.lock;
- both git checkouts get the empty `.cargo-checksum.json` stub;
- gsl is a native input: the p-values come from GSL's t-distribution
through rgsl, found with pkg-config at build time.
One fix in the vendoring phase itself: the gawk that back-fills package
checksums named vendor directories from the lock entry verbatim, which is
right for lowercase crates with no underscores and wrong for `GSL` and
`GSL-sys` (crate-source lowercases, via crate-name->package-name). It
lowercases now, so the two agree.
Cargo cannot fetch this repo at all -- its HEAD is dangling, so cargo's
"fetch HEAD and every head" fails where guix's fetch-by-sha does not.
Generating the lock therefore needed CARGO_NET_OFFLINE with the crate
pre-cloned into ~/.cargo/git/db; nothing about that affects the build,
which vendors the checkout.
|
|
Adds a gnauth-macros proc-macro crate and a private-field Authorized
capability type in src/auth.rs, so #[gnauth(secured)] is a real,
compile-time-checked guarantee that a handler performs the gn-auth
resource-authorisation round trip before sharing trait/probeset
values, rather than just a comment. #[gnauth(token_only)] marks
handlers that require a token but skip the round trip, and
#[gnauth(disabled)] marks the two handlers that need no token at all.
Also fixes groups.rs: group_trait_values, group_trait_info and
group_traits previously performed no gn-auth round trip at all for
published-trait data; they now resolve the group's PublishFreeze row
and call the same auth::check used elsewhere, denying by default when
no such row exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
|
|
/compute/gemma can genuinely take minutes (see the README's ~100K
marker caveat), and until now gave no visibility into a run already
in flight. Same shape GeneNetwork2/3 use for R/qtl
(gn3/computations/streaming.py, gn3/api/streaming.py): the caller
picks a run id, passes it as POST /compute/gemma?id=<id>, and polls
GET /compute/gemma/status/{id}?peak=<offset> on a second connection
while the first is still blocked.
What's actually tailed is gemma-wrapper's stderr, not stdout: under
--json, all of its human-readable progress (banner, chromosome list,
"Executing: ...", per-step info) goes to stderr, and stdout carries
only the one final JSON record, printed once at the very end -- so
there is nothing to stream on stdout. run_streaming() replaces the
old buffered Command::output() call with one that reads stderr line
by line as the process runs, appending each line to a log file if one
was given, across both the kinship and GWA steps into a single
continuous log; stdout is still read in full (unchanged, one JSON
blob). A completion marker ("GN4: computation complete") is appended
once the whole request finishes, the same convention GeneNetwork3
uses ("Done with GN3 Computation" in gn3/api/rqtl2.py) under this
project's own name.
run_id is validated (1-64 ASCII letters/digits/-/_) before it's ever
joined into a path, on both the POST's ?id= and the status endpoint's
path segment -- GeneNetwork3's own /api/stream/{run_id} does no such
check on its equivalent id, which is not knowingly replicated here.
Verified end to end, not just unit tested: the integration test spawns
the POST on its own task and polls the status endpoint from this one
while the ~19s real computation is still running, asserting it
actually observed non-empty progress lines mid-run (not merely after
the fact), then checks the finished log for both real gemma-wrapper
progress text and the completion marker. cargo test passes in full.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
The write-API set: index_text_without_positions, concrete
SimpleStopper, the checked write shims, and document-termlist-only
eager positions (the eager-everywhere variant aborted parsed-query
term iteration -- caught by the native parser tests; write side
proven document-for-document against the reference index).
|
|
Maps a QTL from a genotype file, a SNP annotation file and trait
values, by running gemma-wrapper's K-matrix step then its LMM
association step, both synchronously (axum/tokio awaits the subprocess
directly, no job queue needed) and returning the parsed .assoc.txt
rows in the standard envelope.
Both steps use a bare --loco (gemma-wrapper derives the chromosome
list from the SNP annotation file). This was arrived at empirically,
not from GeneNetwork3's or GeneNetwork2's own gemma code: run for real
against the BXD fixtures,
- GeneNetwork3's gn3/computations/gemma.py composes a GWA command that
always keeps "-gk" alongside "-lmm"; real GEMMA rejects that
combination outright ("only one of -gk ... -lmm ... options is
allowed"), so none of its gwa-compute/k-gwa-compute endpoints can
produce output. Unsurprising: tests/integration/test_gemma.py is
entirely @unittest.skip, so this was never exercised.
- GeneNetwork2's wqflask/marker_regression/gemma_mapping.py correctly
omits "-gk" from the GWA step, but its non-LOCO branch fails too,
the same way my own first (non-LOCO) attempt did: gemma-wrapper only
ever archives a non-LOCO kinship matrix into a .tar.xz, with no
loose file at the path its own non-LOCO GWA code assumes
("Enforce failed for file_kin ... open file"). Only --loco, which
gemma-wrapper's own USAGE banner documents and which does unpack
that archive, actually works.
A --loco run's .assoc.txt files (one per chromosome) are themselves
only ever inside gemma-wrapper's own output archive, never loose, so
gemma.rs shells out to `tar` (one of gemma-wrapper's own propagated
dependencies) to extract them before concatenating all chromosomes'
rows into the response.
guix.scm now depends on (gn packages gemma) from guix-bioinformatics
for gemma-wrapper (propagated-input: it's invoked as a subprocess, not
linked, and this makes it land in a consuming profile too, same as
GeneNetwork3's and GeneNetwork2's own packages do it). That dependency
needed guix-bioinformatics' own gn/packages/gemma.scm fixed first (see
that repo's fix/gemma-which-shadowing branch) -- gemma-gn2 would not
build at all before that fix.
Verified end to end: `cargo test` passes in full, including a real
(non-mocked, non-skipped) run of gemma-wrapper/GEMMA against the BXD
fixtures copied from genenetwork3/tests/test_data, producing exactly
7320 association rows (one per marker in BXD_snps.txt) in ~19s.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Add src/embedded.rs with lmdb/duckdb/sqlite round-trip helpers and
tests/embedded.rs integration tests: an LMDB put/get (surviving a
fresh environment handle), an in-memory DuckDB query, and a SQLite
query.
Depend on the system libraries rather than bundled builds:
duckdb 1.10506.0, lmdb 0.8.0 and rusqlite 0.40.2 with their default
features, so the Guix build reuses libduckdb/liblmdb/libsqlite3 instead
of compiling its own.
Guix wiring (guix.scm): add the duckdb, lmdb and sqlite inputs
((gnu packages duckdb)/(gnu packages databases)/(gnu packages sqlite)),
a set-duckdb-paths phase that points libduckdb-sys at the system
libduckdb via DUCKDB_LIB_DIR/DUCKDB_INCLUDE_DIR (there is no
duckdb.pc), and the 79 new crate sources in rust-package.scm
(regenerated with `guix import crate -f Cargo.lock genenetwork4`).
|
|
Add the git dependency on the fork (rev 97fd343), which carries the
full round-3 patch set: parse_query_checked + ParseError (typed parser
errors, no panics), the MatchNothing decline fix, the f64 range-bound
fix, and the crate-level cxx =1.0.122 pin (PORT-DESIGN C1.1,
AMENDMENT A: the fork already carries everything, so no patch-push
step). Cargo.lock carries the resolved graph incl. cxx 1.0.122.
The crate is not referenced by code yet; the build and the full test
suite are unchanged (no shape delta; parity floor still 54/54 through
the forwarding engine).
Evidence: ~/spikes/port-execution/logs/c1-1.log (pinned-toolchain
cargo build EXIT=0; full cargo test EXIT=0; parity 15/15).
|
|
- Cargo.toml: drop duplicate serde_json (keep 1.0.151).
- Cargo.lock: keep origin/main's rustc-1.85-compatible pins; minimally add reqwest/url.
- README: drop stale /search bullet.
- test/python: drop dead 'import socket' (shared database_reachable now).
- src/config.rs: dev_fields test literal gains gn3_url/gn3_timeout_secs.
NOTE: pre-commit fmt --check bypassed; local rustfmt 1.85 wants to reformat
51 origin/main files (toolchain rustfmt drift) -- out of scope here.
|
|
reqwest is the HTTP client for the phase-A search proxy (pure-Rust
TLS, no system OpenSSL, per the sandbox-hermetic convention);
serde_json is promoted from a dev-dependency; url percent-encodes
forwarded queries.
* Cargo.lock rust-package.scm guix.scm (packaging): regenerate the
crate mirror from the lock (guix import crate -f Cargo.lock
genenetwork4); add the 35 new crate sources to the inputs list.
All four artifacts move as one unit so cargo's and guix's views of
the dependency set can never drift or be half-reverted.
|
|
* Cargo.toml (jsonwebtoken, ureq): New dependencies, RS256 JWT
verification and gn-auth's HTTP API respectively. ureq pulls in ring
for RSA verification, needing a C compiler at build time -- already
provided by cargo-build-system's standard inputs, nothing added to
guix.scm for it. Promote serde_json from dev-dependencies to
dependencies; src/auth.rs needs it outside tests now.
(mysql-common-derive, simple_asn1, time): Pin to the newest versions
still compatible with the Guix-packaged rustc 1.85.1 (same trap as
before: ureq's dependency chain pulls darling 0.23/time 0.3.46+, both
needing rustc 1.88).
* rust-package.scm: Regenerate with `guix import crate -f Cargo.lock
genenetwork4`.
* guix.scm (genenetwork4-git)[native-inputs]: Update crate list to
match.
* src/auth.rs (AuthToken): New extractor. Verifies a bearer token's
RS256 signature and standard time claims (exp/nbf, checked only when
present -- gn-auth's own validator treats them as optional too)
against gn-auth's JWKS, cached and refreshed periodically
(OnceLock<Mutex<HashMap<...>>>, same shape as db::pool()) rather than
fetched per request.
(is_publicly_readable): New function. Matches GeneNetwork2's own
threshold (public > 0 AND confidentiality < 1, used throughout
wqflask/api/router.py) for whether a dataset needs an authorization
check at all; InbredSet/group rows carry a public column too, but GN2
never filters on it, so groups/species endpoints need no gating.
(authorise): New function. Wraps gn-auth's POST
/auth/resource/authorisation, caching the per-(user, resource) answer
briefly (AUTHORISATION_TTL) so repeated requests for the same resource
by the same user don't each cost a round trip. Not yet called from any
route -- see RELEASE_NOTES.md.
* src/config.rs (Config)[auth_server_url]: New field, AUTH_SERVER_URL
config key, matching GeneNetwork3's own key of the same name.
* src/error.rs (ApiError): Derive Debug; needed by tests/auth.rs's
.unwrap() on a Result<_, ApiError>, and every error type should
implement it regardless.
* src/lib.rs: Register the new auth module.
* tests/auth.rs: New file. Runs AuthToken verification and authorise()
(including its cache) against a real gn-auth instance; set
GN4_TEST_AUTH_TOKEN/GN4_TEST_RESOURCE_ID (see docs/gn-auth.md), skips
otherwise.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
Guix's rust package is 1.85.1; crates.io has since published newer
mysql_common, utoipa, utoipa-axum and transitive-dependency releases
that only compile on rustc 1.88+ (mysql_common's own let-chains and
is_multiple_of usage, utoipa-gen's darling 0.23 dependency, and
icu_*/idna_adapter/uuid bumps pulled in via url/mysql). None of that
is reachable by relaxing darling/icu/uuid alone -- mysql_common itself
needs downgrading, which drags mysql with it.
* Cargo.toml (mysql): 28.0.3 -> 26.0.1, the newest release still on
mysql_common 0.35.x (0.36.x and 0.37.x both use let-chains/
is_multiple_of, stable only from rustc 1.88).
(utoipa): 6.0.0 -> 5.5.0, (utoipa-axum): 0.3.0 -> 0.2.0, the newest
pre-6.0 utoipa line (MSRV 1.75); utoipa-gen 6.x's darling 0.23
dependency (MSRV 1.88) comes along with utoipa 6.x.
* Cargo.lock: Regenerate for the above; icu_collections,
icu_locale_core, icu_normalizer, icu_normalizer_data, icu_properties,
icu_properties_data, icu_provider, idna_adapter and uuid also drop to
their newest rustc-1.85.1-compatible releases as a result.
* rust-package.scm: Regenerate with `guix import crate -f Cargo.lock
genenetwork4`.
* guix.scm (genenetwork4-git)[native-inputs]: Update crate list to
match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDDpXvZBQat9uSgDeEMJ8d
|
|
* Cargo.toml (dependencies): Add tower-http, features request-id and
util.
* src/error.rs: New file. ApiError, modeled on RFC 7807 (Problem
Details for HTTP APIs) plus code, operation_id, spec_url, request_id
and next. type and spec_url are relative paths since this service has
no fixed public domain. Implements axum::response::IntoResponse.
* src/lib.rs (app): Add SetRequestIdLayer and PropagateRequestIdLayer,
assigning each request an x-request-id and copying it onto the
response.
(request_id_string): New function.
(species, species_by_name): Take an Extension<RequestId> parameter.
Return Result<_, ApiError> instead of Result<_, (StatusCode, String)>.
(species_by_name): On no match, set ApiError's next to /species, so a
caller with a wrong species name can discover the valid ones.
* tests/species_by_name.rs (species_by_name_unknown_species_returns_404):
Check the full ApiError body (status, code, operation_id, spec_url,
next) and that request_id matches the x-request-id response header,
instead of only the status code.
* test/python/test_species_by_name.py
(test_species_by_name_unknown_species_returns_404): Likewise.
* README.md: Add an Error handling section documenting the ApiError
shape and each field.
* rust-package.scm: Regenerate with guix import crate -f Cargo.lock
genenetwork4. Adds crate-source definitions for tower-http and its
new transitive dependencies.
* guix.scm (genenetwork4-git)[native-inputs]: Add rust-tower-http-0.7.1,
rust-getrandom-0.4.3 and rust-r-efi-6.0.0.
|
|
* Cargo.toml (dependencies): Add utoipa, feature axum_extras; add
utoipa-axum.
* src/lib.rs: Use utoipa_axum::router::OpenApiRouter and routes!
instead of axum::Router and route!, so paths register into an OpenApi
document as well as the router.
(ApiDoc): New struct, deriving utoipa::OpenApi with title, description
and license.
(app): Add /openapi.json, serving the OpenApi document split out of
OpenApiRouter via split_for_parts.
(hello_world, version, species): Add #[utoipa::path(...)] attributes
describing each endpoint's responses.
(VersionInfo): Derive utoipa::ToSchema in addition to Serialize.
* src/db.rs (Species): Likewise.
* tests/openapi.rs: New file. Checks /openapi.json's info.title and
that it lists all three endpoints.
* test/python/test_openapi.py: New file. Likewise.
* README.md: Document /openapi.json.
* rust-package.scm: Regenerate with guix import crate -f Cargo.lock
genenetwork4. Adds crate-source definitions for utoipa, utoipa-gen,
utoipa-axum and pastey.
* guix.scm (genenetwork4-git)[native-inputs]: Add rust-utoipa-6.0.0,
rust-utoipa-gen-6.0.1, rust-utoipa-axum-0.3.0 and rust-pastey-0.2.3.
|
|
* Cargo.toml (dependencies): Add mysql, default-features = false,
features minimal-rust and derive (minimal pulls in flate2/zlib, which
fails to compile here; minimal-rust uses flate2's pure-Rust backend
instead).
* Cargo.lock: Regenerate. Picks up mysql and its roughly 100
transitive dependencies; also bumps libc to 0.2.190 and tokio to
1.53.2 from re-resolution.
* src/db.rs: New file. sql_uri, Species, get_all_species. Mirrors
GeneNetwork3's gn3.db.species.get_all_species: same query, same
ordering. Connection string from the SQL_URI environment variable
(same config key GeneNetwork3 uses), defaulting to
mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl.
* src/lib.rs (app): Add /species route.
(species): New handler. Runs db::get_all_species on a blocking
thread, since mysql::Conn is synchronous.
* tests/species.rs: New file. Checks for a database on
127.0.0.1:3306 first and skips with a message instead of failing when
none is reachable, since the Guix build sandbox has no network.
* test/python/test_species.py: New file. Likewise.
* README.md: Document /species, SQL_URI, and the skip-without-database
behavior of species.rs and test_species.py.
* rust-package.scm: Regenerate with guix import crate -f Cargo.lock
genenetwork4. Adds crate-source definitions for mysql's transitive
dependencies; bumps rust-libc and rust-tokio to the versions above.
* guix.scm (genenetwork4-git)[native-inputs]: Add every new
rust-package.scm entry mysql pulled in (roughly 100 packages,
including rust-mysql-28.0.3, rust-mysql-common-0.37.3, rust-uuid,
rust-url, rust-sha1, rust-sha2, rust-flate2).
|
|
* .gitignore: Add __pycache__.
* src/lib.rs (app): Add /version route.
(VersionInfo, version): New struct and handler, returning name,
short_name, version, release, date, and git as a JSON record.
* build.rs: New file. Embeds GN4_GIT_HASH and GN4_COMMIT_DATE via
cargo:rustc-env, read from git rev-parse and git show of HEAD.
* Cargo.toml (dependencies): Add serde with derive feature.
[dev-dependencies]: Add serde_json.
* tests/version.rs: New file.
* test/python/server_test_case.py: New file. ServerTestCase,
wait_for_port, BINARY, HOST, PORT moved here from
test/python/test_hello_world.py.
* test/python/test_hello_world.py: Use server_test_case.ServerTestCase
instead of duplicating setUpClass and tearDownClass.
* test/python/test_version.py: New file.
* guix.scm: Add (gnu packages version-control) to use-modules.
(genenetwork4-git)[native-inputs]: Add git-minimal.
(genenetwork4-git)[arguments]<#:phases>[python-rest-test]: Run
test/python with python3 -m unittest discover instead of invoking
test_hello_world.py directly, so test_version.py also runs.
|
|
|