about summary refs log tree commit diff
AgeCommit message (Collapse)Author
5 daysbin/bnw: headless network learning, and parity with the web application HEAD mainPjotr
BNW's computation does not need the web application: run.sh drives three C programs over files in a per-run directory, and everything above that is file assembly and rendering. bin/bnw does the same pipeline -- network_score, the k-best search, structure -- in a private working directory, and reports one row per candidate edge with its posterior and whether it survived the threshold. The reference for all of it is the running bnw-container, not a memory of it: its six continuous traits example was run through the web application (bn_genenet.php -> execute_bn_gom.php) and the resulting files kept as fixtures in test/golden/. test/parity_check.py compares our continuous_input.txt and structure_input.txt against those, and the driver's own stdout against the file. Today: byte-identical, 5 edges. Four things that run settled, each of which had been guessed wrong or not at all: * continuous_input.txt is header, then a TYPE LINE (1 = continuous, n = n-level discrete), then data. network_score reads the type before the data, and the web application's file is one line longer than the matrix because of it. Without the line the first data row shifts and network_score segfaults. * structure exits with an uninitialised status while writing a correct file. The web application ignores the status; the driver now judges it by its output and only fails if the file is missing. * get_kbest_nets.cc used SGI's <ext/hash_set>, which modern libstdc++ no longer has. It now uses std::unordered_set with an explicit hasher (specialising std::hash for a pointer type is not allowed), and build.sh compiles the C with -std=gnu89 because the sources predate C99. * our run leaves postProbEachEdge.txt with per-edge posteriors; the web application's flow leaves none behind, so the posteriors are information the endpoint can report that the web app never showed. Exit codes are part of the contract GN4 will code against: 0 ok, 2 bad input, 3 the computation failed, 77 the binaries are not installed (a conventional skip, so a test suite can run where BNW is not). Build output is gitignored. AI plan and status updated in info_files/gn4-integration-plan.md; README gained a "Headless driver" section.
5 daysPlan: a headless driver here, a call-out in GN4, and the parameters measuredPjotr
Two corrections from the owner, both structural, so the plan is rewritten around them rather than patched. The split: BNW's knowledge stays in this repository. Phase 1 becomes a headless driver with a contract GN4 can code against -- `bnw learn --input matrix.tsv --maxparents 4 --k 1 --tiers 3 --threshold 0.5 [--ban] [--white] [--format json|tsv] [--svg] [--workdir] [--keep]`, stdout as one edge per line plus a header naming the parameters and version, exit 2 for bad input and 3 for a failed computation -- with a private work directory per invocation instead of /var/lib/genenet/bnw, the parameters as arguments instead of <key>*.txt files, and no PHP, web server or /var/www. Acceptance is unchanged: byte-identical structure_input.txt and postProbEachEdge.txt against the web app's own answer. GN4's side is then a call-out and nothing more: BNW_CMD in config, a matrix in a temp file, one subprocess, edges parsed out. No knowledge of run.sh, of the C programs or of the <key> layout enters GN4, so they can change without a GN4 release. The guix.scm dependency lands last, once the driver works -- gn-bioinformatics/gn/packages/bnw.scm already pins this repository. The parameters are now measured rather than guessed. GN2 sends none: send_to_bnw.py builds only the payload (traits' shared samples, comma-joined rows, semicolon-terminated, My_Genenet) and the user picks parameters in BNW's own settings page afterwards. BNW's defaults are in sourcecodes/create_tiers_gom_part1.php, which writes parent.txt, k.txt and thr.txt from a | -joined form value: maxparents 4 (with an "All" option capped at variables-1), k 1 (no model averaging), tiers 3, edge-posterior threshold 0.5. A single-call endpoint has to expose all four, because the UI step that would set them is gone. Licence question settled by the owner: we own the code, so shipping the binaries in GN4's profile needs no third-party permission. The repository still has no LICENSE/COPYING and the C sources carry no header -- worth adding before GN4 distributes it.
5 daysPlan for embedding BNW in GeneNetwork4, and an ai-memory store herePjotr
BNW's computation is three C programs driven by run.sh over files in a per-run directory: network_score (local scores, links libRmath), k-best/src/data2netk_poster.sh (posterior per candidate edge) and structure (adjacency at a threshold). The PHP layer above them only builds those files and renders the result, so an endpoint does not need PHP and does not need the web app: it needs a private directory instead of the hardcoded /var/lib/genenet/bnw, the same three binaries, and the two files they leave behind -- structure_input.txt and postProbEachEdge.txt -- as its output. info_files/gn4-integration-plan.md sets out the phases against the engines GN4 has already embedded (bulklmm, qtl2, ravanan): golden files from the existing container first; a driver that runs the C programs without PHP and is byte-comparable to them; POST /compute/bnw taking a dataset+probes, group+ traits or a basket, gated by variables x samples rather than individuals; POST /plot/bnw rendering the SVG from the posted edges; then the part GeneNetwork actually wants -- module scores returned in the shape of trait values, so they plug straight into /compute/rqtl and /compute/gemma -- and tests that compare against the golden files. Risks named in order: combinatorics at 30 nodes, the global directory, whether the k-best search is deterministic (settle it before promising golden-file parity), discrete vs continuous networks (continuous first), and libRmath/ Python/Octave creep into the GN4 profile. Also initialises an ai-memory store for this repository (168 commits, 170 symbols) and ignores it in .gitignore, so the reasoning behind the endpoint accrues where the code lives.
2026-10-03doc: RELEASE_NOTES for run.sh fix, version endpoint, validation releasePjotr
2026-10-03validate the GN2 payload before prep; informative failures throughoutPjotr
Found by an exhaustive small-payload fuzz (test/fuzz_bnw.py): - bn_genenet.php: fail fast with a plain-English problem list for header-only payloads, ragged rows, single variable/sample, all-NA columns and all-NA rows, non-numeric cells and empty payloads; normalize CRLF and squeeze redundant row separators; rename duplicate variable names with a note (prep used to crash on each of these deep inside octave) - runtime_check.php: exe_time no longer fatals ("int + string") when prep did not produce its tables; falls back to a default estimate - execute_bn_gom.php: the FAILED page gains plain-English hints for the recurring octave failure modes - test/bnw_http_tests.py: tests t22-t33 lock the behaviour in - test/fuzz_bnw.py: the parameter-space fuzzer (75 cases)
2026-10-03run.sh: self-locate the C binaries; fix the production run failurePjotr
run.sh cd'ed to /var/lib/genenet/bnw and ran ./network_score etc RELATIVE to that directory -- depending on the activation having copied the binaries there. On production that copy was incomplete (the binaries live in the bnw PACKAGE store item since 00e5b05f removed the vendored ones and build.sh recompiles them), so every structure-learning run died with "./network_score: No such file or directory". run.sh now locates the binaries next to itself: BNW_BIN_DIR=$(dirname $0)/../var_lib_genenet_bnw which in the package is <out>/sourcecodes/../var_lib_genenet_bnw -- provably present in the store item. The per-run working directory stays /var/lib/genenet/bnw.
2026-10-03bn_genenet: sanitize GN2 trait names containing spacesPjotr
GN2 trait display names often contain spaces ("Age at death"), but BNW's file format does not allow them: the octave parsers split on WHITESPACE (strtok), so a spaced name corrupts the header -- the variable count no longer matches the labels and structure learning dies with "A(0,_): subscripts must be either integers 1 to (2^63)-1 or logicals" in mk_adj_mat (observed on production with a real GN2 dataset). bn_genenet.php now sanitizes the NAME ROW of the GN2 payload (content before the first semicolon, i.e. before the comma->tab conversion): whitespace inside a name becomes an underscore. A wrong first attempt applied the sanitization after the conversion and collapsed the tabs as well, merging all names into one variable. New test t21 posts spaced trait names and runs structure learning through to the rendered SVG. Suite: 27/27 (BNW_LOCAL_SOURCE build).
2026-10-03doc: add RELEASE_NOTES.mdPjotr
Summary of the 31 commits since e9a320d0 (the last pre-Guix upstream commit, 2021-10-11), organized by theme: Guix deployment and testing, the GN2 integration fixes (server-side handshake conversion, XSS removal, My_Genenet spelling), the UX status/failure pages for execute_bn_gom.php, the PHP 8 robustness fixes, the build-system changes, the vendored-code removal and the test suites.
2026-10-03execute_bn_gom: show pipeline failures on the status pagePjotr
runTerminalCommand throws a RuntimeException when run.sh or run_octave fails, and execute_bn_gom.php had no try/catch -- the uncaught exception produced a blank page with no forward and no diagnosis (observed on production with real GeneNetwork datasets). Wrap the structure-learning pipeline in try/catch and render the error on the status page ("Structure learning FAILED: ..." plus a link back to the upload page) instead.
2026-10-03execute_bn_gom: show a status page while structure learning runsPjotr
The runner page used to output NOTHING until the C structure learning and octave runs finished (~12s+); the browser sat on a blank page and only then got the window.open forward to the network diagram. Emit a status page first (flushed before the work starts): Structure learning is running. Forwarding to the network diagram when done... If the diagram does not open, click here. (link to layout_svg_no.php?My_key=...) so users get feedback during the run and a manual fallback if the automatic forward is blocked or fails.
2026-10-03make: drop the BNW_LOCAL_SOURCE hookPjotr
The hook was removed from guix-bioinformatics/gn/packages/bnw.scm; the container tests now always build the package from the git origin pinned there (git.genenetwork.org), so push to gn-bnw main (or update the pin) before testing source changes.
2026-10-03bn_genenet: accept the GN2 template spelling My_GenenetPjotr
GN2's bnw_page.html posts the payload field as "My_Genenet" (one 'n' in Genenet), while BNW historically read "My_Gennenet" -- the two never matched, so GN2's POST arrived empty and BNW showed its default form (the "Loading data from GeneNetwork" hang). Accept both spellings. New test t18b posts My_Genenet through the legacy /sourcecodes/ action path the template uses. Suite with BNW_LOCAL_SOURCE (this checkout): 26/26.
2026-10-02test: drop hardcoded factory paths; new containers/ location; forge -LPjotr
$FACTORY renames to $base and every checkout path becomes overridable (GUIX_BIOINF, GN_MACHINES, GUIX_FORGE, RUN_GUIX_CONTAINER; defaults $base/code/<checkout> for the factory layout). CONTAINER_SCM follows the bnw-container move to genenetwork/containers/ and the container build now passes the guix-forge load path required by (forge anubis) since the Anubis enablement.
2026-10-01remove vendored javascript and prebuilt binariesPjotr
The guix package replaces all vendored javascript libraries with guix packages (cytoscape, cytoscape-dagre, cytoscape-panzoom, dagre, d3, jquery, font-awesome, lodash, FileSaver, canvas-toBlob) via symlinks at install time; the bundled copies are stale. accordion.js, create_table.js and table.css are BNW's own code and stay. Also drop the prebuilt binaries and Windows-only BNT mex DLLs: build.sh recompiles network_score, structure and the k-best binaries from source and links libRmath from the rmath-standalone package.
2026-10-01Container ssh READMEPjotr Prins
2026-10-01doc: document $HOME-based container keys and the BNW branch of ↵Pjotr
guix-bioinformatics The pinned container keys are generated by hand in $HOME and never enter a repository; the test runner now ssh-es with $HOME/hello-container-key (CLIENT_KEY override). Also documents restoring gn/packages/bnw.scm from guix-bioinformatics' BNW branch — without it the build silently falls back to unpatched upstream BNW.
2026-09-29build: add a Makefile test driver (test/run/clean) and document itPjotr
'make test' builds the BNW package from this checkout (BNW_LOCAL_SOURCE in guix-bioinformatics) and runs the full container test cycle; 'make run' leaves the container up on :8881; 'make clean' removes test logs, the /tmp VM image and leaked test VMs.
2026-09-28ignores: build artifacts (*.o, patch-shebangs *.real backups)Pjotr
2026-09-28container: deployment fixes for the Guix container/VMPjotr
Changes previously carried as bnw-container-paths.patch in guix-bioinformatics, now applied to the source directly: * unify data directories on /var/lib/genenet/bnw (was /tmp/bnw and ./data/ mixed) * make the shell_exec paths __DIR__-relative and the example/octave helper scripts location-independent, so php-fpm can run them from any working directory * use core-octave randn/exp instead of the statistics-package normrnd/normpdf (not available in the container's octave) * nnodes: strtrim before strfind so the node count parses with the container's octave output * python3 instead of a hardcoded /home/jziebart path * add diag.php, a debug page showing the request and data state * bn_genenet.php: serve the GeneNetwork (GN2) handshake server-side; the My_Gennenet POST is converted (commas -> tabs, semicolons -> newlines) in PHP and the prepared-data page is returned in ONE request. This removes the raw-echo of into a JavaScript string (an XSS hole on public deployments) and is backward compatible for GN2 (same POST, better response).
2026-09-27doc: document the GN2 handshake XSS fix and GN-side impactPjotr
2026-09-27test: t18 now asserts the server-side GN2 conversion (no JS echo)Pjotr
With the XSS fix the My_Gennenet POST returns the prepared-data page in one request; t18 asserts that and that the genenetwork_data JS echo is gone. 25/25.
2026-09-27test: emulate the GeneNetwork (GN2) handshake over HTTPPjotr
GN2 posts My_Gennenet to bn_genenet.php (commas = columns, semicolons = rows); the page converts it client-side to TSV in the hidden searchkey field and auto-submits. The new tests cover the entry point (t18: payload embedded in genenetwork_data() + auto-submit stub), the browserless emulation of the conversion (t19: searchkey/ My_key/MyUpload=NO posted directly, data prepared), and a full GN2 driven structure learning + rendering (t20). 25 checks pass via test/run_container_tests.sh.
2026-09-27ignoresPjotr Prins
2026-09-27test: run the full container cycle (build, start, test, stop)Pjotr
test/run_container_tests.sh builds the bnw-container definition, starts it through the run-guix-system-container daemon, waits for the web server on port 8881, runs the 22-check HTTP suite and the container tool check over ssh, and stops the container (kept on failure or --keep for debugging).
2026-09-27test: catch the layout_svg_no data1=false failure mode and asset downloadsPjotr
Strengthen the example-workflow test: layout_svg_no.php must embed the real SVG ('var data1 = ['), never 'var data1 = false' (the json_encode(file(...)) failure that crashes the page with 'data1.join is not a function' in the browser). Add checks for the rendered SVG being downloadable via /bnw-data/ and for the page script assets (d3.v4.min.js, jquery, font-awesome, accordion, create_table.js) downloading with real content. 22 checks pass.
2026-09-27test: cover all six workflow.php examples (20 checks)Pjotr
Parametrize the example test over every example.php link on workflow.php (example2, example_sci, example_chr2_spleen, example_chl, example_time_series, examplecar15node): each loads the pre-computed example state into a fresh key and renders its network via layout_svg_no.php. All 20 checks pass.
2026-09-27test: add HTTP functional tests and tool checklistPjotr
test/bnw_http_tests.py drives a running BNW over plain HTTP requests (no headless browser): static pages, key handling, upload + octave input preparation, structure learning, network rendering and the worked examples from workflow.php, for both a discrete (8-node) and a continuous (5-node) dataset. 15 checks, all passing against the Guix container build of the upgraded tree. test/check_container_tools.sh verifies every unix tool called externally (octave, dot, python3, the compiled scoring binaries, coreutils glue) exists in the deployment environment. README documents the test suite, the external tool inventory and the PHP8/container issues the tests uncovered (mixed /tmp/bnw and /var/lib/genenet/bnw data dirs, home.php redirect stub, broken keyval_error redirect, hardcoded python path).
2026-08-24Make all shell scripts executable.Frederick Muriuki Muriithi
The code needs the shell scripts to be executable, otherwise the code will raise an error (Permission Denied).
2026-08-24Use new `runTerminalCommand()` wrapper to run commands.Frederick Muriuki Muriithi
2026-08-24Add a wrapper around `proc_open(...)` to use for terminal commands.Frederick Muriuki Muriithi
2026-08-21Use pkgconf to find libRmathFrederick Muriuki Muriithi
Use the libRmath provided by the system rather than the one bundled with the code (this will be removed).
2026-08-18Bugfix: Fix path.Frederick Muriuki Muriithi
2026-08-18Bugfix: Check that key exists before use.Frederick Muriuki Muriithi
2026-08-18Bugfix: Move variable closer to usage.Frederick Muriuki Muriithi
Initialise the `$TextFile` variable within the handling of the "POST" request rather than in the global scope.
2026-08-17Fix undefined $TextinFile/$keyval in bn_after_upload_gom.phpClaude Sonnet 5
$TextinFile was passed to move_uploaded_file() as its $to argument, but its assignment (and the $keyval/$sid/$dir it depends on) was commented out, so $TextinFile was always null/undefined here -- an upload through this page could never succeed even on PHP 7.4, and on PHP 8.1+ it also triggers "Passing null to parameter #1 ($to)... is deprecated". $keyval was similarly used (e.g. in the hidden My_key form field) before it was ever assigned, causing an "Undefined variable" warning that could leak into an unquoted HTML attribute if display_errors is on. Unlike bn_file_load_gom.php (a fresh-upload entry point that mints a new random key), this page is only reached via a redirect from modify_structure_learning.php with an existing My_key already in the query string. Restored the commented-out block, adapted to read $keyval from $_GET (matching how the rest of this file already does it further down) instead of generating a new one, so $TextinFile is a real path before the upload block runs. Verified empirically against PHP 8.3.28: simulating an upload request now correctly reaches move_uploaded_file() with a real path and falls into the file's own pre-existing "Sorry, error uploading" handling instead of silently failing on a null argument. Reviewed by: Frederick M Muriithi <fredmanglis@gmail.com>
2026-08-17Guard arithmetic on file/shell-output-derived values against non-numeric stringsClaude Sonnet 5
PHP 8 throws a fatal TypeError for arithmetic (+ - * /) on a non-numeric string ("Unsupported operand types"); PHP 7.4 only warned and treated it as 0. Several files perform arithmetic directly on values parsed from intermediate state files (nnode.txt, nrows.txt, structure_input*.txt) or from `dot -Tplain` output, with no is_numeric() check — safe as long as those files are well-formed, but a crash risk if a file is missing/empty/truncated or a `dot` output line doesn't parse as expected. Affected: runtime_check.php ($node, $nrows), remove_variables.php and create_tiers_gom_part1.php ($maxplist=$node-1), mat_structure.php (matrix cell multiplication), and the four network_layout_{evd,inv}[,_2].php files ($r_index=$node+N and the dot-output-derived $cell arithmetic in the _2 variants). Fix pattern: coalesce to 0 via `is_numeric($x) ? $x : 0` before the arithmetic, matching PHP 7.4's original fallback-to-0 behavior instead of crashing. Verified empirically against PHP 8.3.28 with runtime_check.php and mat_structure.php: previously, a missing/empty source file caused TypeError: Unsupported operand types: string - int (or string * int) after the fix, the same missing-file scenario produces only the pre-existing "undefined array key" warnings (same as PHP 7.4) and the function completes and returns a value instead of crashing. Reviewed by: Frederick M Muriithi <fredmanglis@gmail.com>
2026-08-17Guard fopen() results with a false-check before fwrite/fprintf/fcloseClaude Sonnet 5
On PHP 7.4, calling fwrite()/fprintf()/fclose() with a false handle (a failed fopen(), e.g. from a bad path, missing directory, or permissions issue) just emits a warning and no-ops. On PHP 8.0+, these functions require a resource argument and throw an uncaught TypeError instead, turning what used to be a silent degradation into a fatal crash of the whole request. This patch wraps every fopen()-then-write call site in the codebase (18 files) with an `if ($handle !== false) { ... }` check, matching the existing pattern used elsewhere in the codebase. Where the same $fpvar handle is threaded through shared helper functions (enter_ban_list/banlist/whitelist, duplicated across remove_variables_processing.php, remove_variables_processing_default.php, modify_structure_learning.php and tier_description_processing_gom.php), the guard was added inside those functions too for consistency, even though the call sites are currently dead/commented-out in three of the four files — so the same landmine doesn't reappear if that code is ever re-enabled. Verified empirically against PHP 8.3.28: fopen() on an unwritable path followed by fwrite() on the resulting `false` throws TypeError: fwrite(): Argument #1 ($stream) must be of type resource, false given without the guard; with the guard, the call is skipped instead of crashing. The normal (successful fopen) path was also re-run through mat_structure.php's structure_change() and still produces identical output to before the change. Reviewed by: Frederick M Muriithi <fredmanglis@gmail.com>
2026-08-17Guard valid_keyval()/valid_input() against null input (PHP 8.1 deprecation)Claude Sonnet 5
These two functions are the single most common entry point in the codebase — ~30+ files call them as valid_keyval($_GET["My_key"]) or valid_input($_POST[...]) with no isset() guard at the call site. When the key is absent, PHP passes null straight into trim(), which is a non-nullable string parameter. As of PHP 8.1 this triggers a "Passing null to parameter #1 ($string) ... is deprecated" notice on every one of those call sites, on every request missing the param. Coalescing to '' inside the two functions themselves fixes this once, for every caller, instead of patching every call site individually. Verified empirically against PHP 8.3.28: valid_keyval(null) / valid_input(null) no longer emit the trim() deprecation notice and still return '' as before (same effective behavior as PHP 7.4). Reviewed by: Frederick M Muriithi <fredmanglis@gmail.com>
2026-08-17Remove orphaned fwrite() call in runtime_check.php that crashes on PHP 8Claude Sonnet 5
$fprows and $lc were never defined anywhere in exe_time(). On PHP 7.4 fwrite() with a null resource argument just emitted warnings and returned false. On PHP 8+, fwrite() requires a resource/false is no longer coerced, so this throws an uncaught TypeError on every call, crashing any page that invokes exe_time() (progress-tracking pages). Verified empirically against PHP 8.3.28: before the fix, calling exe_time() throws TypeError: fwrite(): Argument #1 ($stream) must be of type resource, null given after the fix it returns the computed runtime estimate as before. Reviewed by: Frederick M Muriithi <fredmanglis@gmail.com>
2021-10-11Update home.phpziejd2
2021-10-11Update home.phpziejd2
2021-10-11Update layout_svg_no.phpziejd2
2021-10-11Resolved merge conflictziejd2
2021-10-11Update README.mdziejd2
2021-10-08Update bn_genenet.phpziejd2
2021-10-08Add files via uploadziejd2
2021-10-08Update network_score.cziejd2
2021-10-08Update network_score.cziejd2
2021-10-08Update network_score.cziejd2
2021-10-08Update README.mdziejd2