# BNW This contains the code for the Bayesian Network Webserver (BNW). The master branch will be the stable version of the code for running with GeneNetwork. The BNW website is compbio.uthsc.edu/BNW; the GeneNetwork version is at: bnw.genenetwork.org. There are a couple of files in the info_files directory that provide descriptions of the BNW code and other information. There are five branches. 1) master: This is the version of BNW that should be loaded to GeneNetwork. 2) matlab_original (compbio.uthsc.edu/BNW_1.01): This is functionally identical to the original BNW code. The only difference is that unused files were removed. 3) octave_version (compbio.uthsc.edu/BNW_1.02): This should perform similarly to the original version of BNW. However, instead of Matlab, Octave is used and a couple of new features were added. 4) genenet6_final_1.3: This is the final version of BNW on the GENENET6 server. 5) genenet8_initial_1.3: This is the working version of BNW on GENENET8 when we transitioned to that server. A couple of notes about this version. - This version writes files created by users running BNW to /var/lib/genenet/bnw. - The structure learning package needs to be installed in /var/lib/genenet/bnw. The files in var_lib_genenet_bnw need to be copied to /var/lib/genenet/bnw and build.sh should be run there. - There is a python environment on GENENET8 that is used by BNW. This is in bnw-env, but I am not sure how well this will transition to other servers. ## Headless driver: `bin/bnw` The web application is not needed to run BNW's computation. `bin/bnw` does the same three-stage pipeline -- `network_score`, the k-best search, `structure` -- in a private working directory, and writes machine-readable results: ```sh # build the C programs once (needs gcc, g++ and R's standalone math library) var_lib_genenet_bnw/build.sh # learn a network: variables x samples, header line first bin/bnw learn --input test/golden/six_traits_matrix.tsv ``` ``` # bnw 0.1.0 variables=6 samples=200 maxparents=4 k=1 tiers=3 threshold=0.5 from to posterior in_structure Trait2 Trait1 1 1 Trait2 Trait4 1 1 ... ``` Every candidate edge is reported with its posterior, and `in_structure` says whether it survived `--threshold`. Options: `--maxparents` (default 4), `--k` (1), `--tiers` (3), `--threshold` (0.5) -- BNW's own defaults from `sourcecodes/create_tiers_gom_part1.php`; `--ban`/`--white` for edge constraints; `--format json`; `--svg FILE` for a graphviz drawing; `--workdir DIR` and `--keep` to inspect a run. Exit codes: 0 ok, 2 bad input, 3 the computation failed, 77 the binaries are not installed. ### Parity with the web application `test/parity_check.py` runs the driver on the six continuous traits of BNW's own 8-node example and compares against files the *web application* produced (the `test/golden/` fixtures, captured from the `bnw-container` on 2026-11-12 by posting to `bn_genenet.php` and running `execute_bn_gom.php`): ``` ok continuous_input.txt matches the web application's ok structure_input.txt matches the web application's (5 edge(s)) ok stdout reports the same 5 edge(s) PARITY OK ``` That check is also how the input format was pinned down: `continuous_input.txt` is a header line of variable names, then a **type line** (1 = continuous, n = n-level discrete, one integer per variable), then the data. The type line is not optional -- `network_score` reads it before the data -- and it is the reason the web application's copy has one more line than the matrix. ## Testing (HTTP functional tests) `test/bnw_http_tests.py` tests a *running* BNW instance over plain HTTP requests (no browser, no selenium needed). It covers: * static pages and assets, * the 3-letter key round-trip, * data upload (`bn_genenet.php`) and octave input preparation (`run_prep_input`) for a discrete (8-node) and a continuous (5-node) example dataset, * the structure learning pipeline (`executionprogress_default.php`, `execute_bn_gom.php` -> `run.sh` -> `network_score`, `structure`, the k-best scripts), * network rendering (`graphviz_structure.php`), * ALL SIX worked examples linked from `workflow.php` (example2, example_sci, example_chr2_spleen, example_chl, example_time_series, examplecar15node -> `layout_svg_no.php`). Run it against a local instance: ```sh BASE_URL=http://127.0.0.1:8881 python3 test/bnw_http_tests.py # 20 checks ``` ### GeneNetwork (GN2) integration GN2 posts `My_Gennenet` to `bn_genenet.php`: the payload uses commas as column separators and semicolons as row separators, plus a 3-letter session key (`My_key`, empty = random). The original release echoed the raw payload into a JavaScript string (`genenet_str = ""`), which is an XSS hole on public deployments. It is now fixed server-side: the conversion (`,` -> TAB, `;` -> newline) happens in PHP and the prepared-data page is returned in ONE request. This is backward compatible for GN2 -- same POST, better response; no GN2 changes are required. GN2 can additionally link straight to `executionprogress_default.php?My_key=` after posting to show the computed network. The tests cover the handshake without a browser: t18 posts `My_Gennenet` and asserts the server-side conversion (and that the JS echo is gone), t19 posts the converted `searchkey` directly (the legacy second leg), t20 runs structure learning + rendering on the GN2 data. A second script, `test/check_container_tools.sh`, verifies that every unix tool BNW calls externally exists in the (container) environment; run it inside the container. `test/run_container_tests.sh` does the whole cycle: it builds the system container (gn-machines/genenetwork/services/bnw-container.scm, needing the guix-bioinformatics and gn-machines channels on the load path), starts it through the run-guix-system-container daemon, waits for the web server, runs the HTTP test suite and the tool check, and stops the container again (kept running on failure, or with --keep): ```sh test/run_container_tests.sh # or --keep to inspect afterwards ``` The same cycle is available as `make` targets from the repo root: ```sh make # build from this checkout and run the container tests make run # leave the container running afterwards (web on :8881) make clean # remove test artifacts (logs, pycache, /tmp VM image) # and kill any leaked test VM ``` `make` runs the container tests, which build the BNW package from the origin pinned in `guix-bioinformatics/gn/packages/bnw.scm` (fetched from git.genenetwork.org) -- so push your changes to gn-bnw main (or update that pin) before testing them. If `make` is not on your PATH, run it via `guix shell make -- make test`. ### Container keys The container pins an SSH host key and ships the admin key for password-less root login. Both live **outside the repository**, in the building user's `$HOME` (see the **Keys** section of `gn-machines/genenetwork/services/cgit-README.md` for the background): ```sh ssh-keygen -t ecdsa -f ~/ssh-container-key -N "" -C "container host key" ssh-keygen -t ed25519 -f ~/hello-container-key -N "" -C "container admin key" ``` `bnw-container.scm` fetches the keys from `$HOME` at build time; building without them fails with a missing-file error. The test runner ssh-es into the container with `$HOME/hello-container-key` (override with the `CLIENT_KEY` variable). Private keys never enter any repository. ### Package definition location The BNW package (and the container patch machinery) lives on the **`BNW` branch** of guix-bioinformatics (`gn/packages/bnw.scm`). Until that branch is merged, a working tree built from another branch needs the file restored from the branch: ```sh cd guix-bioinformatics git show BNW:gn/packages/bnw.scm > gn/packages/bnw.scm git show BNW:gn/packages/bnw-container-paths.patch \ > gn/packages/bnw-container-paths.patch ``` Without it the build silently falls back to the unpatched upstream BNW, whose octave preparation scripts fail under the container's octave (the give-away is uploads landing under a random three-letter key instead of the posted one). Memory footprint measured in a Guix system container (8-node example run, php-fpm + nginx + octave + shepherd): * idle: ~58 MB PSS * peak: ~210 MB PSS (octave loading the Bayes Net Toolkit and running the search; grows with the number of nodes) ## Unix tools called externally by the BNW sources Scanned from `shell_exec`/`system`/`proc_open` calls in the PHP code, `system()` calls in the octave run scripts, script shebangs and hardcoded absolute paths: | tool | called from | |-------------------------------|---------------------------------------------------------| | octave (octave-cli) | shebangs of sourcecodes/run_scripts/* (input prep, settings, structure learning, cross validation, plotting) | | dot (graphviz) | php network layout pages (`/usr/bin/dot -Tplain`), octave `createSVG` (`/usr/bin/dot -Tsvg`) | | python3 | plotly figure generation: cv_plotly.py, ts_plotly.py, violin*.py, kfold_plotly.py (called from run_scripts) | | network_score | C binary (var_lib_genenet_bnw, built by build.sh), run by run.sh | | structure | C binary (var_lib_genenet_bnw), run by run.sh | | get_kbest_nets/get_kbest_parents | C++ binaries (k-best/src), used via data2netk_poster.sh | | bash, sh | run.sh, examplefilecopy.sh, filecopy.sh, delete_cv_pred.sh, data2netk_poster.sh, buildk_poster.sh | | mkdir, rm, cp, cut, wc, cat | in the shell scripts above | | top, grep | home.php CPU check (vestigial) | ### Known issues found by the tests (PHP 8 / container deployment) 1. The sources mix two data directories: some code writes uploads to `/var/lib/genenet/bnw`, other parts still use `/tmp/bnw` (path_cd.m, run.sh, execute_bn_gom.php, runtime_check.php, examplefilecopy.sh, ...). Under PHP 8 `runtime_check.php` then dies with "Unsupported operand types" and the upload flow breaks. The Guix package patches unify everything on `/var/lib/genenet/bnw`. 2. `home.php` (top level) is a stub that always redirects to bnw.genenetwork.org and runs a `top|grep apache` CPU check; the actual app home page is sourcecodes/home.php. 3. `valid_keyval()` sends its error header after output has started, so under PHP 8 invalid keys are silently accepted instead of redirecting to keyval_error.php. 4. The plotting scripts hardcode a developer python path (/home/jziebart/python/Python-2.7.15/python); the Guix package patches this to the shipped python3.