1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
|
# BNW
This contains the code for the Bayesian Network Webserver (BNW).
The master branch will be the stable version of the code for running with GeneNetwork. The BNW website is compbio.uthsc.edu/BNW; the GeneNetwork version is at: bnw.genenetwork.org.
There are a couple of files in the info_files directory that provide descriptions of the BNW code and other information.
There are five branches.
1) master: This is the version of BNW that should be loaded to GeneNetwork.
2) matlab_original (compbio.uthsc.edu/BNW_1.01): This is functionally identical to the original BNW code. The only difference is that unused files were removed.
3) octave_version (compbio.uthsc.edu/BNW_1.02): This should perform similarly to the original version of BNW. However, instead of Matlab, Octave is used and a couple of new features were added.
4) genenet6_final_1.3: This is the final version of BNW on the GENENET6 server.
5) genenet8_initial_1.3: This is the working version of BNW on GENENET8 when we transitioned to that server.
A couple of notes about this version.
- This version writes files created by users running BNW to /var/lib/genenet/bnw.
- The structure learning package needs to be installed in /var/lib/genenet/bnw. The files in var_lib_genenet_bnw need to be copied to /var/lib/genenet/bnw and build.sh should be run there.
- There is a python environment on GENENET8 that is used by BNW. This is in bnw-env, but I am not sure how well this will transition to other servers.
## Headless driver: `bin/bnw`
The web application is not needed to run BNW's computation. `bin/bnw` does the
same three-stage pipeline -- `network_score`, the k-best search, `structure` --
in a private working directory, and writes machine-readable results:
```sh
# build the C programs once (needs gcc, g++ and R's standalone math library)
var_lib_genenet_bnw/build.sh
# learn a network: variables x samples, header line first
bin/bnw learn --input test/golden/six_traits_matrix.tsv
```
```
# bnw 0.1.0 variables=6 samples=200 maxparents=4 k=1 tiers=3 threshold=0.5
from to posterior in_structure
Trait2 Trait1 1 1
Trait2 Trait4 1 1
...
```
Every candidate edge is reported with its posterior, and `in_structure` says
whether it survived `--threshold`. Options: `--maxparents` (default 4),
`--k` (1), `--tiers` (3), `--threshold` (0.5) -- BNW's own defaults from
`sourcecodes/create_tiers_gom_part1.php`; `--ban`/`--white` for edge
constraints; `--format json`; `--svg FILE` for a graphviz drawing;
`--workdir DIR` and `--keep` to inspect a run. Exit codes: 0 ok, 2 bad input,
3 the computation failed, 77 the binaries are not installed.
### Parity with the web application
`test/parity_check.py` runs the driver on the six continuous traits of BNW's
own 8-node example and compares against files the *web application* produced
(the `test/golden/` fixtures, captured from the `bnw-container` on
2026-11-12 by posting to `bn_genenet.php` and running `execute_bn_gom.php`):
```
ok continuous_input.txt matches the web application's
ok structure_input.txt matches the web application's (5 edge(s))
ok stdout reports the same 5 edge(s)
PARITY OK
```
That check is also how the input format was pinned down: `continuous_input.txt`
is a header line of variable names, then a **type line** (1 = continuous, n =
n-level discrete, one integer per variable), then the data. The type line is
not optional -- `network_score` reads it before the data -- and it is the
reason the web application's copy has one more line than the matrix.
## Testing (HTTP functional tests)
`test/bnw_http_tests.py` tests a *running* BNW instance over plain
HTTP requests (no browser, no selenium needed). It covers:
* static pages and assets,
* the 3-letter key round-trip,
* data upload (`bn_genenet.php`) and octave input preparation
(`run_prep_input`) for a discrete (8-node) and a continuous
(5-node) example dataset,
* the structure learning pipeline (`executionprogress_default.php`,
`execute_bn_gom.php` -> `run.sh` -> `network_score`, `structure`,
the k-best scripts),
* network rendering (`graphviz_structure.php`),
* ALL SIX worked examples linked from `workflow.php`
(example2, example_sci, example_chr2_spleen, example_chl,
example_time_series, examplecar15node -> `layout_svg_no.php`).
Run it against a local instance:
```sh
BASE_URL=http://127.0.0.1:8881 python3 test/bnw_http_tests.py # 20 checks
```
### GeneNetwork (GN2) integration
GN2 posts `My_Gennenet` to `bn_genenet.php`: the payload uses commas
as column separators and semicolons as row separators, plus a
3-letter session key (`My_key`, empty = random). The original
release echoed the raw payload into a JavaScript string
(`genenet_str = "<?php echo $_POST['My_Gennenet'] ?>"`), which is an
XSS hole on public deployments. It is now fixed server-side: the
conversion (`,` -> TAB, `;` -> newline) happens in PHP and the
prepared-data page is returned in ONE request. This is backward
compatible for GN2 -- same POST, better response; no GN2 changes are
required. GN2 can additionally link straight to
`executionprogress_default.php?My_key=<key>` after posting to show
the computed network.
The tests cover the handshake without a browser: t18 posts
`My_Gennenet` and asserts the server-side conversion (and that the
JS echo is gone), t19 posts the converted `searchkey` directly (the
legacy second leg), t20 runs structure learning + rendering on the
GN2 data.
A second script, `test/check_container_tools.sh`, verifies that every
unix tool BNW calls externally exists in the (container) environment;
run it inside the container.
`test/run_container_tests.sh` does the whole cycle: it builds the
system container (gn-machines/genenetwork/services/bnw-container.scm,
needing the guix-bioinformatics and gn-machines channels on the load
path), starts it through the run-guix-system-container daemon, waits
for the web server, runs the HTTP test suite and the tool check, and
stops the container again (kept running on failure, or with --keep):
```sh
test/run_container_tests.sh # or --keep to inspect afterwards
```
The same cycle is available as `make` targets from the repo root:
```sh
make # build from this checkout and run the container tests
make run # leave the container running afterwards (web on :8881)
make clean # remove test artifacts (logs, pycache, /tmp VM image)
# and kill any leaked test VM
```
`make` runs the container tests, which build the BNW package from the
origin pinned in `guix-bioinformatics/gn/packages/bnw.scm` (fetched
from git.genenetwork.org) -- so push your changes to gn-bnw main (or
update that pin) before testing them. If
`make` is not on your PATH, run it via `guix shell make -- make test`.
### Container keys
The container pins an SSH host key and ships the admin key for
password-less root login. Both live **outside the repository**, in
the building user's `$HOME` (see the **Keys** section of
`gn-machines/genenetwork/services/cgit-README.md` for the background):
```sh
ssh-keygen -t ecdsa -f ~/ssh-container-key -N "" -C "container host key"
ssh-keygen -t ed25519 -f ~/hello-container-key -N "" -C "container admin key"
```
`bnw-container.scm` fetches the keys from `$HOME` at build time;
building without them fails with a missing-file error. The test
runner ssh-es into the container with `$HOME/hello-container-key`
(override with the `CLIENT_KEY` variable). Private keys never enter
any repository.
### Package definition location
The BNW package (and the container patch machinery) lives on the
**`BNW` branch** of guix-bioinformatics (`gn/packages/bnw.scm`).
Until that branch is merged, a working tree built from another branch
needs the file restored from the branch:
```sh
cd guix-bioinformatics
git show BNW:gn/packages/bnw.scm > gn/packages/bnw.scm
git show BNW:gn/packages/bnw-container-paths.patch \
> gn/packages/bnw-container-paths.patch
```
Without it the build silently falls back to the unpatched upstream
BNW, whose octave preparation scripts fail under the container's
octave (the give-away is uploads landing under a random three-letter
key instead of the posted one).
Memory footprint measured in a Guix system container (8-node example
run, php-fpm + nginx + octave + shepherd):
* idle: ~58 MB PSS
* peak: ~210 MB PSS (octave loading the Bayes Net Toolkit and
running the search; grows with the number of nodes)
## Unix tools called externally by the BNW sources
Scanned from `shell_exec`/`system`/`proc_open` calls in the PHP code,
`system()` calls in the octave run scripts, script shebangs and
hardcoded absolute paths:
| tool | called from |
|-------------------------------|---------------------------------------------------------|
| octave (octave-cli) | shebangs of sourcecodes/run_scripts/* (input prep, settings, structure learning, cross validation, plotting) |
| dot (graphviz) | php network layout pages (`/usr/bin/dot -Tplain`), octave `createSVG` (`/usr/bin/dot -Tsvg`) |
| python3 | plotly figure generation: cv_plotly.py, ts_plotly.py, violin*.py, kfold_plotly.py (called from run_scripts) |
| network_score | C binary (var_lib_genenet_bnw, built by build.sh), run by run.sh |
| structure | C binary (var_lib_genenet_bnw), run by run.sh |
| get_kbest_nets/get_kbest_parents | C++ binaries (k-best/src), used via data2netk_poster.sh |
| bash, sh | run.sh, examplefilecopy.sh, filecopy.sh, delete_cv_pred.sh, data2netk_poster.sh, buildk_poster.sh |
| mkdir, rm, cp, cut, wc, cat | in the shell scripts above |
| top, grep | home.php CPU check (vestigial) |
### Known issues found by the tests (PHP 8 / container deployment)
1. The sources mix two data directories: some code writes uploads to
`/var/lib/genenet/bnw`, other parts still use `/tmp/bnw`
(path_cd.m, run.sh, execute_bn_gom.php, runtime_check.php,
examplefilecopy.sh, ...). Under PHP 8 `runtime_check.php` then
dies with "Unsupported operand types" and the upload flow breaks.
The Guix package patches unify everything on
`/var/lib/genenet/bnw`.
2. `home.php` (top level) is a stub that always redirects to
bnw.genenetwork.org and runs a `top|grep apache` CPU check; the
actual app home page is sourcecodes/home.php.
3. `valid_keyval()` sends its error header after output has started,
so under PHP 8 invalid keys are silently accepted instead of
redirecting to keyval_error.php.
4. The plotting scripts hardcode a developer python path
(/home/jziebart/python/Python-2.7.15/python); the Guix package
patches this to the shipped python3.
|