1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
|
# Plan: BNW as a GeneNetwork4 endpoint
Status: **Phase 0 and Phase 1 done** (the driver exists and matches the web
application on BNW's own example; see `test/parity_check.py` and the README's
*Headless driver* section). The GN4 endpoint is not started. Written from the code in this
repository and from how GN4 already embeds the other engines
(`/compute/bulklmm`, `/compute/qtl2`, `/compute/gemma/ravanan`), so that the
first commit of real work has a target to hit and a parity test to pass.
## What BNW computes, in terms of files
The web application is PHP, but the computation is three C programs driven by
one shell script; everything between them is a file in
`/var/lib/genenet/bnw/<key>/`:
```
<key>continuous_input.txt the data matrix (variables × samples, one header line)
<key>ban.txt <key>white.txt edge constraints (banned / forced)
<key>parent.txt maxparents per node
<key>k.txt k for the k-best search
<key>thr.txt edge-posterior threshold
run.sh <key> <maxp> <k> <thr>
→ network_score <input> <ban> <white> <maxp> <prefix> local scores
→ k-best/src/data2netk_poster.sh <input> <prefix> <k> <maxp> posterior per edge
→ structure <input> postProbEachEdge.txt <tmp> structure_input.txt <thr>
→ rm -r <key> temp files gone
```
`structure_input.txt` (the learned adjacency) and `postProbEachEdge.txt` (the
posterior for every candidate edge) are what the PHP layer then turns into a
graphviz source and an SVG. **Those two files are the API's real output** — the
SVG is a rendering of them, and a caller who wants to think about the network
should get the edges, not a picture.
Verified in the code, worth stating because it shapes the embedding:
* the path `/var/lib/genenet/bnw` is hardcoded in `run.sh` and in the PHP, so
concurrent runs share one directory. An endpoint must not inherit that: one
run, one directory, under a cache root GN4 owns.
* the C binaries find their inputs from arguments, so a per-run directory is
enough to make them safe; no PHP is needed to run them.
* `network_score` links `libRmath` (`pkgconf --libs libRmath`), so the build
needs R's math library even though no R is executed.
* nothing in the k-best path was seen to seed a RNG; if the search is
exhaustive over the k best, results are deterministic and a parity test can
compare files byte for byte. **Confirm before promising determinism** — if
there is sampling, the test needs a seed or a tolerance.
## Where the work goes: a headless CLI here, a call-out in GN4
The split is deliberate, and it is the same one GN4 uses for every other
engine it embeds (`gemma-wrapper`, the Julia BulkLMM driver, the R/qtl2 scan):
* **this repository gains a headless driver** — one program that takes a matrix
and parameters, runs the three C programs in a private directory, and writes
machine-readable results. That is where BNW's knowledge stays: the file
layout, the flags, the three-stage pipeline, the C binaries.
* **GN4 only calls out.** Its endpoint resolves inputs from the database or a
basket, writes them out, runs `$BNW_CMD`, parses what comes back, and reports
it. GN4 must not learn the internals of `run.sh`, and it must not gain a
second copy of the parameter semantics.
* **the dependency lands in `guix.scm` last**, once the driver works and is
tested — `guix-bioinformatics/gn/packages/bnw.scm` already pins this
repository for the container tests, so the packaging path exists.
The contract between the two is therefore the first thing to write down, and
the rest of this plan is about making it true.
## What GN4 already has to build on
The shape is the same three-step shape as the engines already embedded:
| piece | precedent in GN4 |
|---|---|
| a script that takes files in and writes rows out | `scripts/bulklmm_scan.jl`, `scripts/qtl2_scan.R`, `scripts/rqtl_wrapper.R` |
| per-endpoint concurrency gate | `src/compute_gate.rs` (`bulklmm`, `qtl2`, `ravanan`) |
| cache keyed by input content | GEMMA's `GEMMA_CACHE_DIR`, `BULKLMM_CACHE_DIR` |
| `POST /compute/X` returning rows + `notes` | `/compute/fitqtl`, `/compute/bulklmm` |
| `POST /plot/X` returning SVG | `/plot/rqtl`, `/plot/bulklmm`, `/plot/genotypes` |
| a "next" link from rows to the plot | `suggest.rs` |
So the work is not a new architecture; it is one script, one module, one gate,
one route, one plotter — each of which has a worked example in the tree.
## Parameters: what BNW and GN2 actually use
Measured, because "look at genenetwork2" turned out to answer only half of it.
**GN2 sends no parameters at all.** `gn2/wqflask/external_tools/send_to_bnw.py`
builds the payload only: the selected traits' shared samples, comma-joined
rows, `;`-terminated — `My_Genenet` posted by `templates/bnw_page.html` to
`https://bnw.genenetwork.org/sourcecodes/bn_genenet.php`. The parameters are
chosen *after* that, by the user, in BNW's own settings page.
**BNW's defaults are in `sourcecodes/create_tiers_gom_part1.php`**, which writes
`<key>parent.txt`, `<key>k.txt` and `<key>thr.txt` from a `|`-joined form value
(`parent|k|tiers|threshold|key`):
| parameter | file | default | note |
|---|---|---|---|
| maxparents | `<key>parent.txt` | **4** | the form also offers `All`, capped at *variables − 1* |
| k (k-best models averaged) | `<key>k.txt` | **1** | 1 means no model averaging |
| tiers | `<key>` … `tier_number` | **3** | variable ordering tiers |
| edge-posterior threshold | `<key>thr.txt` | **0.5** | passed to `structure` |
So an endpoint that is a single call has to expose these four, because the UI
step that would otherwise set them is gone. Defaults should be BNW's own, and
the reply's `notes` should name the values used — a network learned at
threshold 0.5 and one at 0.9 are different answers to the same question.
## The plan
### Phase 0 — a golden reference (half a day)
Run the existing BNW container (`gn-machines/genenetwork/containers/bnw-container.scm`
via `run-guix-system-container`, web on `:8881`, as `test/run_container_tests.sh`
already does) on a **small fixed dataset**, and keep its outputs as golden
files: `continuous_input.txt`, `structure_input.txt`, `postProbEachEdge.txt`,
and the rendered SVG. Everything later is measured against these, not against a
memory of what BNW "should" produce.
Use one of the worked examples in `sourcecodes/example_datasets/` so the input
is not invented, plus one GN-shaped matrix (probes × BXD strains) because that
is the case the endpoint exists for.
Decide here: **deterministic or not** (see the RNG note above), and the
tolerance if not.
### Phase 1 — the driver this repository owns (2–3 days)
A headless program in **this** repository — `bin/bnw` (a shell wrapper around
the existing binaries is enough for the first version) — with a contract GN4
can code against without knowing anything else:
```
bnw learn --input matrix.tsv \
--maxparents 4 --k 1 --tiers 3 --threshold 0.5 \
[--ban edges.tsv] [--white edges.tsv] \
[--format json|tsv] [--svg out.svg] \
[--workdir DIR] [--keep]
matrix.tsv variables × samples: one header line, one row per sample;
column names are the variable names GN4 hands over
stdout the result: one edge per line, {from, to, posterior},
plus a header object naming the parameters and the version
exit codes 0 ok · 2 bad input (the message names the problem)
3 computation failed (the message is the C program's own)
```
Design points that matter more than the flags:
* **a private work directory per invocation** (`--workdir`, default
`mktemp -d`), never `/var/lib/genenet/bnw`. Two concurrent networks must not
collide, and `--keep` exists so a failed run can be inspected.
* **the parameters are arguments, not files**: `<key>parent.txt` and friends
were a consequence of the web app's session keys, not of the computation.
* **machine-readable by default**; the SVG is a convenience (`--svg`) and GN4's
plot endpoint can equally render from the edges.
* **`--version` and the parameters echoed in the header**, so a saved result
can be reproduced later.
* the driver must also work from the container and from a checkout
(`var_lib_genenet_bnw/build.sh` builds `network_score` and `structure`; the
k-best binaries build with `k-best/src/buildk_poster.sh`), and it must not
need PHP, a web server or `/var/www`.
Acceptance: byte-identical `structure_input.txt` and `postProbEachEdge.txt` to
Phase 0's golden files, and one documented invocation that reproduces the web
app's answer.
**Done (2026-11-12).** `bin/bnw` exists; on the six continuous traits of BNW's
own 8-node example its `structure_input.txt` is byte-identical to the
container's, and `test/parity_check.py` checks that plus the input file on every
run. Four things the reference run settled:
* the input file is header, then a **type line** (1 = continuous, n = n-level
discrete), then data -- `network_score` reads the type before the data, which
is why the web application's `continuous_input.txt` has one more line than the
matrix;
* `structure` exits with an uninitialised status while writing a correct file,
so it must be judged by its output -- the web application ignores the status
and so does the driver;
* `get_kbest_nets.cc` used SGI's `<ext/hash_set>`; it now uses
`std::unordered_set` with an explicit hasher (specialising `std::hash` for a
pointer type is not allowed), and `build.sh` compiles the C with
`-std=gnu89` because the sources predate C99;
* the driver produces `postProbEachEdge.txt` with per-edge posteriors; the web
application's flow leaves none in `/var/lib/genenet/bnw/`, so the posteriors
are new information the endpoint can report.
Acceptance: byte-identical `structure_input.txt` and `postProbEachEdge.txt` to
Phase 0's golden files.
### Phase 2 — `POST /compute/bnw` (2–3 days)
Input: enough to build a matrix, in the shapes GN4 already understands —
* `dataset` + `probes` (ProbeSet names, max ~30 variables — BNW's own examples
are 5–15 nodes and the search is exponential in in-degree),
* or `group` + `traits` (published phenotypes),
* or a `basket` (the user's own columns), which is the cheapest way to let
someone try a network on their own data without uploading anything.
Parameters with BNW's own defaults, `notes` saying which were used, plus the
gate: a new `compute_gate::bnw()` sized by **variables × samples**, because
cost grows with the number of nodes, not with the number of individuals.
Cache by digest of (matrix, parameters) in `BNW_CACHE_DIR`, so the same
network is not learned twice.
GN4's side is a call-out and nothing more: `BNW_CMD` in config (default
`bnw`), the matrix written to a temp file, one subprocess, parse the edges. No
knowledge of `run.sh`, of the C programs, or of the `<key>` file layout enters
GN4 — that all stays in this repository, where it can change without a GN4
release.
Return: the edge rows (`from, to, posterior`, sorted by posterior), the
structure in a compact form, the parameter values used, and notes stating the
method's limits — an edge posterior is not a p-value, a high-posterior edge is
not a QTL, and the network is a summary of the *sample's* correlation structure.
`next` links: `/plot/bnw` with the posted rows, and `/compute/bnw/features`.
### Phase 3 — `POST /plot/bnw` (1 day)
SVG from the posted reply: nodes as GN variable names, edges weighted by
posterior, layout either graphviz (`dot`) or BNW's own layout code, so the
picture matches what BNW users already recognise. A posterior heatmap as the
cheap second figure. Follow the existing plot endpoints: SVG out, peak/edge
summary in an `x-plot-summary` header, no mapping re-run.
### Phase 4 — the part GN actually wants: network features (2–3 days)
BNW's point in GeneNetwork is not the diagram; it is **derived traits**. From
the learned structure, score each module (or each node's neighbourhood) per
sample, and hand those scores back as trait values. Then a caller can do:
```
POST /compute/bnw → edges
POST /compute/bnw/features → module scores, one value per sample
POST /compute/rqtl → map a module score (values go straight in)
```
The feature step should return `samples: [{name, value}]` in the same shape a
trait's values arrive in, so it plugs into every mapping endpoint with no
translation layer. That is the same trick `/compute/bnw` uses for its matrix
input, read backwards.
Parity question to settle here: BNW has its own feature/PC-based extraction in
the PHP layer; if we re-derive it, the definition has to be named in the
`notes` (e.g. "first principal component of each module's member traits") so a
user can cite it.
### Phase 5 — tests, docs, memory (1–2 days, interleaved)
* an integration test in `tests/` that runs the endpoint on the Phase 0 dataset
and compares edges to the golden files (skipping when the binaries are not
installed, like the `ravanan`/`bulklmm` tests do);
* a plot test asserting SVG output and that the summary names the strongest
edge;
* `docs/bnw.md`, plus the endpoint's `notes` carrying the same caveats;
* record each phase's decisions in `ai-memory` (this repository's store, now
initialised) so the next session does not re-derive the file interface.
## Risks, in the order they are likely to bite
1. **Combinatorics.** Structure learning over 30 nodes with maxparents 3 is not
the same job as over 12. The endpoint must refuse (or annotate) a request
whose variable count puts it in hours, not minutes — the same reasoning that
made GEMMA permutations a refusal rather than a queue.
2. **The file interface is a global directory.** Any shortcut that keeps
`/var/lib/genenet/bnw` will work in testing and collide in production when
two users learn a network at once.
3. **Determinism**, if the k-best search samples. Then golden-file parity is a
flake generator and the test needs a seed or a rank-correlation tolerance.
4. **Discrete vs continuous networks.** The GN2 path is continuous
(`continuous_input.txt`); the web app also does discrete data through BNW's
own discretisation, which needs Octave/`bnt`. Keep the endpoint on the
continuous path first, and say so, rather than half-supporting both.
5. **Build weight.** `network_score` needs `libRmath`; the container also
carries Python for the cross-validation plotly scripts. The endpoint itself
should need only the three binaries — keep the dependency surface that
small, so a GN4 install does not grow an Octave.
## Open questions for the owner
1. Which parameters should the endpoint expose, and with which defaults —
`maxparents`, `k`, `threshold` only, or also the banned/white edge lists?
2. Is the continuous path enough for the first release?
3. Module features: re-derive them in GN4, or expose BNW's own definition as
the reference and match it?
4. Where may the learned networks be cached — per user (a basket-style SQLite
file) or shared by content digest? A network is derived state, so a content
cache is defensible, but a user may consider their matrix private.
5. ~~Who owns the C binaries' licensing?~~ **Settled: we do.** The C sources
and the driver are ours, so shipping them in GN4's profile needs no
third-party permission. The repository still has no `LICENSE`/`COPYING`
file and the C sources carry no header — worth adding one before GN4
distributes them, so a downstream packager has something to point at.
|