# Plan: BNW as a GeneNetwork4 endpoint Status: **Phase 0 and Phase 1 done** (the driver exists and matches the web application on BNW's own example; see `test/parity_check.py` and the README's *Headless driver* section). The GN4 endpoint is not started. Written from the code in this repository and from how GN4 already embeds the other engines (`/compute/bulklmm`, `/compute/qtl2`, `/compute/gemma/ravanan`), so that the first commit of real work has a target to hit and a parity test to pass. ## What BNW computes, in terms of files The web application is PHP, but the computation is three C programs driven by one shell script; everything between them is a file in `/var/lib/genenet/bnw//`: ``` continuous_input.txt the data matrix (variables × samples, one header line) ban.txt white.txt edge constraints (banned / forced) parent.txt maxparents per node k.txt k for the k-best search thr.txt edge-posterior threshold run.sh → network_score local scores → k-best/src/data2netk_poster.sh posterior per edge → structure postProbEachEdge.txt structure_input.txt → rm -r temp files gone ``` `structure_input.txt` (the learned adjacency) and `postProbEachEdge.txt` (the posterior for every candidate edge) are what the PHP layer then turns into a graphviz source and an SVG. **Those two files are the API's real output** — the SVG is a rendering of them, and a caller who wants to think about the network should get the edges, not a picture. Verified in the code, worth stating because it shapes the embedding: * the path `/var/lib/genenet/bnw` is hardcoded in `run.sh` and in the PHP, so concurrent runs share one directory. An endpoint must not inherit that: one run, one directory, under a cache root GN4 owns. * the C binaries find their inputs from arguments, so a per-run directory is enough to make them safe; no PHP is needed to run them. * `network_score` links `libRmath` (`pkgconf --libs libRmath`), so the build needs R's math library even though no R is executed. * nothing in the k-best path was seen to seed a RNG; if the search is exhaustive over the k best, results are deterministic and a parity test can compare files byte for byte. **Confirm before promising determinism** — if there is sampling, the test needs a seed or a tolerance. ## Where the work goes: a headless CLI here, a call-out in GN4 The split is deliberate, and it is the same one GN4 uses for every other engine it embeds (`gemma-wrapper`, the Julia BulkLMM driver, the R/qtl2 scan): * **this repository gains a headless driver** — one program that takes a matrix and parameters, runs the three C programs in a private directory, and writes machine-readable results. That is where BNW's knowledge stays: the file layout, the flags, the three-stage pipeline, the C binaries. * **GN4 only calls out.** Its endpoint resolves inputs from the database or a basket, writes them out, runs `$BNW_CMD`, parses what comes back, and reports it. GN4 must not learn the internals of `run.sh`, and it must not gain a second copy of the parameter semantics. * **the dependency lands in `guix.scm` last**, once the driver works and is tested — `guix-bioinformatics/gn/packages/bnw.scm` already pins this repository for the container tests, so the packaging path exists. The contract between the two is therefore the first thing to write down, and the rest of this plan is about making it true. ## What GN4 already has to build on The shape is the same three-step shape as the engines already embedded: | piece | precedent in GN4 | |---|---| | a script that takes files in and writes rows out | `scripts/bulklmm_scan.jl`, `scripts/qtl2_scan.R`, `scripts/rqtl_wrapper.R` | | per-endpoint concurrency gate | `src/compute_gate.rs` (`bulklmm`, `qtl2`, `ravanan`) | | cache keyed by input content | GEMMA's `GEMMA_CACHE_DIR`, `BULKLMM_CACHE_DIR` | | `POST /compute/X` returning rows + `notes` | `/compute/fitqtl`, `/compute/bulklmm` | | `POST /plot/X` returning SVG | `/plot/rqtl`, `/plot/bulklmm`, `/plot/genotypes` | | a "next" link from rows to the plot | `suggest.rs` | So the work is not a new architecture; it is one script, one module, one gate, one route, one plotter — each of which has a worked example in the tree. ## Parameters: what BNW and GN2 actually use Measured, because "look at genenetwork2" turned out to answer only half of it. **GN2 sends no parameters at all.** `gn2/wqflask/external_tools/send_to_bnw.py` builds the payload only: the selected traits' shared samples, comma-joined rows, `;`-terminated — `My_Genenet` posted by `templates/bnw_page.html` to `https://bnw.genenetwork.org/sourcecodes/bn_genenet.php`. The parameters are chosen *after* that, by the user, in BNW's own settings page. **BNW's defaults are in `sourcecodes/create_tiers_gom_part1.php`**, which writes `parent.txt`, `k.txt` and `thr.txt` from a `|`-joined form value (`parent|k|tiers|threshold|key`): | parameter | file | default | note | |---|---|---|---| | maxparents | `parent.txt` | **4** | the form also offers `All`, capped at *variables − 1* | | k (k-best models averaged) | `k.txt` | **1** | 1 means no model averaging | | tiers | `` … `tier_number` | **3** | variable ordering tiers | | edge-posterior threshold | `thr.txt` | **0.5** | passed to `structure` | So an endpoint that is a single call has to expose these four, because the UI step that would otherwise set them is gone. Defaults should be BNW's own, and the reply's `notes` should name the values used — a network learned at threshold 0.5 and one at 0.9 are different answers to the same question. ## The plan ### Phase 0 — a golden reference (half a day) Run the existing BNW container (`gn-machines/genenetwork/containers/bnw-container.scm` via `run-guix-system-container`, web on `:8881`, as `test/run_container_tests.sh` already does) on a **small fixed dataset**, and keep its outputs as golden files: `continuous_input.txt`, `structure_input.txt`, `postProbEachEdge.txt`, and the rendered SVG. Everything later is measured against these, not against a memory of what BNW "should" produce. Use one of the worked examples in `sourcecodes/example_datasets/` so the input is not invented, plus one GN-shaped matrix (probes × BXD strains) because that is the case the endpoint exists for. Decide here: **deterministic or not** (see the RNG note above), and the tolerance if not. ### Phase 1 — the driver this repository owns (2–3 days) A headless program in **this** repository — `bin/bnw` (a shell wrapper around the existing binaries is enough for the first version) — with a contract GN4 can code against without knowing anything else: ``` bnw learn --input matrix.tsv \ --maxparents 4 --k 1 --tiers 3 --threshold 0.5 \ [--ban edges.tsv] [--white edges.tsv] \ [--format json|tsv] [--svg out.svg] \ [--workdir DIR] [--keep] matrix.tsv variables × samples: one header line, one row per sample; column names are the variable names GN4 hands over stdout the result: one edge per line, {from, to, posterior}, plus a header object naming the parameters and the version exit codes 0 ok · 2 bad input (the message names the problem) 3 computation failed (the message is the C program's own) ``` Design points that matter more than the flags: * **a private work directory per invocation** (`--workdir`, default `mktemp -d`), never `/var/lib/genenet/bnw`. Two concurrent networks must not collide, and `--keep` exists so a failed run can be inspected. * **the parameters are arguments, not files**: `parent.txt` and friends were a consequence of the web app's session keys, not of the computation. * **machine-readable by default**; the SVG is a convenience (`--svg`) and GN4's plot endpoint can equally render from the edges. * **`--version` and the parameters echoed in the header**, so a saved result can be reproduced later. * the driver must also work from the container and from a checkout (`var_lib_genenet_bnw/build.sh` builds `network_score` and `structure`; the k-best binaries build with `k-best/src/buildk_poster.sh`), and it must not need PHP, a web server or `/var/www`. Acceptance: byte-identical `structure_input.txt` and `postProbEachEdge.txt` to Phase 0's golden files, and one documented invocation that reproduces the web app's answer. **Done (2026-11-12).** `bin/bnw` exists; on the six continuous traits of BNW's own 8-node example its `structure_input.txt` is byte-identical to the container's, and `test/parity_check.py` checks that plus the input file on every run. Four things the reference run settled: * the input file is header, then a **type line** (1 = continuous, n = n-level discrete), then data -- `network_score` reads the type before the data, which is why the web application's `continuous_input.txt` has one more line than the matrix; * `structure` exits with an uninitialised status while writing a correct file, so it must be judged by its output -- the web application ignores the status and so does the driver; * `get_kbest_nets.cc` used SGI's ``; it now uses `std::unordered_set` with an explicit hasher (specialising `std::hash` for a pointer type is not allowed), and `build.sh` compiles the C with `-std=gnu89` because the sources predate C99; * the driver produces `postProbEachEdge.txt` with per-edge posteriors; the web application's flow leaves none in `/var/lib/genenet/bnw/`, so the posteriors are new information the endpoint can report. Acceptance: byte-identical `structure_input.txt` and `postProbEachEdge.txt` to Phase 0's golden files. ### Phase 2 — `POST /compute/bnw` (2–3 days) Input: enough to build a matrix, in the shapes GN4 already understands — * `dataset` + `probes` (ProbeSet names, max ~30 variables — BNW's own examples are 5–15 nodes and the search is exponential in in-degree), * or `group` + `traits` (published phenotypes), * or a `basket` (the user's own columns), which is the cheapest way to let someone try a network on their own data without uploading anything. Parameters with BNW's own defaults, `notes` saying which were used, plus the gate: a new `compute_gate::bnw()` sized by **variables × samples**, because cost grows with the number of nodes, not with the number of individuals. Cache by digest of (matrix, parameters) in `BNW_CACHE_DIR`, so the same network is not learned twice. GN4's side is a call-out and nothing more: `BNW_CMD` in config (default `bnw`), the matrix written to a temp file, one subprocess, parse the edges. No knowledge of `run.sh`, of the C programs, or of the `` file layout enters GN4 — that all stays in this repository, where it can change without a GN4 release. Return: the edge rows (`from, to, posterior`, sorted by posterior), the structure in a compact form, the parameter values used, and notes stating the method's limits — an edge posterior is not a p-value, a high-posterior edge is not a QTL, and the network is a summary of the *sample's* correlation structure. `next` links: `/plot/bnw` with the posted rows, and `/compute/bnw/features`. ### Phase 3 — `POST /plot/bnw` (1 day) SVG from the posted reply: nodes as GN variable names, edges weighted by posterior, layout either graphviz (`dot`) or BNW's own layout code, so the picture matches what BNW users already recognise. A posterior heatmap as the cheap second figure. Follow the existing plot endpoints: SVG out, peak/edge summary in an `x-plot-summary` header, no mapping re-run. ### Phase 4 — the part GN actually wants: network features (2–3 days) BNW's point in GeneNetwork is not the diagram; it is **derived traits**. From the learned structure, score each module (or each node's neighbourhood) per sample, and hand those scores back as trait values. Then a caller can do: ``` POST /compute/bnw → edges POST /compute/bnw/features → module scores, one value per sample POST /compute/rqtl → map a module score (values go straight in) ``` The feature step should return `samples: [{name, value}]` in the same shape a trait's values arrive in, so it plugs into every mapping endpoint with no translation layer. That is the same trick `/compute/bnw` uses for its matrix input, read backwards. Parity question to settle here: BNW has its own feature/PC-based extraction in the PHP layer; if we re-derive it, the definition has to be named in the `notes` (e.g. "first principal component of each module's member traits") so a user can cite it. ### Phase 5 — tests, docs, memory (1–2 days, interleaved) * an integration test in `tests/` that runs the endpoint on the Phase 0 dataset and compares edges to the golden files (skipping when the binaries are not installed, like the `ravanan`/`bulklmm` tests do); * a plot test asserting SVG output and that the summary names the strongest edge; * `docs/bnw.md`, plus the endpoint's `notes` carrying the same caveats; * record each phase's decisions in `ai-memory` (this repository's store, now initialised) so the next session does not re-derive the file interface. ## Risks, in the order they are likely to bite 1. **Combinatorics.** Structure learning over 30 nodes with maxparents 3 is not the same job as over 12. The endpoint must refuse (or annotate) a request whose variable count puts it in hours, not minutes — the same reasoning that made GEMMA permutations a refusal rather than a queue. 2. **The file interface is a global directory.** Any shortcut that keeps `/var/lib/genenet/bnw` will work in testing and collide in production when two users learn a network at once. 3. **Determinism**, if the k-best search samples. Then golden-file parity is a flake generator and the test needs a seed or a rank-correlation tolerance. 4. **Discrete vs continuous networks.** The GN2 path is continuous (`continuous_input.txt`); the web app also does discrete data through BNW's own discretisation, which needs Octave/`bnt`. Keep the endpoint on the continuous path first, and say so, rather than half-supporting both. 5. **Build weight.** `network_score` needs `libRmath`; the container also carries Python for the cross-validation plotly scripts. The endpoint itself should need only the three binaries — keep the dependency surface that small, so a GN4 install does not grow an Octave. ## Open questions for the owner 1. Which parameters should the endpoint expose, and with which defaults — `maxparents`, `k`, `threshold` only, or also the banned/white edge lists? 2. Is the continuous path enough for the first release? 3. Module features: re-derive them in GN4, or expose BNW's own definition as the reference and match it? 4. Where may the learned networks be cached — per user (a basket-style SQLite file) or shared by content digest? A network is derived state, so a content cache is defensible, but a user may consider their matrix private. 5. ~~Who owns the C binaries' licensing?~~ **Settled: we do.** The C sources and the driver are ours, so shipping them in GN4's profile needs no third-party permission. The repository still has no `LICENSE`/`COPYING` file and the C sources carry no header — worth adding one before GN4 distributes them, so a downstream packager has something to point at.