about summary refs log tree commit diff
path: root/RELEASE_NOTES.md
blob: 12cc2e771a44874c0b635d3d5088d88796be3609 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
# BNW release notes

Changes since `e9a320d0` (*Update home.php*, the last pre-Guix upstream
commit), summarized for users and deployers.  34 commits, 2021-10-11 →
2026-10-03.

## Deployment / Guix packaging

* **Guix system container and VM** (`7f460113`): BNW deploys as a Guix
  system container (`gn-machines/genenetwork/containers/bnw-container.scm`)
  and VM.  The package definition lives in
  `guix-bioinformatics/gn/packages/bnw.scm`, pinned to commits of this
  repository and fetched from git.genenetwork.org.
* **Container test cycle** (`7b6a8ccf`, `f76f79b8`): `make` builds the
  container from the pinned origin, starts it through the
  run-guix-system-container daemon, runs the HTTP test suite, and stops
  it again; `make run` leaves it up, `make clean` removes artifacts.
  Checkout paths are overridable (`base`, `GUIX_BIOINF`, ...).
* **Anubis AI-bot firewall support** (in gn-machines): the web UI is
  challenge-protected, while the GeneNetwork workflow endpoints, the
  static assets and the .htm tutorial pages are exempt — the full
  modeling flow runs without ever hitting a challenge.
* **Pinned container SSH keys** (`10ae4902`, `74f922e3`): host and admin
  keys live outside the repository in the building user's `$HOME`, so
  every container presents the same SSH host key.

## GeneNetwork (GN2) integration

* **Single-POST handshake, XSS fixed** (`8380798b`, `75761501`): GN2
  posts `My_Gennenet` (commas = columns, semicolons = rows) and the
  conversion now happens **server-side** in PHP, returning the
  prepared-data page in one request.  The previous release echoed the
  raw payload into a JavaScript string (`genenet_str = "<?php echo ... ?>"`),
  an XSS hole on public deployments — that echo is gone.
* **Accept the GN2 field spelling** (`3616c964`): GN2's template posts
  `My_Genenet` (one 'n'), while BNW historically read `My_Gennenet`.
  Both spellings are accepted now.
* **Sanitize trait names with spaces** (`18dcbf7b`): GN2 trait display
  names often contain spaces ("Age at death"); BNW's file format does
  not allow them (the octave parsers split on whitespace), so spaced
  names corrupted the header and structure learning died with
  "A(0,_): subscripts must be ... integers" in mk_adj_mat.  The GN2
  handshake now replaces whitespace inside name-row names with
  underscores before the separator conversion.
* GN2 may also supply its own session key via `$_POST["My_key"]` and
  later deep-link to `executionprogress_default.php?My_key=<key>`.

## UX: feedback during structure learning

* **Status page instead of a blank browser** (`1de3cbe0`): the
  `execute_bn_gom.php` runner used to output nothing for the whole
  run (~12 s+).  It now immediately shows a status page — "Structure
  learning is running.  Forwarding to the network diagram when
  done…" — flushed before the work starts.
* **Click-here fallback** (`1de3cbe0`): the status page links straight
  to the network diagram in case the automatic forward is blocked.
* **Pipeline failures are shown, not blank** (`e18ba415`): when
  `run.sh`/`run_octave` fails, the error (command, exit code, stdout,
  stderr) is rendered on the status page with a link back to the
  upload page, instead of an uncaught RuntimeException killing the
  page mid-flight.

## Bug fixes (PHP 8 compatibility and robustness)

* Guard `valid_keyval()`/`valid_input()` against null input (PHP 8.1
  deprecation) (`4825da8e`); guard fopen() results before
  fwrite/fprintf/fclose (`9cae7ea1`); guard arithmetic on non-numeric
  file/shell values (`a1f64bda`).
* Fix undefined `$TextinFile`/`$keyval` in bn_after_upload_gom.php
  (`9f820106`) and an orphaned fwrite() that crashed runtime_check.php
  on PHP 8 (`55909cd3`).
* Misc fixes: missing key check (`47e8ac23`), path fix (`6254c4b4`),
  variable moved to first use (`02c5eef4`).
* Shell scripts made executable (`f7d65f6a`) — the php pages invoke
  them by name.

## Build system

* **pkgconf finds libRmath** (`ea3796d3`): `build.sh` no longer
  hard-codes paths; under Guix the package passes the
  rmath-standalone library path explicitly.
* **`runTerminalCommand()` wrapper** (`d1e617f3`, `a08f24c6`): a thin
  proc_open wrapper (with argument escaping) replaces scattered
  shell_exec/system calls and raises RuntimeExceptions on failure.

## Vendored third-party code removed

* **Vendored javascript libraries and prebuilt binaries removed from
  the repository** (`00e5b05f`): the js libs (jquery, d3, cytoscape,
  FileSaver, font-awesome, ...) come from Guix packages and are
  symlinked in at package build time; the prebuilt .o/.dll/.so
  binaries are recompiled from source by `build.sh`.  111 files /
  ~36k lines of vendored code dropped.

## Tests

* `test/bnw_http_tests.py`: 26 checks covering the full user workflow
  (static pages → key validation → upload → octave preparation → C
  structure learning → network rendering), all six workflow.php
  examples, the GN2 handshake (server-side conversion, no JS echo,
  template spelling through the legacy `/sourcecodes/` action path),
  and the asset downloads.
* `test/check_container_tools.sh`: verifies every unix tool BNW calls
  externally exists in the container environment.

## Internal

* Ignore-list hygiene for build artifacts (`0a39376c`, `3c3f6c0d`).
* Test-path documentation: no hardcoded factory paths; checkout roots
  and per-checkout overrides are environment variables (`4dd78e7b`).
* `make` no longer exports a local-source hook; the tests always build
  the origin pinned in guix-bioinformatics (`dd8dc307`) — push to
  gn-bnw main (or update the pin) before testing source changes.

* **Self-locating binaries in run.sh** (`f9d5c411`): `run.sh` cd'ed
  to `/var/lib/genenet/bnw` and ran `./network_score` etc. relative to
  that directory, depending on the activation having copied the
  binaries there — on production that copy was incomplete and every
  structure-learning run died with "./network_score: No such file or
  directory".  run.sh now locates the binaries next to itself in the
  package store item
  (`$(dirname $0)/../var_lib_genenet_bnw`), so the flow works
  regardless of what `/var/lib/genenet/bnw` contains.
* **`/version.php` deployment-validation endpoint** (via the Guix
  package): serves JSON with the package version and the exact
  gn-bnw commit the deployment was built from
  (`{"name":"bnw","version":"1.0.0","commit":"..."}`).  After a
  redeploy, `curl https://<host>/version.php` shows what is running
  — no more inferring the version from error messages.

## Input validation and informative failures (this release)

The GN2 entry point (`bn_genenet.php`) now validates the posted payload
before any file is written or octave runs, and fails with a plain-English
problem list instead of a deep octave crash.  Found by an exhaustive
small-payload search (`test/fuzz_bnw.py`, 75 cases over sizes, data
types, special characters, missing data and separators):

- header-only payload (traits with no sample values loaded in GN2)
  -> "No sample data rows were received ..." (was: "Could not open
  <key>continuous_input.txt" deep in octave)
- ragged rows, single variable, single sample, non-numeric cells,
  empty payload, all-NA data -> each with a specific message
- a variable whose every entry is NA -> rejected (prep crashed on it)
- every row carrying an NA -> rejected with an explanation (after
  prep's row removal nothing remains: "fscanf: invalid stream number")
- duplicate variable names (same trait from two data sets) -> renamed
  with a note instead of crashing
- CRLF line endings and redundant row separators (`;` plus newline)
  are normalized instead of corrupting the data files
- `runtime_check.php` (`exe_time`) no longer fatals with "Unsupported
  operand types: int + string" when prep did not produce its tables
- the structure-learning failure page now appends plain-English hints
  for the recurring octave failure modes (unlaid-out/cyclic networks,
  memory, missing prep files)

Validated end-to-end on production: the 75-case input-space sweep
produces zero crashes and zero blank pages; the header-only case that
prompted this work (traits with no sample values loaded in GeneNetwork)
returns the explanation above.  One genuine limitation surfaced: with
exactly two variables the scorer can prefer a cyclic network that the
layout engine cannot draw (`sources(0): subscripts must be ...`); the
failure page explains this — the deeper fix lives in the C scoring code.

`test/bnw_http_tests.py` is now 39 checks (t22–t33 added for the
validation layer); `test/fuzz_bnw.py` is the parameter-space fuzzer.