about summary refs log tree commit diff
path: root/docs/nginx.md
blob: 2d7cafe500bc67387440e790213dbe552c5d9512 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
# Publishing genenetwork4 under a path prefix with nginx

How to put genenetwork4 on `https://vesuvio.pu.ac.ke/gn4/` while the
server itself listens on a loopback port, with no gn-auth: a bearer
token minted from `scripts/gn-auth-stub.py` (see "Authentication"
below).

Three things separate this from "just proxy it":

- genenetwork4 assumes it is served at `/`. Every link it emits is
  **root-relative** (`"href": "/groups"`, `spec_url: "/openapi.json"`,
  `type: "/errors/..."`), so under `/gn4` they point outside the mount
  point unless nginx (or the client) puts the prefix back. See
  "Links the API emits".
- `/compute/*` runs for minutes in a single request; nginx's default
  one-minute read timeout cuts it off mid-computation.
- `POST /get-test-token` is deliberately unusable through a proxy (see
  README, "Behind nginx or in a container"). With no gn-auth, tokens
  come from the stub instead.

## The server behind nginx

```
GENENETWORK4_BIND=127.0.0.1 \
GENENETWORK4_PORT=3000 \
GENENETWORK4_TEST_TOKEN=false \
AUTH_SERVER_URL=http://127.0.0.1:8081/ \
SQL_URI=mysql://webqtlout:webqtlout@127.0.0.1/db_webqtl \
XAPIAN_DB_PATH=/path/to/xapian-index \
SEARCH_ENGINE=native \
genenetwork4
```

`SEARCH_ENGINE=native` is what makes `/search` use `XAPIAN_DB_PATH`. The
default is `forward`, which proxies to GeneNetwork3 at `GN3_URL`
(`127.0.0.1:5003`) -- correct when gn3 is running alongside, and a `502
backend_unreachable` when it is not, which is easy to mistake for the
proxy being misconfigured. See docs/search-engines.md.

- `GENENETWORK4_BIND=127.0.0.1` -- nginx is on the same host, so nothing
  else needs the port. The default is `0.0.0.0` (all interfaces).
- `GENENETWORK4_TEST_TOKEN=false` -- do not register
  `POST /get-test-token` at all. Belt and braces with nginx's own
  refusal below; the setting fails closed (only `true`/`1`/`yes`/`on`
  keep it).
- `AUTH_SERVER_URL` stays **local**: it is where *this process* fetches
  the signing keys from, not a URL that clients need. Nothing about the
  public `/gn4` prefix enters it.
- Any of `SQL_URI`, `XAPIAN_DB_PATH`, `GN3_URL`, `GEMMA_*`, ... as that
  deployment needs; see README.

## nginx

```nginx
server {
    listen 443 ssl;
    http2 on;
    server_name vesuvio.pu.ac.ke;

    ssl_certificate     /etc/nginx/ssl/vesuvio.pu.ac.ke.pem;
    ssl_certificate_key /etc/nginx/ssl/vesuvio.pu.ac.ke.key;

    # genenetwork4 is mounted at /gn4/. The trailing slash on proxy_pass
    # is what translates the prefix away: /gn4/species -> /species.
    # Without it nginx forwards "/gn4/species" and every request 404s.
    location /gn4/ {
        proxy_pass http://127.0.0.1:3000/;

        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP         $remote_addr;

        # The API says it is forwarding, so genenetwork4 refuses
        # POST /get-test-token for these requests even if it is
        # registered (GENENETWORK4_TEST_TOKEN=false above is the
        # stronger measure; this is the one that survives a config
        # mistake).
        #
        # Belt and braces, independent of the application:
        location = /gn4/get-test-token { return 404; }

        # /compute/gemma, /compute/gemma/ravanan and /compute/rqtl are
        # synchronous: one request, one long computation (16-20s on the
        # tiny BXD fixtures, minutes on real datasets, and
        # /compute/gemma/ravanan on Slurm measured 158s with Slurm's
        # default scheduler interval). The default 60s read timeout
        # returns a 504 while the server keeps working.
        proxy_read_timeout 3600;
        proxy_send_timeout 3600;

        # nginx's 1m default is far below what the API accepts: a
        # /compute/* body carries one value per individual plus covariates,
        # and /plot/* is posted a whole mapping result. Keep this at least
        # as large as the application's own 64 MiB limit
        # (BODY_LIMIT_BYTES in src/lib.rs), or nginx refuses first.
        client_max_body_size 64m;

        # No streaming responses: the /compute/*/status/{run_id}
        # endpoints are polled and return accumulated lines as JSON, so
        # buffering is fine and left at its default.

        # Root-relative links in JSON bodies, rewritten to stay under
        # /gn4. See "Links the API emits" below before enabling this.
        sub_filter_types application/json;
        sub_filter_once off;
        sub_filter '"href":"/'        '"href":"/gn4/';
        sub_filter '"spec_url":"/'    '"spec_url":"/gn4/';
        sub_filter '"type":"/errors'  '"type":"/gn4/errors';
        sub_filter '"next":"/'        '"next":"/gn4/';
        sub_filter '"href": "/'       '"href": "/gn4/';
        sub_filter '"spec_url": "/'   '"spec_url": "/gn4/';
        sub_filter '"type": "/errors' '"type": "/gn4/errors';
        sub_filter '"next": "/'       '"next": "/gn4/';
    }

    # /gn4 without the trailing slash: redirect so relative and absolute
    # links agree about where the mount point is.
    location = /gn4 { return 301 /gn4/; }
}
```

### The access log, and who is using the API

The format in `etc/nginx-vesuvio.pu.ac.ke.conf` records the caller's
identity, which the API sends back as a response header:

```nginx
log_format gn4 '$remote_addr - $remote_user [$time_local] "$request" '
               '$status $body_bytes_sent rt=$request_time '
               'rid=$upstream_http_x_request_id '
               'user=$upstream_http_x_gn4_user '
               'ua="$http_user_agent"';
```

`user=` is the token's subject (`pjotrp`, `robw`, ...) -- the API sets
`x-gn4-user` on every authenticated response for exactly this. Without it
a log answers "which address" and never "which user", and per-user
success becomes guesswork from user agents and behaviour; `user=-` marks a
request that carried no valid token (scanners, anonymous reads). With it,
per-user traffic and failure rates are one `awk` away:

```sh
awk '{for(i=1;i<=NF;i++){if($i ~ /^user=/)u=$i; if($i ~ /^[0-9][0-9][0-9]$/)s=$i} print u, s}' \
  /var/log/nginx/gn4.access.log | sort | uniq -c | sort -rn | head
```

Restart or reload nginx after changing the format; the old and new lines
differ in field count, so cut over between rotations.

**Install it *instead of* the distribution's default site.** A
`sites-enabled/default` that certbot has edited to serve this hostname
will win over this block, silently: `sites-enabled/*` is globbed in
alphabetical order and nginx uses the **first** server block whose
`server_name` matches, so `default` takes 443 and the gn4 block is the
one nginx discards. The only sign is a warning at startup or reload --

```
nginx: [warn] conflicting server name "vesuvio.pu.ac.ke" on 0.0.0.0:443, ignored
```

-- and the symptoms are a stock "Welcome to nginx!" page at `/` and
nginx's own `404` (not genenetwork4's JSON) for `/gn4/...`, at HTTP/1.1
even though this block sets `http2 on`. Remove or disable the default
site (`rm /etc/nginx/sites-enabled/default`, or delete its
certbot-managed `server` block for this hostname) and reload; grep the
error log for `conflicting server name` after any change to the vhost
set.

For the `vesuvio.pu.ac.ke/gn4` deployment as configured,
[`etc/nginx-vesuvio.pu.ac.ke.conf`](../etc/nginx-vesuvio.pu.ac.ke.conf) is
the same thing as a complete, drop-in file -- TLS, log format, the `/gn4`
redirect, and the `/get-test-token` refusal. `/compute/*` (R/qtl, GEMMA)
is proxied and gated on a valid token, at the cost of letting any token
holder name server-side paths for R and gemma to read — the operator
checklist in [`docs/www/operator-checklist.md`](www/operator-checklist.md)
covers that trade, what the box needs installed, and how to re-block it.

`http2 on;` needs nginx >= 1.25.1; on older versions use `listen 443
ssl http2;`. The block above (minus the `ssl_*` lines and the TLS
listener) is tested against nginx 1.28: `nginx -t` accepts it, and with
a real genenetwork4 behind it `/gn4/version` answers, the `/gn4`
redirect and the `/gn4/get-test-token` refusal fire, envelopes come back
with `/gn4/...` hrefs, and an error body comes back with
`"type":"/gn4/errors/..."`, `"spec_url":"/gn4/openapi.json"` and
`"next":"/gn4/species"`.

## Links the API emits

Every success envelope carries `next: [{"href": ...}]`, and every error
carries `spec_url`, `type` and sometimes `next`. All of them are
root-relative by design -- genenetwork4 has no idea it is behind a
prefix -- so a client at `https://vesuvio.pu.ac.ke/gn4/species` is told
to call `https://vesuvio.pu.ac.ke/groups`.

Three ways to deal with that, in order of preference:

1. **Give the service its own host** (`https://gn4.example.org/`) and drop
   the prefix. Nothing needs rewriting, and this is what the rest of the
   README assumes.
2. **Rewrite the three keys in nginx**, as in the config above. It is
   narrow on purpose: only the values of `"href"`, `"next"`,
   `"spec_url"` and `"type"` are touched, so the OpenAPI document's
   `"paths"` (which are legitimately root-relative, and which clients
   resolve against the document's own URL) are left alone. Four keys,
   each needing both a compact and a spaced spelling: `"href"` and
   `"next"` are objects in an envelope's `next` array and bare strings
   in an error body, `"spec_url"` is always a bare string, `"type"`
   appears only on errors. `"next":[` (an array) deliberately does not
   match the `"next":"/` filter, so an envelope keeps its
   `href`-rewritten objects and nothing else.
   `sub_filter_types application/json;` is required -- nginx rewrites
   only `text/html` by default -- and `sub_filter_once off;` because an
   envelope carries several links. Three consequences worth knowing: the
   server's `serde` output has no space after the colon, so the compact
   spellings are the ones that actually fire; nginx drops
   `Content-Length` and sends these responses chunked; and the rewrite is
   textual, so a `detail` string that happened to contain `"href":"/`
   would be rewritten too (nothing does: `notes`/`detail` come from this
   server, never from database text or request input).
3. **Let clients re-root**: prepend `/gn4` to any `href`, `spec_url` or
   `type` that starts with `/`. Correct without touching the server,
   but every client has to know to do it.

`/openapi.json` itself is served fine through the proxy. Its `paths` are
root-relative, which is correct OpenAPI: a tool that reads the document
from `https://vesuvio.pu.ac.ke/gn4/openapi.json` resolves them under
`/gn4` on its own. A tool that hard-codes `paths` against the host root
will call the wrong place -- that is the same prefix problem, one level
further out.

## Authentication without gn-auth

The stub (`scripts/gn-auth-stub.py`, see README "Tests") is enough for a
deployment that only needs bearer tokens checked:

```
scripts/gn-auth-stub.py --token-file /run/gn4/public.token
```

and set `AUTH_SERVER_URL=http://127.0.0.1:8081/` (the stub's default) on
genenetwork4. Notes:

- Keep `AUTH_SERVER_URL` **internal**. Do not route `/gn4/` or anything
  else on the public site to the stub: its signing key
  (`test/data/test_rsa_private.pem`) is committed to this repository, so
  anyone who can fetch its JWKS can mint tokens genenetwork4 accepts.
  The JWKS only needs to be reachable from the genenetwork4 process.
- Tokens minted this way carry `iss` = `http://127.0.0.1:8081/`, which
  is what genenetwork4 compares against `AUTH_SERVER_URL` verbatim
  (trailing slash included). They are valid only while that
  `AUTH_SERVER_URL` is unchanged, and only against this server.
- A token minted without `--grant-resource-id` is a **public-data-only**
  token: public datasets read normally, every non-public one is a 403,
  dataset metadata stays visible. `--expires-in 30d` gives it a month;
  nothing invalidates it but its own `exp`, and it keeps working as long
  as the stub is reachable when genenetwork4 refreshes its copy of the
  keys (cached an hour).
- Hand the token out of band. `POST /gn4/get-test-token` is not usable
  through the proxy: nginx connects from `127.0.0.1`, which would make
  every client look local, so genenetwork4 refuses any request carrying
  `X-Forwarded-For`/`Forwarded`/`X-Real-IP` -- and the config above sets
  them. That refusal is the point, not an obstacle; keep it.

Swapping in a real gn-auth later changes only `AUTH_SERVER_URL` (and
whatever `iss` gn-auth mints -- it must be the same string). See
docs/gn-auth.md.

## Checking it

From a client, plainly:

```
curl -s https://vesuvio.pu.ac.ke/gn4/version
curl -s https://vesuvio.pu.ac.ke/gn4/openapi.json | head -c 200
TOKEN=$(cat /run/gn4/public.token)
curl -s -H "Authorization: Bearer $TOKEN" \
     "https://vesuvio.pu.ac.ke/gn4/species" | head -c 200
curl -s -H "Authorization: Bearer $TOKEN" \
     "https://vesuvio.pu.ac.ke/gn4/groups/BXD/traits?limit=3"
```

On the box itself, two things trip up the obvious `curl -H "Host: ..."
https://localhost`:

* TLS validates the **hostname in the URL**, not the `Host:` header, which
  only arrives after the handshake -- so `https://localhost` fails against
  a certificate for `vesuvio.pu.ac.ke` no matter what `Host:` says. Use
  `--resolve` to connect to loopback while still verifying the real name.
* use the name that is actually on the certificate. Check it rather than
  assuming (`openssl s_client -connect 127.0.0.1:443 -servername
  vesuvio.pu.ac.ke </dev/null | openssl x509 -noout -text | grep -A1
  'Subject Alternative Name'`); a mismatch here and a mismatch in
  `server_name` are the same symptom -- the default server answers instead,
  so `/` shows whatever else is configured on the host and `/gn4/...` is a
  404 from nginx rather than from genenetwork4.

```
R="--resolve vesuvio.pu.ac.ke:443:127.0.0.1"
curl -sSI $R https://vesuvio.pu.ac.ke/       | head -3   # 302 -> /gn4/
curl -sSI $R https://vesuvio.pu.ac.ke/gn4    | head -3   # 301 -> /gn4/
curl -s   $R https://vesuvio.pu.ac.ke/gn4/version        # the version envelope
curl -s $R -o /dev/null -w '%{http_code}\n' -X POST \
     https://vesuvio.pu.ac.ke/gn4/get-test-token         # 404: not registered
curl -s $R -o /dev/null -w '%{http_code}\n' -X POST \
     https://vesuvio.pu.ac.ke/gn4/compute/correlation    # 401: proxied, token needed
```

A 502 instead means nginx is fine and genenetwork4 is not listening on
`GENENETWORK4_PORT`; check `curl -s http://127.0.0.1:3000/version` directly.

The `/version` reply also names the git revision and build date, which
is the quickest way to confirm the proxy is talking to the process you
think it is.

To check the link rewriting, follow a `next` from a body:

```
curl -s -H "Authorization: Bearer $TOKEN" \
     https://vesuvio.pu.ac.ke/gn4/groups/BXD | grep -o '"href": "[^"]*"' | head -3
```

Each one should start with `/gn4/`. Without the `sub_filter` lines they
start with `/`, and resolve outside the mount point. The same check on
an error (`/gn4/species/nonexistent`) shows `/gn4/errors/...`,
`/gn4/openapi.json` and `/gn4/species`.

## When a request looks like it hangs

`/compute/gemma` and `/compute/rqtl` really do run for minutes: they are
documented as synchronous, single-request computations. Pass `?id=` and
poll `/gn4/compute/gemma/status/{run_id}` (or `/compute/rqtl/status/`)
on a second connection to see progress; both are ordinary JSON
responses, so nothing about the proxy needs to change for them.
`/compute/gemma/ravanan` additionally waits on ravanan (and on Slurm,
when `RAVANAN_BATCH_SYSTEM=slurm-api` is set), which is where the
two-minute orders of magnitude come from -- see docs/ravanan-slurm.md.