aboutsummaryrefslogtreecommitdiff
path: root/doc/GUIX-Reproducible-from-source.org
blob: fffa95713518c877486f2ac26e8c245ebc4bfb01 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
#+TITLE: Installing GeneNetwork services with GNU Guix

* Table of Contents                                                     :TOC:
 - [[#introduction][Introduction]]
 - [[#binary-deployment-through-gnu-guix][Binary deployment through GNU Guix]]
   - [[#quick-installation-recipe][Quick installation recipe]]
   - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]]
   - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]]
   - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]]
   - [[#step-4-install-and-run-gn2][Step 4: Install and run GN2]]
 - [[#from-source-deployment][From source deployment]]
 - [[#create-archive][Create archive]]
 - [[#source-deployment][Source deployment]]
   - [[#run-your-own-copy-of-gn2][Run your own copy of GN2]]
   - [[#set-up-nginx-port-forwarding][Set up nginx port forwarding]]
 - [[#source-deployment-and-other-information-on-reproducibility][Source deployment and other information on reproducibility]]
   - [[#update-to-recent-guix][Update to recent guix]]
   - [[#install-gn2][Install GN2]]
   - [[#run-gn2][Run GN2]]

* Introduction

Large system deployments tend to get very complex. In this document we
explain the GeneNetwork deployment system which is based on GNU Guix
(see Pjotr's [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]] and the main [[README.org]] doc).

* Binary deployment through GNU Guix
** Quick installation recipe

This is a recipe for quick and dirty installation of GN2. For
convenience everything is installed as root, though in reality only
GNU Guix has to be installed as root. I tested this recipe on a fresh
install of Debian 8.3.0 (in KVM) though it should work on any modern
Linux distribution (including CentOS).

Note that GN2 consists of an approx. 5 GB installation including
database. If you use a virtual machine we recommend to use at least
double.

** Step 1: Install GNU Guix

Fetch the GNU Guix binary from [[https://www.gnu.org/software/guix/download/][here]] (middle panel) and follow
[[https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html][instructions]]. Essentially, download and unpack the tar ball (which
creates directories in /gnu and /var/guix), add build users and group
(Guix builds software as unpriviliged users) and run the Guix daemon
after fixing the paths (also known as the 'profile').

Once you have succeeded, you have to [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-the-key][set the key]] (getting permission
to download binaries from the GNU server) and you should be able to
install the hello package using binary packages (no building)

#+begin_src bash
export PATH=~/.guix-profile/bin:$PATH
guix pull
guix package -i hello --dry-run
#+end_src

Which should show something like

: The following files would be downloaded:
:   /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10
:   ...
#+end_src

means binary installs.  The actual installation command of 'hello' is

#+begin_src bash
guix package -i hello
hello
  Hello, world!
#+end_src

If you actually see things building it means that Guix is not yet
properly installed and up-to-date, i.e., the key is missing or you
need to do a 'guix pull'. Press Ctrl-C to interrupt.

If you need more help we have another writeup in [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#binary-installation][guix-notes]]. To get
rid of the locale warning see [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-locale][set-locale]].

** Step 2: Checkout the GN2 git repositories

To fixate the software dependency graph GN2 uses git repositories of
Guix packages. First install git if it is missing

#+begin_src bash
guix package -i git
export GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt
#+end_src

check out the git repositories (gn-deploy branch)

#+begin_src bash
cd ~
mkdir genenetwork
cd genenetwork
git clone --branch gn-deploy https://github.com/genenetwork/guix-bioinformatics
git clone --branch gn-deploy --recursive https://github.com/genenetwork/guix guix-gn-deploy
cd guix-gn-deploy
#+end_src bash

To test whether this is working try:

#+begin_src bash
#+end_src bash

** Step 3: Authorize the GN Guix server

GN2 has its own GNU Guix binary distribution server. To trust it you have
to add the following key

#+begin_src scheme
(public-key
 (ecc
  (curve Ed25519)
  (q #11217788B41ADC8D5B8E71BD87EF699C65312EC387752899FE9C888856F5C769#)
 )
)
#+end_src

by pasting it into the command

#+begin_src bash
guix archive --authorize
#+end_src

and hit Ctrl-D.

Now you can use the substitute server to install GN2 binaries.

** Step 4: Install and run GN2

Since this is a quick and dirty install we are going to override the
GNU Guix package path by pointing the package path to our repository:

#+begin_src bash
rm /root/.config/guix/latest
ln -s ~/genenetwork/guix-gn-deploy/ /root/.config/guix/latest
#+end_src

Now check whether you can find the GN2 package with

#+begin_src bash
env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ guix package -A genenetwork2
  genenetwork2    2.0-a8fcff4     out     gn/packages/genenetwork.scm:144:2
#+end_src

(ignore the source file newer then ... messages, this is caused by the
/root/.config/guix/latest override).

And install with

#+begin_src bash
env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ \
  guix package -i genenetwork2 \
  --substitute-urls="http://guix.genenetwork.org"
#+end_src

Note: the order of the substitute url's may make a difference in speed
(put the one first that is fastest for your location and time of day).

Note: if your system starts building or gives an error it may well be
Step 3 did not succeed. The installation should actually be smooth at
this point and only do binary installs (no compiling).

After installation you should be able to run genenetwork2 after updating
the Guix suggested environment vars. Check the output of

#+begin_src bash
guix package --search-paths
export PYTHONPATH="/root/.guix-profile/lib/python3.8/site-packages"
export R_LIBS_SITE="/root/.guix-profile/site-library/"
#+end_src

and copy-paste the listed exports into the terminal before running:

#+begin_src bash
genenetwork2
#+end_src

It will complain that the database is missing. See the next section on
running MySQL server for downloading and installing a MySQL GN2
database. After installing the database restart genenetwork2 and point
your browser at [[http://localhost:5003/]].

End of the GN2 installation recipe!

* From source deployment

GNU Guix allows for [[https://github.com/pjotrp/guix-notes/blob/master/REPRODUCIBLE.org][reproducible deployment]] based on a checked out
Guix repository - use gn-stable for that:

#+begin_src sh   :lang bash
mkdir genenetwork
cd genenetwork
git checkout https://github.com/genenetwork/guix.git gn-stable-guix
git checkout https://github.com/genenetwork/guix-bioinformatics.git
#+end_src

Next build guix (and run) following the instructions in [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][guix-notes]].

Once that is done we can add the guix-bioinformatics path with

: env GUIX_PACKAGE_PATH=../guix-bioinformatics command

So

#+begin_src sh   :lang bash
#+begin_src sh   :lang bash
gn-stable-guix$ env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix package -A genenetwork
genenetwork1    1.0-d622c803b   out     ../guix-bioinformatics/gn/packages/bioinformatics.scm:163:2
genenetwork2    2.0-9e9475053   out     ../guix-bioinformatics/gn/packages/bioinformatics.scm:215:2
#+end_src sh   :lang bash

Install with

#+begin_src sh   :lang bash
gn-stable-guix$ env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix package -i genenetwork2
#+end_src sh   :lang bash

* Create archive

: env GUIX_PACKAGE_PATH=../../genenetwork/guix-bioinformatics/ ./pre-inst-env guix archive --export -r genenetwork2 > guix_gn2-2.0-9e9475053.nar


* Source deployment

This section gives a more elaborate instruction for installing GN2
from source.

First execute above 4 steps:

   - [[#step-1-install-gnu-guix][Step 1: Install GNU Guix]]
   - [[#step-2-checkout-the-gn2-git-repositories][Step 2: Checkout the GN2 git repositories]]
   - [[#step-3-authorize-the-gn-guix-server][Step 3: Authorize the GN Guix server]]
   - [[#step-4-install-and-run-gn2-][Step 4: Install and run GN2 ]]


** Run your own copy of GN2

At some point you may want to fix the source code. Assuming you have
Guix and Genenetwork2 installed (as described above) clone the GN2
repository from https://github.com/genenetwork/genenetwork2.

Copy-paste the paths into your terminal (mainly so PYTHON_PATH and
R_LIBS_SITE are set) from the information given by guix:

: guix package --search-paths

Inside the repository:

: cd genenetwork2
: ./bin/genenetwork2

Will fire up your local repo http://localhost:5003/ using the
settings in ./etc/default_settings.py. These settings may
not reflect your system. To override settings create your own from a copy of
default_settings.py and pass it into GN2 with

: ./bin/genenetwork2 $HOME/my_settings.py

and everything *should* work (note the full path to the settings
file). This way we develop against the exact same dependency graph of
software.

If something is not working, take a hint from the settings file
that comes in the Guix installation. It sits in something like

: cat ~/.guix-profile/lib/python3.8/site-packages/genenetwork2-2.0-py2.7.egg/etc/default_settings.py

** Set up nginx port forwarding

nginx can be used as a reverse proxy for GN2. For example, we want to
expose GN2 on port 80 while it is running on port 5003. Essentially
the configuration looks like

#+begin_src js
    server {
        listen 80;
        server_name test-gn2.genenetwork.org;
        access_log  logs/test-gn2.access.log;

        proxy_connect_timeout       3000;
        proxy_send_timeout          3000;
        proxy_read_timeout          3000;
        send_timeout                3000;

        location / {
            proxy_set_header   Host      $http_host;
            proxy_set_header   Connection keep-alive;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Host $server_name;
            proxy_pass         http://127.0.0.1:5003;
        }
}
#+end_src js

Install the nginx webserver (as root)

: guix package -i nginx

The nginx example configuration examples can be found in the Guix
store through

: ls -l /root/.guix-profile/sbin/nginx
: lrwxrwxrwx 3 root guixbuild 66 Dec 31  1969 /root/.guix-profile/sbin/nginx -> /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/sbin/nginx

Use that path

: ls /gnu/store/g0wrcl5z27rmk5b52rldzvk1bzzbnz2l-nginx-1.8.1/share/nginx/conf/
:   fastcgi.conf            koi-win             scgi_params
:   fastcgi.conf.default    mime.types          scgi_params.default
:   fastcgi_params          mime.types.default  uwsgi_params
:   fastcgi_params.default  nginx.conf          uwsgi_params.default
:   koi-utf                 nginx.conf.default  win-utf

And copy any relevant files to /etc/nginx.  A configuration file for
GeneNetwork (reverse proxy) port forwarding can be found in the source
repository under ./etc/nginx-genenetwork.conf. Copy this file to /etc
(still as root)
: cp ./etc/nginx-genenetwork.conf /etc/nginx/

Make dirs

: mkdir -p /var/spool/nginx/logs

Add users

: adduser nobody ; addgroup nobody

Run nginx

: /root/.guix-profile/sbin/nginx -c /etc/nginx/nginx-genenetwork.conf -p /var/spool/nginx

* Source deployment and other information on reproducibility

See the document [[GUIX-Reproducible-from-source.org]].

** Update to recent guix

We now compile Guix from scratch.

Create, install and run a recent version of the guix-daemon by
compiling the guix repository you have installed with git in
step 2. Follow [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][these]] steps carefully after

: cd ~/genenetwork/guix-gn-deploy

Make sure to restart the guix daemon and run guix client from this
directory.

** Install GN2

Reinstall genenetwork2 using the new tree

#+begin_src bash
env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ ./pre-inst-env guix package -i genenetwork2 --substitute-urls="http://guix.genenetwork.org https://mirror.guixsd.org"
#+end_src bash

Note the use of ./pre-inst-env here!

Actually, it should be the same installation as in step 4, so nothing
gets downloaded.

** Run GN2

Make a note of the paths with

#+begin_src bash
./pre-inst-env guix package --search-paths
#+end_src bash

or this should also work if guix is installed

#+begin_src bash
guix package --search-paths
#+end_src bash

After setting the paths for the server

#+begin_src bash
export PATH=~/.guix-profile/bin:$PATH
export PYTHONPATH="$HOME/.guix-profile/lib/python3.8/site-packages"
export R_LIBS_SITE="$HOME/.guix-profile/site-library/"
export GUIX_GTK3_PATH="$HOME/.guix-profile/lib/gtk-3.0"
export GI_TYPELIB_PATH="$HOME/.guix-profile/lib/girepository-1.0"
export XDG_DATA_DIRS="$HOME/.guix-profile/share"
export GIO_EXTRA_MODULES="$HOME/.guix-profile/lib/gio/modules"
#+end_src bash

run the main script (in ~/.guix-profile/bin)

#+begin_src bash
genenetwork2
#+end_src bash

will start the default server which listens on port 5003, i.e.,
http://localhost:5003/.

OK, we are where we were before with step 4. Only difference is that we
used our own compiled guix server.