1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
|
This repository houses Guix configuration for GeneNetwork machines and
containers.
The git repo lives at https://git.genenetwork.org/gn-machines/
* GeneNetwork development container
The GeneNetwork development container is currently run on /tux02/. It runs
continuous integration and continuous deployment services for
genenetwork2, genenetwork3 and several other associated projects.
To build and install the container, you will need the
[[https://gitlab.com/genenetwork/guix-bioinformatics][guix-bioinformatics]] and [[https://git.systemreboot.net/guix-forge/][guix-forge]] channels. Once these channels are
pulled and available, on /tux02/, run
#+BEGIN_SRC shell
$ ./genenetwork-development-deploy.sh
#+END_SRC
/tux02/ is configured with a systemd service to run this
container. Restart it.
#+BEGIN_SRC shell
$ sudo systemctl restart genenetwork-development-container
#+END_SRC
* Virtuoso container for tux01 production
The virtuoso container is run on /tux01/ production. It runs virtuoso
alone.
To build and install the container, you will need the
guix-bioinformatics channel. Once guix-bioinformatics is pulled and
available, on /tux01/, run
#+begin_src shell
$ ./virtuoso-deploy.sh
#+end_src
/tux01/ is configured with a systemd service to run this
container. Restart it.
#+begin_src shell
$ sudo systemctl restart virtuoso-container
#+end_src
* Secure virtuoso authentication
In containers containing virtuoso instances, it is important to secure
authentication by changing default user passwords and disabling
unnecessary users. See [[https://issues.genenetwork.org/topics/systems/virtuoso][virtuoso gemtext documentation]] on passwords for
more details.
* Getting a Shell into the Container
You can get a shell into the container with something like:
#+BEGIN_SRC sh
sudo guix container exec 89086 /run/current-system/profile/bin/bash --login
#+END_SRC
When you start the container, you can get a shell into the container using the ~nsenter~ command. You will need the process ID of the container, which you can see on container startup or on your can get with something like:
#+BEGIN_SRC sh
ps -u root -f --forest | grep -A4 '/usr/local/bin/genenetwork-development-container'
#+END_SRC
Where =/usr/local/bin/genenetwork-development-container= is the path used for
invoking (running) the system container.
Once you have a listing, get the PID for the =shepherd= process. You could pipe
the output of the command above to ~grep 'shepherd'~ to ease your search.
That will give you output of the form:
#+BEGIN_EXAMPLE
11869 pts/3 00:00:00 shepherd
#+END_EXAMPLE
From the guix [/operating-system/ Reference](https://guix.gnu.org/manual/en/html_node/operating_002dsystem-Reference.html)
under the *packages* option, the list of packages installed under the global
profile are found in */run/current-system/profile*, for example:
#+BEGIN_SRC sh
/run/current-system/profile/ls /gnu/store
#+END_SRC
to list the files under */gnu/store*
With that knowledge, we can now get a shell using ~nsenter~ as follows:
#+BEGIN_SRC sh
sudo nsenter -a -t 11869 /run/current-system/profile/bin/bash \
--init-file /run/current-system/profile/etc/profile
#+END_SRC
which will give you a bash shell with the ~PATH~ environment variable setup
correctly to give you access to all packages in the global profile.
* Troubleshooting Tips
** Use Profiles
When troubleshooting, we need to be using the correct profile that has all the necessary dependencies. Use a [[https://ci.genenetwork.org/channels.scm][channels]] file to set up a profile. An example of a channel that was fixed at Python 3.9:
#+begin_src scheme
(list (channel
(name 'gn-bioinformatics)
(url "https://gitlab.com/genenetwork/guix-bioinformatics")
(branch "master")
(commit
"9939feb61ea29881d42628bc58a43886f7da6573"))
(channel
(name 'guix-forge)
(url "https://git.systemreboot.net/guix-forge/")
(branch "main")
(introduction
(make-channel-introduction
"0432e37b20dd678a02efee21adf0b9525a670310"
(openpgp-fingerprint
"7F73 0343 F2F0 9F3C 77BF 79D3 2E25 EE8B 6180 2BB3")))))
#+end_src
Activate a profile by:
#+begin_src bash
export GUIX_PROFILE=~/.guix-extra-profiles/genenetwork
. $GUIX_PROFILE/etc/profile
#+end_src
Double-check to confirm that you are using the correct channel using =guix describe=. The use of profiles is well documented [[https://issues.genenetwork.org/topics/guix-profiles][here]]---it should match your channels.scm file.
** View Logs
When troubleshooting our containers, all our log files are located in "/export2/guix-containers/genenetwork-development/var/log":
#+begin_src sh
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork2.log
tail /export2/guix-containers/genenetwork-development/var/log/cd/genenetwork3.log
#+end_src
Note that to be able to view log files, you have to have root permissions.
** Back-ups
When running borg, you can run into the following error:
#+begin_src text
root@tux02:/export3/local/home/bonfacem# borg list /export3/backup/tux01/borg/borg-tux01/
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)
#+end_src
To fix it:
: borg config repo id
: rm ~/.config/borg/security/REPO_ID/manifest-timestamp
: borg delete --cache-only REPO
|
