1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
|
# Setting Up or Migrating Production Across Machines
## Tags
* type: documentation, docs, doc
* status: in-progress
* assigned: fredm
* priority: undefined
* keywords: migration, production, genenetwork
* interested-parties: pjotrp, zachs
## Introduction
Recent events (Late 2024 and early 2025) have led to us needing to move the production system from one machine to the other several time, due to machine failures, disk space, security concerns, and the like.
In this respect, a number of tasks rise to the front as necessary to accomplish for a successful migration. Each of the following sections will detail a task that's necessary for a successful migration.
## Copy Over Auth Database
We need to synchronise the authorisation database. We can copy this over from the production system, or the backups
* TODO: Indicate where the backups for the auth database are here!
Steps (flesh out better):
* Extract backup (or copy from existing production system)
* Stop the (new) container (if it's running)
* Backup the (new) container's auth-db file (
* Place the auth db file in the correct place in the container's filesystem:
* Backup existing secrets
* Login to the `/auth/admin/dashboard` of the auth server (e.g. https://cd.genenetwork.org/auth/admin/dashboard)
* If client with the CLIENT_ID in the secrets exists
* 1. update the uris for that client, if it doesn't exist, create an entirely new client and replace both the CLIENT_ID and CLIENT_SECRET in the secrets file.
* 2. Click on the "Change Secret" button and generate a new secret. Replace the secret in the secrets file with the newly generated secret
* If client with the CLIENT_ID in the secrets DOES NOT exist, register a new client, setting up the appropriate URIs and endpoints, and then add/replace both the CLIENT_ID and CLIENT_SECRET in the secrets file.
* Restart (new) container
## Set Up the Database
=> /topics/systems/restore-backups Extract the latest database from the backups.
=> /topics/deploy/installation Configure MariaDB according to this document.
## Set Up the File System
* TODO: List the necessary directories and describe what purpose each serves. This will be from the perspective of the container — actual paths on the host system are left to the builders choice, and can vary wildly.
* TODO: Prefer explicit binding rather than implicit — makes the shell scripts longer, but no assumptions have to be made, everything is explicitly spelled out.
The container(s) need access to various files and directories from the host system in order to work correctly.
Filesystem bindings could be linked to wildly different paths on different physical host machines, therefore, we shall examine the bindings from the point of view of the paths within the container, rather than forcing a particular file system layout on the host systems themselves.
### File System Bindings
#### /var/genenetwork
This binding must be READ/WRITE within the container.
The purpose is to hold varying files that are specific to the genenetwork system(s). Examples of the files are:
* "gn-meta" and "synteny" files for GN3
* genotype files
* session files for various systems (GN2, gn-uploader, etc.)
#### /var/lib/acme
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/lib/redis
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/lib/virtuoso
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/log
* **TODO**: Describe this: Writable? What is this binding for?
#### /etc/genenetwork
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/lib/xapian
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/lib/genenetwork/sqlite/gn-auth
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/lib/genenetwork/sqlite/genenetwork3
* **TODO**: Describe this: Writable? What is this binding for?
#### /run/mysqld
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/lib/gn-docs.git
* **TODO**: Describe this: Writable? What is this binding for?
#### /opt/gn/tmp
* **TODO**: Describe this: Writable? What is this binding for?
#### /export/data/virtuoso/
* **TODO**: Describe this: Writable? What is this binding for?
#### /export/data/gn-docs
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/genenetwork/sessions
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/lib/genenetwork/uploader
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/lib/genenetwork/sqlite/gn-uploader
* **TODO**: Describe this: Writable? What is this binding for?
#### /var/lib/genenetwork/gn-guile
* **TODO**: Describe this: Writable? What is this binding for?
## Redis
We currently (2025-06-11) use Redis for:
* Tracking user collection (this will be moved to SQLite database)
* Tracking background jobs (this is being moved out to SQLite databases)
* Tracking running-time (not sure what this is about)
* Others?
We do need to copy over the redis save file whenever we do a migration, at least until the user collections and background jobs features have been moved completely out of Redis.
## Container Configurations: Secrets
* TODO: Detail how to extract/restore the existing secrets configurations in the new machine
## Build Production Container
* TODO: Add notes on building
* TODO: Add notes on setting up systemd
## NGINX
* TODO: Add notes on streaming and configuration of it thereof
## SSL Certificates
* TODO: Add notes on acquisition and setup of SSL certificates
## DNS
* TODO: Migrate DNS settings
|
