summary refs log tree commit diff
path: root/topics/systems/update-production-checklist.gmi
blob: 0e3fb48eef0f9c4a406bec38722110e4705de6ca (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
# Update production checklist


# Tasks

* [X] Install underlying Debian
* [X] Get guix going
* [ ] Check database
* [ ] Check gemma working
* [ ] Check global search
* [ ] Check authentication
* [ ] Check sending E-mails
* [ ] Make sure info.genenetwork.org can reach the DB
* [ ] Backups

The following are at the system level

* [ ] Make journalctl presistent
* [ ] Update certificates in CRON
* [ ] Run trim in CRON

# Install underlying Debian

For our production systems we use Debian as a base install. Once installed:

* [X] set up git in /etc and limit permissions to root user
* [X] add ttyS0 support for grub and kernel - so out-of-band works
* [X] start ssh server and configure not to use with passwords
* [X] start nginx and check external networking
* [X] mount old root
* [X] Clean up /etc/profile (remove global profile.d loading)
* [ ] set up E-mail routing

It may help to mount the old root if you have it. Now it is on

```
mount /dev/sdd2 /mnt/old-root/
```

# Get Guix going

* [X] Mount bind /gnu on a large partition
* [X] Move /gnu/store to larger partition
* [X] Install Guix daemon
* [X] Update Guix daemon and setup in systemd (if necessary)
* [X] Make available in /usr/local/guix-profiles

Next move the /gnu store to a large partion and hard mount it in /etc/fstab with

```
/export2/gnu /gnu none defaults,bind 0 0
```

We can bootstrap with the Debian guix package (though I prefer the guix-install.sh script these days, mostly because it is more modern).

=> https://guix.gnu.org/manual/en/html_node/Binary-Installation.html


Run guix pull

```
guix pull --url=https://codeberg.org/guix/guix  -p ~/opt/guix-pull
```

Use that also to install guix in /usr/local/guix-profiles

```
guix package -i guix -p /usr/local/guix-profiles/guix
```

and update the daemon in systemd accordingly. After that I tend to remove /usr/bin/guix

The Debian installer configures guix. I tend to remove the profiles from /etc/profile so people have a minimal profile.

# Check database

* [X] Install mariadb
* [ ] Recover database
* [ ] Test permissions
* [ ] Mariadb update my.cnf

Basically recover the database from a backup is the best start and set permissions. We usually take the default mariadb unless production is already on a newer version - so we move to guix deployment.

On tux02 mariadb-10.5.8 is running. On Debian it is now 10.11.11-0+deb12u1, so we should be good. On Guix is 10.10 at this point.

```
apt-get install mariadb-server
```

Next unpack the database files and set permissions to the mysql user. And (don't forget) update the /etc/mysql config files.

Restart mysql until you see:

```
mysql -u webqtlout -p -e "show databases"
+---------------------------+
| Database                  |
+---------------------------+
| 20081110_uthsc_dbdownload |
| db_GeneOntology           |
| db_webqtl                 |
| db_webqtl_s               |
| go                        |
| information_schema        |
| kegg                      |
| mysql                     |
| performance_schema        |
| sys                       |
+---------------------------+
```

=> topics/systems/mariadb/mariadb.gmi

## Recover database

We use borg for backups. First restore the backup on the PCIe. Also a test for overheating!


# Check sending E-mails

The swaks package is quite useful to test for a valid receive host:

```
swaks --to testing-my-server@gmail.com --server smtp.network
=== Trying smtp.network:25...
=== Connected to smtp.network.
<-  220 mailrouter8.network ESMTP NO UCE
 -> EHLO tux04.network
<-  250-mailrouter8.network
<-  250-PIPELINING
<-  250-SIZE 26214400
<-  250-VRFY
<-  250-ETRN
<-  250-STARTTLS
<-  250-ENHANCEDSTATUSCODES
<-  250-8BITMIME
<-  250-DSN
<-  250 SMTPUTF8
 -> MAIL FROM:<root@tux04.network>
<-  250 2.1.0 Ok
 -> RCPT TO:<pjotr2020@thebird.nl>
<-  250 2.1.5 Ok
 -> DATA
<-  354 End data with <CR><LF>.<CR><LF>
 -> Date: Thu, 06 Mar 2025 08:34:24 +0000
 -> To: pjotr2020@thebird.nl
 -> From: root@tux04.network
 -> Subject: test Thu, 06 Mar 2025 08:34:24 +0000
 -> Message-Id: <20250306083424.624509@tux04.network>
 -> X-Mailer: swaks v20201014.0 jetmore.org/john/code/swaks/
 ->
 -> This is a test mailing
 ->
 ->
 -> .
<-  250 2.0.0 Ok: queued as 4157929DD
 -> QUIT
<-  221 2.0.0 Bye                                                                                                                             === Connection closed with remote host
```

An exim configuration can be

```
dc_eximconfig_configtype='smarthost'
dc_other_hostnames='genenetwork.org'
dc_local_interfaces='127.0.0.1 ; ::1'
dc_readhost=''
dc_relay_domains=''
dc_minimaldns='false'
dc_relay_nets=''
dc_smarthost='smtp.network'
CFILEMODE='644'
dc_use_split_config='false'
dc_hide_mailname='false'
dc_mailname_in_oh='true'
dc_localdelivery='maildir_home'
```

And this should work:

```
swaks --to myemailaddress --from john@network --server localhost
```

# Backups

* [ ] Create an ibackup user.
* [ ] Install borg (usually guix version)
* [ ] Create a borg passphrase