1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
|
# Meeting Notes
## 2024-08-09
* @shelby figure out Claude Sonnet stuff: NOT DONE, main focus was on the paper
* @shelby planning session for next work and tasks for Priscilla.
* @shelby format output for ingested paper so that we can test the RAG engine
* @shelby run RAGAs against the queries already in the system to perform comparison with new papers
* @bmunyoki implementation code work to edit Rif + WIki SQL n RDF data
* @bmunyoki @shelby group paper on dissertation to target Arxiv
* @bmunyoki and @alex help Alex deploy gn-guile code on tux02
* @bmunyoki review UI code editor work
* @alex address comments in UI work
* @felix @fred push RQTL bundles to uploader
* @felix look for means to fix metadata challenge ie. trouble associating data we upload and metadata that provides descriptions. The metadata doesn't meeting requirements.
* @felix figure out how to fix large data uploads ie. most data sets are large e.g. 45GB. Uploader cannot handle these large files.
* @felix @jnduli programming learning: started building a web server to learn backend using Flask.
* @felix (@bmunyoki / @alex) learning emacs so that he figures out how to track times
* @jnduli fix group creation bug in gn-auth
* @jnduli edit rif metadata using gn3
* @jnduli update documentation for gn-auth setup
* @jnduli investigate more bugs related to gn-auth
Note: When setting up sync between @jnduli and @felix, add @bmunyoki too.
## 2024-08-02
* DONE: @bmunyoki virtuos and xapian updated in prod
* @bmunyoki code work to edit Rif + WIki SQL n RDF data: WIP, we have desired API, but we need to implement code.
* NOT DONE: @bmunyoki group paper on dissertation to target Arxiv
* DONE: @bmunyoki fix case insensitivity in Xapian search
* DONE: @jnduli review Alex patches
* DONE: @bmunyoki: updated gn2 and gn3 on git.genetwork server. Shared QA code with @shelby on a special branch.
* @bmunyoki @jnduli: fixed minor bug on xapian reflected with stemming.
* @shelby figure out Claude Sonnet stuff: NOT DONE, main focus was on the paper
* IN PROGRESS: @shelby edit paper with @pjtor
* @shelby planning session for next work and tasks for Priscilla.
* @shelby use RAGAS to test R2R with the new papers (follow up on the ingestion of papers tasks)
* @shelby and @boni to discuss R2R and interfacing with Virtuoso: deprioritized, we'll figure out interfacing with R2R. Implementation to happen later.
* DONE: @jnduli get up to speed on gn-auth
* @alex have an instance of gn-guile running on production: Code in prod, but needs to liase with Boni to get this working.
* @jgart getting genecup and rshiny containers to run as normal users instead of root users. May use libvirts APIs; or podman/docker as normal user; or rewriting the services as guix home services: system container doesn't have work around this, there's no work around. Because guix by default needs root to run as a system container. We also need sudo since at root level we define our system containers in a systemd that needs to be run as root. Why systemd? Systemd no one needs to run this.
### Meeting with Sevila on Masters Papers
- mainly stylistic changes provided.
- provide an email explaining how long ethical review took, so that he follows up on unexpected delays.
- met up with Dr Betsy, once done with defences in October (hopefully), and Boni may get his degree before graduation next year, to facilitate Boni applying for PhD.
### Guix Root Container
- With docker, to prevent the need for sudo, we usually create a docker group, and add users that need to run this to this group. Can this ahppen in guix?
- Guix has a guix group. Why haven't we done this??? @jgart and @boni
## 2024-07-26
Plan for this week:
* NOT DONE, needs a meeting: @bmunyoki virtuoso and xapian are up-to-date in prod. Boni doesn't have root access in production, so coordination with Fred and Zach is causing delays.
* Apis design DONE, actual CODE incomplete: @bmunyoki update RIF+WIKI on SQL and RDF from gn2 website
* DONE: @bmunyoki and @shelby review dissertation for Masters
* DONE, needs to review new changes: @bmunyoki and @jgart to review patches for `genecup` and `rshiny`.
* @bmunyoki and @jnduli to review patches for markdown parser
* DONE, patches sent. @alexm add validation and document to markdown parser.
* DONE: @shelby ingest ageing data to RAG, 10% left to complete.
* DONE: @shelby do another round for editting on the AI paper
* IN PROGRESS: @shelby RAG engine only works with OpenAI, figure out Claude Sonnet integration
* IN PROGRESS: @jnduli get up to speed on gn-auth
* @jgart enabling acme service in genecup and rshiny containers.
* @jnduli and @bmunyoki to attempt to get familiar with R2R
Nice to have:
* @bmunyoki fix CI job for GN transformer database i.e. instead of checksums just run full job once per month: scheme script created that dumps the example files, next step is to create Gexp that runs this script. Bandwidth constraints.
## 2024-07-23
### LLM Meeting (@shelby+@bmunyoki)
* There's no clear way of ingesting human-readable data with context into the RAG Graph from RDF.
* What specific graph should we ingest into the RAG Graph from RDF? @bmunyoki suggested RIF, PubMed Metadata. We'll figure this out.
* @bonfacem recommended: Much better to work with SPARQL than directly with TTL files.
* We've uploaded rdf triples, yet they loose their strength as the RAG system is not undergirded with a knowledge graph. @bonfacem should read the following for more context and should reach out to @shelby on how to move forward with SPARQL more concretely:
=> https://r2r-docs.sciphi.ai/cookbooks/knowledge-graph#r2r-knowledge-graph-configuration
* We need to test the knowledge graph backend of R2R to see how feasible it is to use with the existing data (RDF).
* Fahamu just stored the object and lost the subject+predicate
* Loop in Alex.
## 2024-07-19
Plan for this week:
* DONE: @jgart getting `genecup` app to run in a guix container i.e. `gunicorn service` should then run `genecup`, similar to how gn2 and gn-uploader work. Patches sent to Boni, include `genecup` and `rshiny` and the container patches are tested.
* @jgart enable acme certificates for `genecup` container: Should just enable a single form, let's use arun's email since its what we use for all our services. Reverse proxy happens inside the container. Add a comment explaining that this shouldn't be standard python set up.
* INPROGRESS: @bmunyoki virtuoso and xapian are up-to-date in prod:
* NOT DONE: @bmunyoki update RIF+WIKI on SQL and RDF from gn2 website
* INPROGRESS: @bmunyoki fix CI job for GN transformer database i.e. instead of checksums just run full job once per month: scheme script created that dumps the example files, next step is to create Gexp that runs this script. Bandwidth constraints.
* @bmunyoki and @shelby review dissertation for Masters: @bonz needs to send updated version. Also reviewed another masters by Johannes.
* ON HOLD: @alexm rewrite UI code using htmx
* INPROGRESS: @alexm address review comments in markdown parser. Api endpoints are getting reimplemented. Needs to add validation and documentation and send v2 patches for review.
* DONE: @shelby compile ingesting 500 more papers into RAG engine
* @shelby ingesting ageing research into the RAG engine: diabetes reseach is ingested, ageing will be done later.
* NOT DONE: @shelby RAG engine only works with OpenAI, figure out Claude Sonnet integration
* DONE: @shelby @bmunyoki @alexm to define the problem with RDF triple stores
* DONE: @jnduli finish up on RIF update
* IN PROGRESS: @jnduli get up to speed on gn-auth
AOB
* RAG engine uses R2R for the integration. It would be great if we could integrate this into guix. @shelby will send @jgart the paper on how we use the RAG.
## 2024-07-12
Plan for this week:
* @shelby use Claude Sonnet with R2R RAG engine with 1000 papers and fix bugs: 500 papers ingested into R2R, remaining with 500.
* @shelby final run through for paper 1 before Pjotr's review. DONE, configurations fixed. New repo gnai that contains the results and will contain R2R stuff.
* NOT DONE: @shelby and @bmunyoki review dissertation paper for Masters
* @shelby @bmunyoki @alexm to define the problem with RDF triple stores
* @alexm integrate the markdown parser: DONE, patches sent to Boni
* @alexm rewrite UI code using htmx: NOT DONE
* @bmunyoki investigate why xapian index isn't getting rebuilt: DONE
* @bmunyoki investigate discrepancies between wiki and rif search: DONE, get this to prod to be tested
* @jnduli update the generif_basic table from NCBI: IN PROGRESS.
* @jnduli blog post of preference for documentation: DONE.
We have qa.genenetwork.com. We need to have this set up to `qa.genenetwork.com/paper1` so that we always have the system that was used for this. How?
Nice to Haves
* @bmunyoki Nice to have tag for paper1: Fix this with Boni and get done later on/iron them out then.
* @bmunyoki fix CI job that transforms gn tables to TTL: Move this to running a cron job once per month instead of
## 2024-06-24
Plan for this Week:
* CANCELED: @bmunyoki Remove boolean prefixes from search where it makes sense.
* DONE: @bmunyoki GeneWiki + GeneRIF search in production. Mostly needs to be run in prod to see impact.
* DONE: @jnduli Children process termination when we kill the main index-genenetwork script
* CANCELED: @bmunyoki Follow up on getting virtuoso child pages in production
* IN PROGRESS @alexm push endpoints for editting and making commits for markdown files
* DONE: @all Reply to survey from Shelby
* DONE: @jnduli Fix JS import orders (without messing up the rest of Genenetwork)
* DONE: @jnduli fix search results when nothing is found
* CANCELED: @jnduli test out running guix cron jobs locally
* NOT DONE: @Jnduli mention our indexing documentation in gn2 README
Note: For qa.genenetwork.com, we chose to pause work on this until papers are done.
Review for last week
* DONE: @bmunyoki rebuild guix container with new mcron changes
* WIP: @jnduli attempts to make UI change that shows all supported keys in the search: Blocked because our JS imports aren't ordered correctly and using `boolean_prefixes` means our searches don't work as we'd expect.
* WIP: @bmunyoki create an issue with all the problems experienced with search and potential solutions. Make sure it has replication steps, and plans for solutions. Issue was created but we need to get a better understanding for how cis and trans searches work.
* TODO: @bmunyoki and @jnduli genewiki indexing: PR for WIKI indexing is completed, but we didn't test it out due to the outage caused by RAM and our script. We don't have a way to easily instrument how much RAM our process uses and how to kill the process.
* DONE: @bmunyoki demoes and documents how to run and test guix cron job for indexing
* DONE: @bmunyoki trains @jnduli on how to review patchsets from emails
* DONE: @jnduli Follow up notes on setting up local index-genenetwork search
* DONE: @alexm handling with graduation, AFK
* TODO: @bmunyoki follow up with Rob to makes sure he tests search after everything is complete: He got some feedback and Rob is out of Town but wants RIF and Wiki search by July 2nd.
Nice to haves:
* TODO: minor: bonfacem makes sure that mypy/pylint in CI runs against the index-genenetwork script.
* TODO: @bmunyoki follow up how do we make sure that xapian prefix changes in code retrigger xapian indexing?
- howto: xapian prefix changes, let's maintain a hash for the file and store it in xapian
- howto: for RDF changes, since we have ttl files, if this ever changes we trigger the script. It's also nice to be able to automatically also load up data to virtuoso if this file changes.
## 2024-06-21
### Outage for 2024-06-20
What happened?
2024-06-19: Dev experienced intermittent ssh connections, and got logged out
2024-06-19: Dev assumed this wasn't related to the server problems and clocked out
2024-06-20: Another team mate experienced problems accessing git.genenetwork.org and reported this to the genenetwork sphere channel
2024-06-20: We noted that CI and CD services were also down
2024-06-20: We assumed it could be a DNS issue and followed up with our providers??
2024-06-20: Realized it was because the server was out of RAM. Killing processes resolved the issue
What can we learn from this?
* If we experience network problems, communicate to other team members.
* Our gunicorn processes are expensive each taking 17GB.
* We don't have monitoring and alerting for our server resources.
* Shouldn't OOM have helped with this?
* How much memory/resources do the scripts we run use? This can help us know the impact before running in production.
* If we start multiple processes from python, similar to the index-genenetwork script, does sending SIGTERM or SIGKILL kill the children processes or will they remain as orphans? Note: there were orphans, so we should investigate ways of killing the script next time completely.
How do we prevent something similar from happening in the future?
* Ask if anything our server is slow and attempt to inspect this. Add documentation for quick bash scripts to run for this.
* Reduce the amount of gunicorn processes to reduce their memory footprints. How did we end up with the no of processes we currently have? What impact will reducing this have on our users?
* Attempt to get an estimate memory footprint for `index-genenetwork` and use this to determine when it's safe to run the script or not. This can even end up integrated into the cron job.
* Create some alerting mechanism for sane thresholds that can be send to a common channel/framework e.g. when CPU usage > 90%, memory usage > 90% etc. This allows someone to be on the look out in case something drastic needs to be taken.
* Python doesn't kill child processes when SIGTERM is used. This means when testings, we were creating more and more orphaned processes. Investigate how to propagate the SIGTERM signal to all children processes.
=> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate
## 2024-06-18
### Agenda
* Last week review: DONE, made good progress on what we planned. We need to make sure we're more aware of the TODOs we had.
* Plan for this week: DONE, LET'S GOOO!
* Reviewing patches: added to week's goals.
* Search bugs discussions: Boni will create an issue with all details and we'll plan further on how to attack this.
If we have a plan for the week, and something comes up that breaks our plan:
* Are we aware that it broke our plans?
* Communicate the impact this may have on the plan.
### Plan for Week
* TODO: @bmunyoki rebuild guix container with new mcron changes
* TODO: @jnduli attempts to make UI change that shows all supported keys in the search
* TODO: @bmunyoki create an issue with all the problems experienced with search and potential solutions. Make sure it has replication steps, and plans for solutions.
* TODO: @bmunyoki follows up to make sure RDF changes are visible in production and fix issues that come up
* TODO: @bmunyoki and @jnduli genewiki indexing
* TODO: @bmunyoki demoes and documents how to run and test guix cron job for indexing
* TODO: @bmunyoki trains @jnduli on how to review patchsets from emails
* TODO: @jnduli attempts to add stronger types to index-genenetwork script, to make it explicit that we're using MonadicDicts
* TODO: @jnduli Documentation improvements for GN2 and GN3 and auth? None done last week.
* TODO: @jnduli Follow up notes on setting up local index-genenetwork search
Nice to haves:
* TODO: minor: bonfacem makes sure that mypy in CI runs against the index-genenetwork script.
* TODO: @bmunyoki improve search documentation and fix bugs in the frontend: binary term search doesn't work as expected
* TODO: @bmunyoki follow up with Rob to makes sure he tests search after everything is complete
* TODO: @bmunyoki follow up how do we make sure that xapian prefix changes in code retrigger xapian indexing?
## 2024-06-11
### Agenda
* Local checks to do before PRs:
In gn3, make sure to run:
> pylint main.py setup.py wsgi.py setup_commands tests gn3 scripts
> TODO jnduli run this: mypy --show-error-codes .
DONE: led to a PR that fixed all mypy errors in index-genenetwork
> pytest -k unit_test
* Set up new DB before sync + fixing any problems that occur: DONE, no errors after Jnduli set up new DB.
### Generif Indexing
* Probeset data exists in SQL. Generif metadata exists in RDF.
* Write code that queries RDF for Generif metadata and enriches exising Probeset query.
* DONE: checksums in Generif rdf output
* TODO: jnduli look at xapian docs and their example for python bindings: DONE, led to a WIP PR for wiki data indexing and a local custom script to search the index without the need to run genenetwork web service
* TODO: bonfacem makes sure tux02 indexing works: DONE, there's a working patchset sent to Arun for review.
* TODO: bonfacem make changes to mcron and guix-machines: DONE, there's a working patch set sent to Arun. Follow up needs to be done for the learnings gained
### How would GeneWiki work?
* GeneWiki = GeneRif data from NCBI
* Workflow would be similar to Generif Indexing. We need to figure out if we'll need an extra RDF query or if we can modify the existing SPARQL query.
* TODO: jnduli attempts to add stronger types to index-genenetwork script, to make it explicit that we're using MonadicDicts: Not DONE, got stuck trying to run index-genenetwork locally and removing global variables from the script..
* TODO: bonfacem makes sure that mypy in CI runs against the index-genenetwork script: NOT DONE, needs a separate PR that will be sent to Arun.
|