diff options
Diffstat (limited to 'tasks')
| -rw-r--r-- | tasks/alexm.gmi | 29 | ||||
| -rw-r--r-- | tasks/bonfacem.gmi | 49 | ||||
| -rw-r--r-- | tasks/felixl.gmi | 28 | ||||
| -rw-r--r-- | tasks/fredm.gmi | 16 | ||||
| -rw-r--r-- | tasks/pjotrp.gmi | 56 | ||||
| -rw-r--r-- | tasks/programmer-team/meetings.gmi | 82 | ||||
| -rw-r--r-- | tasks/zachs.gmi | 7 |
7 files changed, 235 insertions, 32 deletions
diff --git a/tasks/alexm.gmi b/tasks/alexm.gmi index 88d3927..b3031d3 100644 --- a/tasks/alexm.gmi +++ b/tasks/alexm.gmi @@ -1,4 +1,4 @@ -# Tasks for Fred +# Tasks for Alex ## Description @@ -16,11 +16,24 @@ You can refine the search by constraining the checks some more, e.g. to get high # Tasks -* [ ] Make GNQA reliable (with @fahamu) -* [ ] Improve UX for GNQA (with @shelbys) -* [ ] GNQA add abstracts pubmed (with @shelbys) -=> ../issues/fetch-pubmed-references-to-gnqa +## This week + +* [X] Rqtl2 - BXD output work on CD +* [X] Rqtl2 - ITP output - 3K individuals - family file +* [ ] Minor refactorings +* [ ] Accelerate Xapian functionality +* [ ] Start application - Pwani +* [+] Create terminal output for external processes on *PRODUCTION* (Rqtl1, Rqtl2, GEMMA, pair-scan are done --- WGCNA as a pilot, with @bonfacem and @pjotrp) + +## Next week(s) -* [ ] Edit markdown/gemtext pages through web UI (with @bonfacem) -* [ ] GNQA add GN metadata with @bonfacem -* [ ] Create terminal output for external processes (WGCNA as a pilot, with @bonfacem and @pjotrp) +* [+] Create terminal output for external processes (Rqtl1, Rqtl2, pair-scan are done --- WGCNA as a pilot, with @bonfacem and @pjotrp) +* [ ] GNQA says there are no results, but has them + +## Done + +* [X] Make GNQA reliable (with @fahamu) +* [X] Improve UX for GNQA (with @shelbys) -- Adrian wants to use our AI UX for their setup +* [X] GNQA add abstracts pubmed (with @shelbys) +=> ../issues/fetch-pubmed-references-to-gnqa +* [X] Edit markdown/gemtext pages through web UI (with @bonfacem) diff --git a/tasks/bonfacem.gmi b/tasks/bonfacem.gmi index 52f4027..7514e5b 100644 --- a/tasks/bonfacem.gmi +++ b/tasks/bonfacem.gmi @@ -8,9 +8,44 @@ ## Tasks -* [X] Indexing generif data / Improve Local Search -* [ ] Add hashes to RDF metadata -* [-] Brain Data (To be spec'ed further) +### This week + +* [ ] Describe Generif/wikidata access for Rob in an email with test account on CD +* 1. Send email to Rob +* 2. Work on production +* [ ] Migrate fahamuai RAG to VPS and switch tokens to GGI OpenAI account +* 1. Running AI server using (our) VPS and our tokens +* 2. Read the code base +* 3. Try ingestion and prompt (REST API) +* 4. Backup state from production Elixir +* 5. Assess porting it to Guix (don't do any work) - minimum version Elixir +* [ ] xapian search, add dataset size keys, as well as GN accession id, trait id, and date/year +* - Improve xapian markdown docs to show all used fields/keys with examples +* - genewiki search (link in table? check with Rob) +* - base line with GN1 search - add tests +* - Fix missing search term for sh* - both menu search and global search +* - Use GN1 as a benchmark for search results (mechanical Rob?) +* - Xapian ranges for markers +* [ ] With Alex get rqtl2 demo going in CD (for BXD) + +### Later + +* [ ] Sample/individual/strain/genometype counts for PublishData only - ProbeSetData? https://github.com/genenetwork/genenetwork2/blob/testing/scripts/sample_count.py - mirror in RDF and use global search +* - search for all traits that have more than X samples +* [+] Use aider - give a presentation in the coming weeks +* [ ] Add case attributes to RDF and share with Felix (depends on @felixl) + +### Even later + +* [ ] Rest API for precompute output (mapping with GEMMA) +* [ ] GNQA add GN metadata (to RAG) +* - Focus on RIF +* - triple -> plain text +* - bob :fatherof nancy -> Bob is the father of Nancy. + +## Later + +* [ ] AI improvements ### On going tasks @@ -34,3 +69,11 @@ Should something in one of these closed issues be amiss, we can always and shoul Currently closed issues are: => https://issues.genenetwork.org/search?type=closed-issue&query=assigned%3ABonfaceKilz%20AND%20type%3Aissue%20AND%20is%3Aclosed Closed Issues + +* [X] Indexing generif data / Improve Local Search +* [X] lmdb publishdata output and share with Pjotr and Johannes + +## Done + +* [X] Add lmdb output hashes with index and export LMDB_DATA_DIRECTORY +* [X] Share small database with @pjotrp and @felixl diff --git a/tasks/felixl.gmi b/tasks/felixl.gmi index 209e8c9..b39ff7b 100644 --- a/tasks/felixl.gmi +++ b/tasks/felixl.gmi @@ -1,4 +1,4 @@ -# Tasks for Munyoki +# Tasks for Felix ## Tags @@ -6,12 +6,34 @@ * assigned: felixl * status: in progress -## October +## Tasks +#### This week + +* [ ] Finalize the slide deck - so it can be read on its own +* [ ] Analyse and discuss BXD case attributes with Rob --- both group level and dataset level +* [ ] Sane representation of case attributes in RDF with @bonfacem +* [X] Present C.elegans protocol and example mappings with GEMMA/Rqtl +* [ ] Uploader - setting up code with @fredm +* - [X] run small database mysql locally +* - [X] aider with Sonnet + code fixes +* - [ ] document - add to code base - merge with Fred's tree - share changes with Pjotr & team +* [ ] Review paper: one-liners for @pjotrp - why is this important for GN and/or thesis +* - [ ] list of relevant papers with one-liners - the WHY +* [ ] Sort @alexm application with Pwani = this week + +### Later weeks + +* [ ] Review paper on genotyping +* [ ] HS Rat +* [ ] Prepare others for C.elegans * [ ] Upload Arabidopsis dataset * [ ] Upload Medaka dataset -## Tasks +### Done + + + ### On going tasks => https://issues.genenetwork.org/search?query=assigned%3Afelixl+AND+is%3Aopen&type=open-issue All in-progress tasks diff --git a/tasks/fredm.gmi b/tasks/fredm.gmi index 5e7e71d..1cd3125 100644 --- a/tasks/fredm.gmi +++ b/tasks/fredm.gmi @@ -1,5 +1,21 @@ # Tasks for Fred +# Tags + +* kanban: fredm +* assigned: @fredm +* status: in progress + +# Tasks + +* [ ] Add drives to Penguin2, see issues/systems/penguin2-raid5 +* [X] Move production files from sdc to sde +* [ ] Fix password weakness +* [ ] Fix gn-docs and editing, e.g. facilities page by gn-guile in container +* [ ] Unifiy container dirs +* [ ] Fix wikidata gene aliases (see mapping page) with @pjotrp +* [ ] Public SPARQL container? + ## Description These are the tasks and issues to be handled by Fred. diff --git a/tasks/pjotrp.gmi b/tasks/pjotrp.gmi index b284c46..e5e25fe 100644 --- a/tasks/pjotrp.gmi +++ b/tasks/pjotrp.gmi @@ -6,24 +6,41 @@ * assigned: pjotrp * status: in progress -# Notes - -The tasks here should probably be broken out into appropriately tagged issues, where they have not - they can be found and filtered out with tissue (formerly gnbug). +# Current -=> https://issues.genenetwork.org +## 1U01HG013760 -Generally work applies to NIH/R073237482 and other grants. +* Prefix-Free Parsing Compressed Suffix Tree (PFP) for tokenization +* Mempig -# Current +* [+] create backup server with @fredm +* [+] Moni builds 1U01HG013760 +* [+] test framework wfmash - vertebrate tree and HPC compute? +* [ ] gbam - data compression with Nick and Hasithak +* [ ] accelerate wfmash with @santiago and team +* [ ] package wfmash and Rust wfa2-lib +* [ ] add Ceph for distributed network storage 1U01HG013760 +* [ ] Work on pangenome genotyping 1U01HG013760 +* [ ] update freebayes into Debian (version #) +* - [ ] static build and prepare for conda +* [ ] update vcflib into Debian (version #) +* - [ ] static build and prepare for conda +* [ ] pangenome as a 1st class input for GEMMA +* kilifish pangenome with Paul and Dario +* RAG with Shelby and Bonz ## Systems +* [ ] Backup production databases on Tux04 +* - [ ] Dump redis +* - [ ] Dump sqlite +* - [ ] Dump mariadb * [ ] make sure production is up to scratch (see stable below) -* [ ] backup tux04 -* [ ] add Ceph for distributed network storage 1U01HG013760 +* [ ] synchronize git repos for public, CD, fallback and production using sheepdog and document * [ ] drop tux02 backups on balg01 -* [ ] drop backups NL +* [ ] drop backups in NL * [ ] reintroduce borg-borg +* [X] Small database public ## Ongoing tasks (current/urgent) @@ -33,8 +50,6 @@ Generally work applies to NIH/R073237482 and other grants. * [ ] Check email setup tux04 * [+] Julia as part of GN3 deployment => /topics/deploy/julia.gmi -* [ ] Work on pangenome genotyping 1U01HG013760 -* [+] Moni builds 1U01HG013760 # Tasks @@ -51,11 +66,11 @@ Now (X=done +=WIP _=kickoff ?=?) * [+] Build leadership team * [+] gBAM * [ ] p-value global search -* [+] Xapian search add tags, notmuch style (with @zachs) +* [+] Xapian search add tags, notmuch style (with @bonfacem and @zachs) => ../issues/systems/octopus -* [ ] Add R/qtl2 and multi-parent support with Karl (DO and Magic populations) +* [+] Add R/qtl2 and multi-parent support with Karl (DO and Magic populations) * [+] Fix slow search on Mariadb? Moving to xapian * [.] GeneNetwork paper * + [ ] add FAIR statement @@ -86,11 +101,7 @@ Later ### Set up stable server instance with new hardware * [ ] ssh-shell access for git markdown -* [ ] R/qtl2 with Karl and Alex -* [+] Set up opensmtpd as a service - + [ ] Add package dependency - + [X] Test on open port 25 - + [ ] Add public-inbox (Arun) +* [+] R/qtl2 with Karl and Alex, see [alex.gmi] => ./machine-room.gmi machine room @@ -118,3 +129,12 @@ Later * [X] Fix mariadb index search - need to upgrade mariadb to convert final utf8mb4, see => ../issues/slow-sql-query-for-xapian-indexing.gmi * [X] Debian/free software issues incl. vcflib work in Zig and release +* [X] Set up opensmtpd as a service + +# Notes + +The tasks here should probably be broken out into appropriately tagged issues, where they have not - they can be found and filtered out with tissue (formerly gnbug). + +=> https://issues.genenetwork.org + +Generally work applies to NIH/R073237482 and other grants. diff --git a/tasks/programmer-team/meetings.gmi b/tasks/programmer-team/meetings.gmi new file mode 100644 index 0000000..d972b3b --- /dev/null +++ b/tasks/programmer-team/meetings.gmi @@ -0,0 +1,82 @@ +# Weekly meetings + +In this document we will track tasks based of our weekly meetings. This list sets the agenda +on progress for the next week's meeting. + +## 02-10-2024 +## @felixm +* [ ] Use Aider to contribute and cover to Fred's coding. Share useful prompts. +* [ ] Feed relevant papers to GPT and find similar summary for other datasets. Start with C-Elegans. + + +## @bonfacem +* [ ] Share values with PJ. +* [ ] Assume LMDB files are transient. When hash doesn't exist, generate the hash for that dataset. Use LMDB to store key value pairs of hashes. +* [ ] Add dump script to gn-guile. +* [ ] Add Case Attributes in Virtuoso. + +## @alex +* [ ] Push R/QTL2 to production +* [ ] Have R/QTL2 work for ITP + +Nice to have: +* Think about editing publish data and consequent updates to LMDB. + +## @pjotr +* Kickstart UTHSC VPN access for Felix and Alex. + +## 01-20-2024 +### @bonfacem + +* [ ] Report: OpenAI on Aider - use AI for programming - discuss with @alexm + +=> https://issues.genenetwork.org/topics/ai/aider + +* [-] Metadata: Provide list of case attributes for BXD to @flisso +* [-] Code UI: GeneRIF and GenWiki should work from the mapping page - encourage people to use + - anyone logged in can edit + - If RIF does not exist point to GeneWiki + - If GeneWiki does not exist provide edit page +* [ ] Code export: Exporting traits to lmdb PublishData - @alexm helps with SQL + - missing data should not be an X + - run lmdb design (first code) by @pjotrp + - start exporting traits for Johannes (he will need to write a python reader) +* Later: Improve the work/dev container for @alexm + +### @flisso + +* [ ] Write: Uploader protocol. NOTES: Finished with C-elegans. Yet to test with other datasets. +* [ ] Script: Run Reaper +* [ ] Data: Case attributes - with @bonfacem +* [ ] Write: Create protocol to upload case attributes + +### @alexm + +* [ ] Code: Rqtl2 match Rqtl1: match scan changes. Notes: PR out and added tests. +* [ ] Bug: Fix pair scan. NOTES: Fixed it. But can't test it now since CD is down. +* Later: AI changes + +### @Pjotr + +* [ ] Code: Work on precompute with GEMMA (w. Jameson) +* [ ] Code: Take Bonface's trait files when they become available + + +## 01-27-2024 + +Last week's error with CD and production downtime: +* [ level 1] Container: Error messages when data not loaded in Virtuoso, Indexing. +* [ level 2] Sheepdog: Check services --- sheepdog. Health checkpoints. +* [ level 3] User feedback. Escalate errors correctly to the users, so they can report to coders + +### @bonfacem +* [ ] Troubleshoot CD. +* [ ] Export files in lmdb. Yohannes read file in Python example +* [ ] Metadata: Provide list of case attributes for BXD to @flisso +* [ ] Aider: See if it can generate some guile and python. Give an example. + +### @alexm +* [ ] UI for R/Qtl2. + +### @flisso +* [ ] Look at Fred Python code for the uploader and report on this. diff --git a/tasks/zachs.gmi b/tasks/zachs.gmi new file mode 100644 index 0000000..6ae3df1 --- /dev/null +++ b/tasks/zachs.gmi @@ -0,0 +1,7 @@ +# Tasks for Zach + +# Tasks + +* [ ] Move non-ephemeral data out of redis into sqlite DB - see JSON dump +* - [ ] Collections +* - [ ] permanent URIs(?) |