aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2021-09-15Reorganise modulesFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * The heatmap generation does not fall cleanly within the computations or db modules. This commit moves it to the higher level gn3 module.
2021-09-15Fix format of arguments and expected valuesFrederick Muriuki Muriithi
* tests/unit/computations/test_heatmap.py: ordering is not longer provided as a list of tuples; the ordering values are just a list of numbers now. This commit updates the test to take this into consideration. * tests/unit/computations/test_qtlreaper.py: the 'Chr' value if numeric, is represented by an actual number, not a string. This commit updates the code to take this into consideration.
2021-09-15Add missing sample file for testsFrederick Muriuki Muriithi
* tests/unit/db/data/genotypes/genotype_sample1.geno: new file Add a missing sample data file needed for unit tests.
2021-09-09Update proof-of-concept codeMuriithi Frederick Muriuki
* Add individual heatmaps * Add dendograms * Merge multiple heatmaps to single plot Updated the proof of concept code to provide a sample of what is needed to generate the appropriate heatmaps. To generate the sample heatmaps, one can run something like: env SQL_URI="mysql://webqtlout:webqtlout@127.0.0.1:3306/db_webqtl" \ python3 qtlfilesexport.py assuming that the database can be accessed at 127.0.0.1:3306
2021-09-08Ease search for traits and chromosomesMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * Return a dict of values rather than list for the traits and chromosomes to ease searching through the data.
2021-09-08Parse Chr value as int where possibleMuriithi Frederick Muriuki
* To ease sorting of data by numerical order down the line, sort the "Chr" values by numerical order.
2021-09-08Remove extraneous text to ease sortingMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * Change the id from 'T<n>' to simply '<n>' to ease sorting of the trait results by numerical order rather than string order.
2021-09-08Fix the traits order computations for clusteringMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/computations/heatmap.py: Fix ordering function * tests/unit/computations/test_heatmap.py: update test The order of the traits is important for the clustering algorithm, since the clustering seems to use the distance of one trait from another to determine how to order them. This commit also gets rid of the xoffset argument that is not important to the ordering, and was used in the older GN1 to determine how to draw the clustering lines.
2021-09-06Provide function to organise parsed QTLReaper resultsMuriithi Frederick Muriuki
* gn3/computations/qtlreaper.py: Provide a function to organise the results by trait for easier use down the line. * tests/unit/computations/test_qtlreaper.py: provide a test to ensure that the organising function works as expected.
2021-09-06Leave "Chr" value as string when parsingMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * The "Chr" value seems to be mostly a name of some sort, despite it being, seemingly an number. This commit parses the "Chr" value as a string. It also updates the tests to expec a string, rather than a number for "Chr" values.
2021-09-06Handle type-coercion exceptionsMuriithi Frederick Muriuki
* gn3/computations/qtlreaper.py: handle exceptions Sometimes, the values being parsed are plain strings and cannot be cast to the float types. This commit handles that by casting only those values that can be cast to float, and returning the others as strings.
2021-09-06Find nearest markerMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * Migrate the `web.webqtl.heatmap.Heatmap.getNearestMarker` function in GN1 to GN3.
2021-09-01Fix linting and typing issuesMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi
2021-09-01Built top-level genotype file parsing functionMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/db/genotypes.py: parse genotype files * tests/unit/db/test_genotypes.py: test parsing is correct Add the overall genotype files parsing function and tests to check that the parsing works as expected.
2021-09-01Parse data lines into markersMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/db/genotypes.py: parse data lines in file to genetic markers. * tests/unit/db/test_genotypes.py: test that parsing works. Add some tests to check that the parsing of the markers works as expected, and add the code to actually parse the markers.
2021-09-01Parse the genotype file's data headerMuriithi Frederick Muriuki
* gn3/db/genotypes.py: parse data header * tests/unit/db/test_genotypes.py: check that header's parse works correctly. Add tests to check that the parser works as expected. Add code to implement the parsing and pass the tests.
2021-09-01Implement parsing of genotype labelsMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/db/genotypes.py: parse genotype labels * tests/unit/db/test_genotypes.py: test that genotype labels are parsed correctly As part of parsing the genotype files into usable python data structures, this commit adds a function to parse the label lines (beginning with "@") into the appropriate values.
2021-08-31Fix linting errors, minor bugs and reorganise codeMuriithi Frederick Muriuki
* Fix some linting errors and some minor bugs caught by the linter. Move the `random_string` function to separate module for use in multiple places in the code.
2021-08-31Update `heatmap_data` function: remove extraneous dataMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/computations/heatmap.py: update function * gn3/db/traits.py: new function Remove extraneous data and arguments from the function. - Load the genotype file - Generate traits file - Provide both raw traits data, and exported traits data in return
2021-08-31Fix testMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * The number of the arguments to the function changed, and so the tests for the function needed to be updated.
2021-08-31Parse QTLReaper outputsMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/computations/qtlreaper.py: pass output files * tests/unit/computations/data/qtlreaper/main_output_sample.txt: sample test data * tests/unit/computations/data/qtlreaper/permu_output_sample.txt: sample test data * tests/unit/computations/test_qtlreaper.py: add tests Add code to parse the QTLReaper output data files.
2021-08-31Merge branch 'main' of github.com:genenetwork/genenetwork3 into ↵Muriithi Frederick Muriuki
heatmap_generation
2021-08-31Fix bugs with `run_reaper` functionMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/computations/qtlreaper.py: Fix some bugs * qtlfilesexport.py: Test out running rust-qtlreaper Test out the qtlreaper interface code and fix some bugs caught in the process.
2021-08-31Provide utilities for genotype filesMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/db/genotypes.py: New module * gn3/settings.py: Add new configuration variable * qtlfilesexport.py: Test out new code Add a module containing functions dealing with the genotype files. Add a configuration variable to point to the location of the genotype files.
2021-08-30Update documentation on genotype filesMuriithi Frederick Muriuki
* Provide documentation on downloading and using the genotype files.
2021-08-30Fix issues with traits file formatMuriithi Frederick Muriuki
* README.md: update header: Traits ==> Trait * gn3/computations/qtlreaper.py: update header: Traits ==> Trait * qtlfilesexport.py: Choose only BXD strains Rename the first column header from "Traits" to "Trait" to correspond with what `rust-qtlreaper` expects. Choose only the BXD strains for the proof-of-concept example - this helped bring out the fact that the traits file SHOULD NOT contain a strain column for a strain that does not exist in the genotype file in consideration. If the traits file has a strain column which does not exist in the genotype file, then `rust-qtlreaper` fails with a panic, since, from what I can tell, it tries to get a value from the genotype file for the non-existent strain, which results to a `None` type. Subsequent attempts at running an operation on the `None` type lead to the panic.
2021-08-30Remove empty lineMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * Remove empty line at the end of the traits file
2021-08-30Fix some linting errors and minor bugs.Muriithi Frederick Muriuki
2021-08-30Implement module for interfacing with rust-qtlreaperMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/computations/heatmap.py: move `generate_traits_file` function to new module * gn3/computations/qtlreaper.py: new module to interface with the `rust-qtlreaper` utility. * gn3/settings.py: Provide setting for the path to the `rust-qtlreaper` utility * qtlfilesexport.py: Move `random_string` function to new module. Update to use functions in new module. Provide a module with functions to be used to interface with `rust-qtlreaper`. This module essentially contains all the functions that are needed to build the files needed for, and to run the qtlreaper utility.
2021-08-30Document acquired knowledge on `rust-qtlreaper`Muriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/heatmaps/heatmaps.py: document format of the traits file To assist future developers, and development of the system, this commit documents some of the hard-won knowledge about the operation of the system to ease future development of the system. The documentation, if good, might also help with future onboarding of new developers to the system.
2021-08-28README: updated install instructionsPjotr Prins
2021-08-28Add explanation of using a guix profilePjotr Prins
2021-08-28Fix issue that caused R/qtl to not use the first covariate when using ↵zsloan
stratified permutations + covariates
2021-08-27Test out generation of traits fileMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * As part of the development effort, this commit provides a proof-of-concept as a reference for generating the traits data file. It might be useful for verifying that the functions that are called are working as is expected.
2021-08-27Provide intermediate data in final resultsMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * Seeing as not every requirement/feature has been migrated over at this time, this commit just provides all the intermediate data representations in the final return of the function for later use down the line.
2021-08-27Export trait data to fileMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * Provide a function to export the given strains and traits data into a traits file for use with `rust-qtlreaper`.
2021-08-27Rework strains and trait values retrievalMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * Rework the strains and values retrieval function to more closely correspond to the working of the original code in GN1
2021-08-26Added # permutations to verbose printzsloan
2021-08-26Update imported module nameMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * guix.scm: (gn packages golang) ==> (gnu packages golang) csvdiff has moved to upstream guix and been removed from latest guix-bioinformatics.
2021-08-26Add rust-qtlreaperMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * guix.scm: new dependency (rust-qtlreaper)
2021-08-20Merge branch 'main' of github.com:genenetwork/genenetwork3 into ↵Muriithi Frederick Muriuki
heatmap_generation
2021-08-20Fix typing issue(s) caught by mypyMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/computations/heatmap.py: Use `Sequence` type not `Iterator` type
2021-08-20Retrieve the strains with valid valuesMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/computations/heatmap.py: add function to get strains with values * tests/unit/computations/test_heatmap.py: new tests Add function to get the strains whose values are not `None` from the `trait_data` object passed in. This migrates https://github.com/genenetwork/genenetwork1/blob/master/web/webqtl/heatmap/Heatmap.py#L215-221 into a separate function that can handle that and be tested independently of any other code.
2021-08-20Add tests for ordering and implement functionMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/computations/heatmap.py: implement new ordering function * tests/unit/computations/test_heatmap.py: add new tests Implement the ordering function to migrate the setup of the `neworder` variable from GN1 to GN3. This migration is incomplete, since there is dependence on the return from the `web.webqtl.heatmap.Heatmap.draw` function in form of the `d_1` variable in some of the paths. The thing is, this `d_1` variable, and the `xoffset` variable seem to be used for laying out things on the drawn heatmap, and might actually end up not being needed for the new system using plotly, which has other ways of laying out things on the drawing. For now though, this commit "shims" the presence of these values until when the use of these variables is confirmed as present or absent in the new GN3 system.
2021-08-20Minor correlation fixes (#36)Alexander Kabui
* fix key error for (*tissue_cor) tissue correlation * update tests for tissue correlation * rename speed_compute to fast_compute * pep8 formatting
2021-08-18Test the clusteringMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/computations/heatmap.py: Fix clustering bugs * tests/unit/computations/test_heatmap.py: Add new tests. Fix linting issues. Test and fix the clustering function.
2021-08-18Fix obvious linting errorsMuriithi Frederick Muriuki
* Fix linting errors that do not change the function of the code.
2021-08-18Make child sequence a listMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * Since the `slink` function assigns values to the `listcopy` variable and its children, this commit ensures that the sequence is a list to allow for the assignment. If the child-sequence is a tuple, that would lead to an exception.
2021-08-18Provide top-level `riset` key-value pairMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * Provide the expected, top-level `riset` key-value pair and eliminate the redundant key-value pair.
2021-08-18Add tests and fix errors caught with testsMuriithi Frederick Muriuki
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/clustering.gmi * gn3/computations/heatmap.py: fix errors * tests/unit/computations/test_heatmap.py: new tests Add new tests with the expected source data format, and expected results. Fix all errors that were caught by running the tests
generated by cgit v1.2.3 (git 2.44.0) at 2024-11-22 18:45:06 +0000