aboutsummaryrefslogtreecommitdiff
path: root/gn3/computations
AgeCommit message (Collapse)Author
2022-03-08Remove unused function and its testsFrederick Muriuki Muriithi
2022-03-08Fix tests, and issues caught by testsFrederick Muriuki Muriithi
Fix some issues caught by tests due to changes introducing the hand-off of the partial correlations computations to an external process Fix some issues due to the changes that introduce context managers for database connections Update some tests to take the above two changes into consideration
2022-02-25Fix issue where 0's were treated as False for the primary trait inzsloan
correlations In the original version of the if statement* I believe it was interpreted as "if a_val and (b_val is not None)". This caused values of 0 for a_val (the primary trait's values) to be evaluated as False. I changed it to compare both a_val and b_val to None. This seems to have fixed the issue. * if (a_val and b_val is not None)
2022-02-24gn3: computations: Call Popen with context manager.Arun Isaac
Context managers should be preferred when allocating resources. * gn3/computations/wgcna.py (stream_cmd_output): Call Popen with context manager.
2022-02-24gn3: Explicitly specify UTF-8 to be the file encoding.Arun Isaac
When the encoding is not specified explicitly, the system default encoding is used. This is not recommended. * gn3/computations/ctl.py (call_ctl_script), gn3/computations/gemma.py (generate_pheno_txt_file), gn3/computations/parsers.py (parse_genofile), gn3/computations/partial_correlations.py (partial_correlations_fast), gn3/computations/rqtl.py (process_rqtl_output, process_perm_output), gn3/computations/wgcna.py (dump_wgcna_data, call_wgcna_script), gn3/fs_helpers.py (jsonfile_to_dict): Explicitly specify UTF-8 to be the file encoding. * tests/unit/computations/test_gemma.py (TestGemma.test_generate_pheno_txt_file), tests/unit/computations/test_wgcna.py (TestWgcna.test_create_json_file): Test for call to open with encoding='utf-8' argument.
2022-02-21Fix minor issues introduced while fixing linting errorsFrederick Muriuki Muriithi
2022-02-21Fix a myriad of linter issuesFrederick Muriuki Muriithi
* Use `with` in place of plain `open` * Use f-strings in place of `str.format()` * Remove string interpolation from queries - provide data as query parameters * other minor fixes
2022-02-21Test partial corrs API with mix of existing and non-existing control traitsFrederick Muriuki Muriithi
Test that the partial correlations endpoint handles a mix of existing and non-existing control traits gracefully and issues a warning to the user. Summary of changes: * gn3/computations/partial_correlations.py: Issue a warning for all non-existing control traits * gn3/db/partial_correlations.py: update queries - use `INNER JOIN` for tables instead of comma-separated list of tables * tests/integration/conftest.py: Add `db_conn` fixture to provide a database connection to the tests. This will probably be changed in the future to connect to a temporary database for tests. * tests/integration/test_partial_correlations.py: Add test to check for correct behaviour with a mix of existing and non-existing control traits
2022-02-19Test partial corrs endpoint with non-existing control traitsFrederick Muriuki Muriithi
Test that if the endpoint is queried and not a single one of the control traits exists in the database, then the endpoint will respond with a 404 (not-found) status code. Summary of changes: * gn3/computations/partial_correlations.py: Check whether any control trait is found. If none is found, return "not-found" message. * gn3/db/partial_correlations.py: Fix bug in Geno query. * tests/integration/test_partial_correlations.py: Add test for non-existing control traits. Rename function to make it clearer what it is testing for. Remove obsoleted comments.
2022-02-18Test partial correlations endpoint with non-existent primary traitsFrederick Muriuki Muriithi
Test that the partial correlations endpoint responds with an appropriate "not-found" message and the corresponding 404 status code in the case where a request is made and the primary trait requested for does not exist in the database. Summary of the changes in each file: * gn3/api/correlation.py: generalise the building of the response * gn3/computations/partial_correlations.py: return with a "not-found" if the primary trait does not exist in the database * gn3/db/partial_correlations.py: Fix a number of bugs that led to exceptions in the case that the primary trait did not exist * pytest.ini: register a `slow` pytest marker * tests/integration/test_partial_correlations.py: Add a new test to check for an appropriate 404 response in case of a primary trait that does not exist in the database.
2022-02-17Add property tests for `dictify_by_samples`Frederick Muriuki Muriithi
Add property tests using pytest and hypothesis to test that the expected properties hold for the `gn3.computations.partial_correlations.dictify_by_samples` function.
2022-02-08Merge iterations to remove unnecessary computationsFrederick Muriuki Muriithi
Do all the work in a single iteration to avoid unnecessary iterations that hamper performance.
2022-02-08Remove multiprocessing for stabilityFrederick Muriuki Muriithi
Web servers are long-running processes, and python is not very good at cleaning up after itself especially in forked processes - this leads to memory errors in the web-server after a while. This commit removes the use of multiprocessing to avoid such failures.
2022-02-08Give sorting functions more descriptive namesFrederick Muriuki Muriithi
2022-02-08Use multiprocessing to speed up computationFrederick Muriuki Muriithi
This commit refactors the code to make it possible to use multiprocessing to speed up the computation of the partial correlations. The major refactor is to move the `__compute_trait_info__` function to the top-level of the module, and provide to it all the other necessary context via the new args.
2022-02-08Remove unnecessary computationFrederick Muriuki Muriithi
In Python3 when slicing, seq[:min(some_val, len(seq))] == seq[:some_val] because Python3 will just return a copy of the entire sequence if `some_val` happens to be larger/greater than the length of the sequence. This commit removes the unnecessary call to `min()`
2022-02-02refactor code for invoking ctl scriptAlexander Kabui
2022-02-02pep8 formattingAlexander Kabui
2022-02-02new line fixAlexander Kabui
2022-02-02fix comprehension listAlexander Kabui
2022-01-22process ctl plot data imgAlexander Kabui
2022-01-22read stdout data;handle exceptionsAlexander Kabui
2022-01-22init file to call ctl scriptAlexander Kabui
2022-01-12Update return type. Remove duplicate import.Frederick Muriuki Muriithi
The function is a generator function, since it uses a `yield` statement, and thus returns a generator object, that contains a tuple object. This fixes that. We also remove a duplicate import.
2022-01-10Use the correct letter case for the keysFrederick Muriuki Muriithi
* Use the correct case for the keys inorder to retrieve the correct values.
2022-01-10Indicate that string is an f-stringFrederick Muriuki Muriithi
* The string had the f-string syntax to format the values to be inserted into the string, but was missing the 'f' before the opening quotes to signify to python that this was an f-string. This commit fixes that.
2022-01-10Remove all pairs with 'None' as the valueFrederick Muriuki Muriithi
* Remove all key-value pairs whose value is None.
2022-01-10Replace unoptimised function with optimised oneFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Replace unoptimised function with one optimised to give better performance. The optimisation done here is to fetch multiple items/traits from the database per query, rather than the original form, which fetched a single item/trait from the database per query.
2022-01-10Convert NaN to NoneFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi Comment: https://github.com/genenetwork/genenetwork3/pull/67#issuecomment-1000828159 * Convert NaN values to None to avoid possible bugs with the string replace method used before.
2022-01-10Add optimised entry for partial correlationsFrederick Muriuki Muriithi
Issue: * Function `gn3.computations.partial_correlations_optimised.partial_correlations_entry` is a copy of the `gn3.computations.partial_correlation.partial_correlations_entry` function that is optimised for better performance. The optimised function is intended to replace the unoptimised one, but it is included in this commit for comparison purposes, and to maintain some historical context for doing it this way.
2022-01-10Rework database functions to fetch multiple itemsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * In an attempt to optimise the performance of the partial correlations feature, this commit reworks some database access functions to fetch multiple items from the database, per query, unlike their original forms which would fetch a single item per query. This reduces queries to the database, and should hopefully improve the responsiveness of the partial correlations feature.
2022-01-05Merge pull request #64 from jgarte/type-hint-normalize-valuesBonfaceKilz
Adds type hint for normalize_values function
2021-12-24Fix typing errorsFrederick Muriuki Muriithi
2021-12-24Fix linting errorsFrederick Muriuki Muriithi
2021-12-24Fix sortingFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Update the sorting algorithm, for literature and tissue correlations so that it sorts the results by the correlation value first then by the p-value next.
2021-12-24Return the correlation method usedFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Return the correlation method used
2021-12-24Reduce the total amount of data to be outputFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * There is a lot of data that is not necessary in the final result. This commit removes that data, retaining only data relevant for the display.
2021-12-24Add dataset type to the resultsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * The dataset type is relevant for the display of the data, therefore, this commit presents the dataset type as part of the results.
2021-12-17Add "success" status to final computation resultsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
2021-12-14linting: Fix obvious linting issuesFrederick Muriuki Muriithi
2021-12-14mypy: ignore some imports and errorsFrederick Muriuki Muriithi
* Ignore some missing library stubs * Ignore some typing errors * Fix obvious typing errors
2021-12-14Adds type hint for normalize_values functionjgart
2021-12-14TO REVERT: Add logging to see data frameFrederick Muriuki Muriithi
2021-12-14Remove any items with less than 3 samplesFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * pingouin raises an exception whenever one attempts to use it to compute the partial correlation with data that has less than 3 samples.
2021-12-14Fix dataset: use target dataset not primaryFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Use the target dataset to load the target traits, not the primary trait's dataset, since they might differ.
2021-12-13Provide missing functionFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Import the missing function.
2021-12-13Fix the removal of controls for corresponding Nones in targetsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Fix the code, so that it removes all control values, whose corresponding target values are None, without throwing an error.
2021-12-13Return the primary and control traits in addition to resultsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * In addition to the partial correlation results, this commit enables the return of the chosen primary trait and the selected control traits. This data is required for presentation on the results page.
2021-12-13Run partial correlations against chosen databaseFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Run the partial correlations against the database that the user selects, and not the one from which the primary trait is. This was a bug in the code.
2021-12-10format normalize function doc stringAlexander Kabui
generated by cgit v1.2.3 (git 2.44.0) at 2024-11-25 21:50:09 +0000