aboutsummaryrefslogtreecommitdiff
path: root/gn3/computations/partial_correlations.py
AgeCommit message (Collapse)Author
2022-05-24New script to compute partial correlationsFrederick Muriuki Muriithi
* Add a new script to compute the partial correlations against: - a select list of traits, or - an entire dataset depending on the specified subcommand. This new script is meant to supercede the `scripts/partial_correlations.py` script. * Fix the check for errors * Reorganise the order of arguments for the `partial_correlations_with_target_traits` function: move the `method` argument before the `target_trait_names` argument so that the common arguments in the partial correlation computation functions share the same order.
2022-05-21Use multiprocessing to improve performanceFrederick Muriuki Muriithi
2022-05-21Process primary, target and control traits in a single iterationFrederick Muriuki Muriithi
Rework the code to process the traits in a single iteration to improve performance.
2022-05-21Return generator object rather than tuplesFrederick Muriuki Muriithi
Return generator objects rather than pre-computed tuples to reduce the number of iterations needed to process the data, and thus improve the performance of the system somewhat.
2022-05-06Fix linting and typing errorsFrederick Muriuki Muriithi
2022-05-06Hook up pcorrs with target traits computationsFrederick Muriuki Muriithi
Enable the endpoint to actually compute partial correlations with selected target traits rather than against an entire dataset. Fix some issues caused by recent refactor that broke pcorrs against a dataset
2022-05-05Compute partial correlation with selected traitsFrederick Muriuki Muriithi
Compute partial correlations against a selection of traits rather than against an entire dataset.
2022-05-05Extract common error checking. Rename function.Frederick Muriuki Muriithi
* Extract the common error checking code into a separate function * Rename the function to make its use clearer
2022-05-03Refactor: Remove unnecessary loopFrederick Muriuki Muriithi
Remove an unnecessary looping construct to help with speeding up the partial correlations somewhat.
2022-03-11Fix typing errorsFrederick Muriuki Muriithi
2022-03-08Remove unused function and its testsFrederick Muriuki Muriithi
2022-02-24gn3: Explicitly specify UTF-8 to be the file encoding.Arun Isaac
When the encoding is not specified explicitly, the system default encoding is used. This is not recommended. * gn3/computations/ctl.py (call_ctl_script), gn3/computations/gemma.py (generate_pheno_txt_file), gn3/computations/parsers.py (parse_genofile), gn3/computations/partial_correlations.py (partial_correlations_fast), gn3/computations/rqtl.py (process_rqtl_output, process_perm_output), gn3/computations/wgcna.py (dump_wgcna_data, call_wgcna_script), gn3/fs_helpers.py (jsonfile_to_dict): Explicitly specify UTF-8 to be the file encoding. * tests/unit/computations/test_gemma.py (TestGemma.test_generate_pheno_txt_file), tests/unit/computations/test_wgcna.py (TestWgcna.test_create_json_file): Test for call to open with encoding='utf-8' argument.
2022-02-21Fix minor issues introduced while fixing linting errorsFrederick Muriuki Muriithi
2022-02-21Fix a myriad of linter issuesFrederick Muriuki Muriithi
* Use `with` in place of plain `open` * Use f-strings in place of `str.format()` * Remove string interpolation from queries - provide data as query parameters * other minor fixes
2022-02-21Test partial corrs API with mix of existing and non-existing control traitsFrederick Muriuki Muriithi
Test that the partial correlations endpoint handles a mix of existing and non-existing control traits gracefully and issues a warning to the user. Summary of changes: * gn3/computations/partial_correlations.py: Issue a warning for all non-existing control traits * gn3/db/partial_correlations.py: update queries - use `INNER JOIN` for tables instead of comma-separated list of tables * tests/integration/conftest.py: Add `db_conn` fixture to provide a database connection to the tests. This will probably be changed in the future to connect to a temporary database for tests. * tests/integration/test_partial_correlations.py: Add test to check for correct behaviour with a mix of existing and non-existing control traits
2022-02-19Test partial corrs endpoint with non-existing control traitsFrederick Muriuki Muriithi
Test that if the endpoint is queried and not a single one of the control traits exists in the database, then the endpoint will respond with a 404 (not-found) status code. Summary of changes: * gn3/computations/partial_correlations.py: Check whether any control trait is found. If none is found, return "not-found" message. * gn3/db/partial_correlations.py: Fix bug in Geno query. * tests/integration/test_partial_correlations.py: Add test for non-existing control traits. Rename function to make it clearer what it is testing for. Remove obsoleted comments.
2022-02-18Test partial correlations endpoint with non-existent primary traitsFrederick Muriuki Muriithi
Test that the partial correlations endpoint responds with an appropriate "not-found" message and the corresponding 404 status code in the case where a request is made and the primary trait requested for does not exist in the database. Summary of the changes in each file: * gn3/api/correlation.py: generalise the building of the response * gn3/computations/partial_correlations.py: return with a "not-found" if the primary trait does not exist in the database * gn3/db/partial_correlations.py: Fix a number of bugs that led to exceptions in the case that the primary trait did not exist * pytest.ini: register a `slow` pytest marker * tests/integration/test_partial_correlations.py: Add a new test to check for an appropriate 404 response in case of a primary trait that does not exist in the database.
2022-02-17Add property tests for `dictify_by_samples`Frederick Muriuki Muriithi
Add property tests using pytest and hypothesis to test that the expected properties hold for the `gn3.computations.partial_correlations.dictify_by_samples` function.
2022-02-08Merge iterations to remove unnecessary computationsFrederick Muriuki Muriithi
Do all the work in a single iteration to avoid unnecessary iterations that hamper performance.
2022-02-08Remove multiprocessing for stabilityFrederick Muriuki Muriithi
Web servers are long-running processes, and python is not very good at cleaning up after itself especially in forked processes - this leads to memory errors in the web-server after a while. This commit removes the use of multiprocessing to avoid such failures.
2022-02-08Give sorting functions more descriptive namesFrederick Muriuki Muriithi
2022-02-08Use multiprocessing to speed up computationFrederick Muriuki Muriithi
This commit refactors the code to make it possible to use multiprocessing to speed up the computation of the partial correlations. The major refactor is to move the `__compute_trait_info__` function to the top-level of the module, and provide to it all the other necessary context via the new args.
2022-02-08Remove unnecessary computationFrederick Muriuki Muriithi
In Python3 when slicing, seq[:min(some_val, len(seq))] == seq[:some_val] because Python3 will just return a copy of the entire sequence if `some_val` happens to be larger/greater than the length of the sequence. This commit removes the unnecessary call to `min()`
2022-01-10Use the correct letter case for the keysFrederick Muriuki Muriithi
* Use the correct case for the keys inorder to retrieve the correct values.
2022-01-10Indicate that string is an f-stringFrederick Muriuki Muriithi
* The string had the f-string syntax to format the values to be inserted into the string, but was missing the 'f' before the opening quotes to signify to python that this was an f-string. This commit fixes that.
2022-01-10Remove all pairs with 'None' as the valueFrederick Muriuki Muriithi
* Remove all key-value pairs whose value is None.
2022-01-10Replace unoptimised function with optimised oneFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Replace unoptimised function with one optimised to give better performance. The optimisation done here is to fetch multiple items/traits from the database per query, rather than the original form, which fetched a single item/trait from the database per query.
2022-01-10Convert NaN to NoneFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi Comment: https://github.com/genenetwork/genenetwork3/pull/67#issuecomment-1000828159 * Convert NaN values to None to avoid possible bugs with the string replace method used before.
2022-01-10Rework database functions to fetch multiple itemsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * In an attempt to optimise the performance of the partial correlations feature, this commit reworks some database access functions to fetch multiple items from the database, per query, unlike their original forms which would fetch a single item per query. This reduces queries to the database, and should hopefully improve the responsiveness of the partial correlations feature.
2021-12-24Fix typing errorsFrederick Muriuki Muriithi
2021-12-24Fix sortingFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Update the sorting algorithm, for literature and tissue correlations so that it sorts the results by the correlation value first then by the p-value next.
2021-12-24Return the correlation method usedFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Return the correlation method used
2021-12-24Reduce the total amount of data to be outputFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * There is a lot of data that is not necessary in the final result. This commit removes that data, retaining only data relevant for the display.
2021-12-24Add dataset type to the resultsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * The dataset type is relevant for the display of the data, therefore, this commit presents the dataset type as part of the results.
2021-12-17Add "success" status to final computation resultsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
2021-12-14linting: Fix obvious linting issuesFrederick Muriuki Muriithi
2021-12-14mypy: ignore some imports and errorsFrederick Muriuki Muriithi
* Ignore some missing library stubs * Ignore some typing errors * Fix obvious typing errors
2021-12-14TO REVERT: Add logging to see data frameFrederick Muriuki Muriithi
2021-12-14Remove any items with less than 3 samplesFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * pingouin raises an exception whenever one attempts to use it to compute the partial correlation with data that has less than 3 samples.
2021-12-14Fix dataset: use target dataset not primaryFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Use the target dataset to load the target traits, not the primary trait's dataset, since they might differ.
2021-12-13Provide missing functionFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Import the missing function.
2021-12-13Fix the removal of controls for corresponding Nones in targetsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Fix the code, so that it removes all control values, whose corresponding target values are None, without throwing an error.
2021-12-13Return the primary and control traits in addition to resultsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * In addition to the partial correlation results, this commit enables the return of the chosen primary trait and the selected control traits. This data is required for presentation on the results page.
2021-12-13Run partial correlations against chosen databaseFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Run the partial correlations against the database that the user selects, and not the one from which the primary trait is. This was a bug in the code.
2021-12-09Prevent error on no result. Fix indexingFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * If the dataset name is not found, don't cause an exception, instead, return the provided search name. * Use the correct inner object
2021-12-08Provide group from primary traitFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * From the collections page, the group is not present, so this commit retrieves the group value from the primary trait.
2021-11-29Fix linting errorsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
2021-11-29Provide entry-point function for the partial correlationsFrederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Provide the entry-point function to the partial correlation feature. This is the function that ochestrates the fetching of the data, and processing it for output by the API endpoint (to be implemented).
2021-11-23Fix a myriad of linting errorsFrederick Muriuki Muriithi
* Fix linting errors like: - Unused variables - Undeclared variable errors (mostly caused by typos, and wrong names) - Missing documentation strings for functions etc.
2021-11-23Migrate `getPartialCorrelationsNormal`Frederick Muriuki Muriithi
Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Migrate the `web.webqtl.correlation.PartialCorrDBPage.getPartialCorrelationsNormal` function in GN1. * Remove function obsoleted by newer implementation of the code
generated by cgit v1.2.3 (git 2.44.0) at 2024-11-29 19:43:06 +0000