aboutsummaryrefslogtreecommitdiff
path: root/gn3/computations
AgeCommit message (Expand)Author
2021-12-08Provide group from primary trait...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * From the collections page, the group is not present, so this commit retrieves the group value from the primary trait. Frederick Muriuki Muriithi
2021-11-29Fix linting errors...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi Frederick Muriuki Muriithi
2021-11-29Provide entry-point function for the partial correlations...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Provide the entry-point function to the partial correlation feature. This is the function that ochestrates the fetching of the data, and processing it for output by the API endpoint (to be implemented). Frederick Muriuki Muriithi
2021-11-23Fix a myriad of linting errors...* Fix linting errors like: - Unused variables - Undeclared variable errors (mostly caused by typos, and wrong names) - Missing documentation strings for functions etc. Frederick Muriuki Muriithi
2021-11-23Migrate `getPartialCorrelationsNormal`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Migrate the `web.webqtl.correlation.PartialCorrDBPage.getPartialCorrelationsNormal` function in GN1. * Remove function obsoleted by newer implementation of the code Frederick Muriuki Muriithi
2021-11-19Avoid rounding: compare floats approximately...Notes: https://github.com/genenetwork/genenetwork3/pull/56#issuecomment-973798918 * As mentioned in the notes, rather than rounding to an arbitrary number of decimal places, it is a much better practice to use approximate comparisons of floats for the tests. Frederick Muriuki Muriithi
2021-11-18Fix some linting errors...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Fix some obvious linting errors and remove obsolete code Frederick Muriuki Muriithi
2021-11-18Replace code migrated from R with pingouin functions...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Replace the code that was in the process of being migrated from R in GeneNetwork1 with calls to pingouin functions that achieve the same thing. Since the functions in this case are computing correlations and partial correlations, rather than having home-rolled functions to do that, this commit makes use of the tried and tested pingouin functions. This avoids complicating our code with edge-case checks, and leverages the performance optimisations done in pingouin. Frederick Muriuki Muriithi
2021-11-15Fix bugs in recursive partial correlations...* gn3/computations/partial_correlations.py: Remove rounding. Fix computation of remaining covariates * tests/unit/computations/partial_correlations_test_data/pcor_rec_blackbox_test.txt: reduce the number of covariates to between one (1) and three (3) * tests/unit/computations/test_partial_correlations.py: fix some minor bugs It turns out that the computation complexity increases exponentially, with the number of covariates. Therefore, to get a somewhat sensible test time, while retaining a large-ish number of tests, this commit reduces the number of covariates to between 1 and 3. Frederick Muriuki Muriithi
2021-11-15Fix the columns in built data frame...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * When the z value is a Sequence of sequences of values, each of the internal sequences should form a column of its own, and not a row, as it was originally set up to do. Frederick Muriuki Muriithi
2021-11-12Merge branch 'main' of github.com:genenetwork/genenetwork3 into partial-corre...Frederick Muriuki Muriithi
2021-11-11Reimplement correlations2.compute_correlation using pearsonr....correlations2.compute_correlation computes the Pearson correlation coefficient. Outsource this computation to scipy.stats.pearsonr. When the inputs are constant, the Pearson correlation coefficient does not exist and is represented by NaN. Update the tests to reflect this. * gn3/computations/correlations2.py: Remove import of sqrt from math. (compute_correlation): Reimplement using scipy.stats.pearsonr. * tests/unit/computations/test_correlation.py: Import math. (TestCorrelation.test_compute_correlation): When inputs are constant, set expected correlation coefficient to NaN. Arun Isaac
2021-11-11Reimplement __items_with_values using list comprehension....* gn3/computations/correlations2.py: Remove import of reduce from functools. (__items_with_values): Reimplement using list comprehension. Arun Isaac
2021-11-11pylint fixes and pep8 formattingAlexander Kabui
2021-11-11fix target and base sample data orderAlexander Kabui
2021-11-11fix:spawned processes memory issuesAlexander Kabui
2021-11-11replace list with generatorsAlexander Kabui
2021-11-09Implement remaining part of `partial_correlation_recursive` function...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: implement remaining portion of `partial_correlation_recursive` function. * tests/unit/computations/test_partial_correlations.py: add parsing for new data format and update tests Frederick Muriuki Muriithi
2021-11-09Fix bug: if three columns, ensure last is "z"...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Fix a bug, caught when the function is called in a recursive form, with the "z*" columns reducing for each cycle through the recursive form. As it was, the last cycle through the recursive form would end up with a DataFrame with the columns "x", "y", and "z0" rather than the columns "x", "y", "z". This commit handles that edge case to ensure that the column name is changed from "z0" to simply "z". Frederick Muriuki Muriithi
2021-11-04Create blackbox tests for some functions migrated from R...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: new stub functions (partial_correlation_matrix, partial_correlation_recursive) * tests/unit/computations/partial_correlations_test_data/pcor_mat_blackbox_test.csv: blackbox sample data and results for variance-covariance matrix method * tests/unit/computations/partial_correlations_test_data/pcor_rec_blackbox_test.csv: blackbox sample data and results for recursive method * tests/unit/computations/test_partial_correlations.py: Tests for new function Provide some blackbox testing sample data for checking the operation of the functions migrated from R. Frederick Muriuki Muriithi
2021-11-04Stub `determine_partials`...Issue: * Stub out `determine_partials` which is a migration of `web.webqtl.correlation.correlationFunction.determinePartialsByR` in GN1. The function in GN1 has R code from line 188 to line 344. This will need to be converted over to Python. This function will also need tests. Frederick Muriuki Muriithi
2021-11-04Implement `compute_partial_correlations_fast`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Implement `compute_partial_correlations_fast` that is a partial migration of `web.webqtl.correlation.PartialCorrDBPage.getPartialCorrelationsFast` in GN1. This function will probably be reworked once the dependencies are fully migrated. It also needs tests to be added. Frederick Muriuki Muriithi
2021-11-04Retrieve indices of the selected samples...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: New function (good_dataset_samples_indexes). * tests/unit/computations/test_partial_correlations.py: Tests for new function (good_dataset_samples_indexes) Get the indices of the selected samples. This is a partial migration of the `web.webqtl.correlation.PartialCorrDBPage.getPartialCorrelationsFast` function in GN1. Frederick Muriuki Muriithi
2021-11-04Explicitly round the values...* Explicitly round the values to prevent issues with the type-checker Frederick Muriuki Muriithi
2021-11-04Specify ten (10) decimal places...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: specify 10 decimal places * tests/unit/computations/test_partial_correlations.py: update examples Slight differences in python implementations, possibly hardware and operating systems could cause the value of float (double) values to be different in the less significant parts of the decimal places. This commit limits the usable part of the decimals to the first 10 decimal places for now. Frederick Muriuki Muriithi
2021-11-04Fix linting and typing errors...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi Frederick Muriuki Muriithi
2021-11-04Complete `build_temporary_tissue_correlations_table`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: Remove comments after updating usage of the function at call point * gn3/db/correlations.py: Complete the implementation of the `build_temporary_tissue_correlations_table` function Frederick Muriuki Muriithi
2021-11-04Complete implementation of `batch_computed_tissue_correlation`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Complete the implementation of the `batch_computed_tissue_correlation` function Frederick Muriuki Muriithi
2021-11-04Move `correlations_of_all_tissue_traits`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: new function (`correlations_of_all_tissue_traits`). * gn3/db/correlations.py: delete function (`correlations_of_all_tissue_traits`). Move the function to `gn3.computations.partial_correlations` module and comment out the db-access code. Rework it to receive, as arguments, the data it previously fetched from the database, and add comments on future rework to get the function working again. Frederick Muriuki Muriithi
2021-11-04Implement `tissue_correlation` function...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: New function (tissue_correlation) * tests/unit/test_partial_correlations.py -> tests/unit/computations/test_partial_correlations.py: Move module. Implement tests for new function Migrate the `cal_tissue_corr` function embedded in the `web.webqtl.correlation.correlationFunction.batchCalTissueCorr` function in GN1 and implement tests to ensure it works correctly. Frederick Muriuki Muriithi
2021-11-04Move the function to computations module...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * The function `batch_computed_tissue_correlation` is a pure computations function with no expressions accessing the database, as far as I can tell, therefore, this commit moves the function over to the gn3.computations.partial_correlations module that holds the pure computation functions. Frederick Muriuki Muriithi
2021-11-04Move the partial_correlations module to gn3.computations...* Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Move the partial_correlations.py module to the gn3.computations module, since it contains the computations for partial correlations. Frederick Muriuki Muriithi
2021-11-04Partially implement `partial_correlation_recursive`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: Implement one path for the `gn3.computations.partial_correlations.partial_correlation_recursive` function. * gn3/settings.py: Add a setting for how many decimal places to round to * tests/unit/computations/test_partial_correlations.py: Update test to take the number of decimal places into consideration Implement a single path (where the z value is a vector and not a matrix) for the `partial_correlation_recursive` function. Frederick Muriuki Muriithi
2021-11-04Implement `build_data_frame`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: new function (`build_data_frame`) * tests/unit/computations/test_partial_correlations.py: Add tests for new function Add a new function to build a pandas DataFrame object from the provided values: - x: a vector of floats (represented with a tuple of floats) - y: a vector of floats (represented with a tuple of floats) - z: a vector OR matrix of floats (represented with a tuple of floats or a tuple of tuples of floats) Frederick Muriuki Muriithi
2021-11-04Create blackbox tests for some functions migrated from R...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: new stub functions (partial_correlation_matrix, partial_correlation_recursive) * tests/unit/computations/partial_correlations_test_data/pcor_mat_blackbox_test.csv: blackbox sample data and results for variance-covariance matrix method * tests/unit/computations/partial_correlations_test_data/pcor_rec_blackbox_test.csv: blackbox sample data and results for recursive method * tests/unit/computations/test_partial_correlations.py: Tests for new function Provide some blackbox testing sample data for checking the operation of the functions migrated from R. Frederick Muriuki Muriithi
2021-11-01Stub `determine_partials`...Issue: * Stub out `determine_partials` which is a migration of `web.webqtl.correlation.correlationFunction.determinePartialsByR` in GN1. The function in GN1 has R code from line 188 to line 344. This will need to be converted over to Python. This function will also need tests. Frederick Muriuki Muriithi
2021-11-01Implement `compute_partial_correlations_fast`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Implement `compute_partial_correlations_fast` that is a partial migration of `web.webqtl.correlation.PartialCorrDBPage.getPartialCorrelationsFast` in GN1. This function will probably be reworked once the dependencies are fully migrated. It also needs tests to be added. Frederick Muriuki Muriithi
2021-11-01Merge branch 'main' of github.com:genenetwork/genenetwork3 into partial-corre...Frederick Muriuki Muriithi
2021-11-01Retrieve indices of the selected samples...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: New function (good_dataset_samples_indexes). * tests/unit/computations/test_partial_correlations.py: Tests for new function (good_dataset_samples_indexes) Get the indices of the selected samples. This is a partial migration of the `web.webqtl.correlation.PartialCorrDBPage.getPartialCorrelationsFast` function in GN1. Frederick Muriuki Muriithi
2021-10-29Feature/biweight reimplementation (#47)...* add biweight reimplementation with pingouin * delete biweight scripts and tests * add python-pingouin to guix file * delete biweight paths * mypy fix:pingouin mising imports * pep8 formatting && pylint fixesAlexander Kabui
2021-10-29Explicitly round the values...* Explicitly round the values to prevent issues with the type-checker Frederick Muriuki Muriithi
2021-10-29Specify ten (10) decimal places...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: specify 10 decimal places * tests/unit/computations/test_partial_correlations.py: update examples Slight differences in python implementations, possibly hardware and operating systems could cause the value of float (double) values to be different in the less significant parts of the decimal places. This commit limits the usable part of the decimals to the first 10 decimal places for now. Frederick Muriuki Muriithi
2021-10-29Fix linting and typing errors...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi Frederick Muriuki Muriithi
2021-10-29Complete `build_temporary_tissue_correlations_table`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: Remove comments after updating usage of the function at call point * gn3/db/correlations.py: Complete the implementation of the `build_temporary_tissue_correlations_table` function Frederick Muriuki Muriithi
2021-10-29Complete implementation of `batch_computed_tissue_correlation`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Complete the implementation of the `batch_computed_tissue_correlation` function Frederick Muriuki Muriithi
2021-10-29Move `correlations_of_all_tissue_traits`...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: new function (`correlations_of_all_tissue_traits`). * gn3/db/correlations.py: delete function (`correlations_of_all_tissue_traits`). Move the function to `gn3.computations.partial_correlations` module and comment out the db-access code. Rework it to receive, as arguments, the data it previously fetched from the database, and add comments on future rework to get the function working again. Frederick Muriuki Muriithi
2021-10-29Implement `tissue_correlation` function...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * gn3/computations/partial_correlations.py: New function (tissue_correlation) * tests/unit/test_partial_correlations.py -> tests/unit/computations/test_partial_correlations.py: Move module. Implement tests for new function Migrate the `cal_tissue_corr` function embedded in the `web.webqtl.correlation.correlationFunction.batchCalTissueCorr` function in GN1 and implement tests to ensure it works correctly. Frederick Muriuki Muriithi
2021-10-29Move the function to computations module...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * The function `batch_computed_tissue_correlation` is a pure computations function with no expressions accessing the database, as far as I can tell, therefore, this commit moves the function over to the gn3.computations.partial_correlations module that holds the pure computation functions. Frederick Muriuki Muriithi
2021-10-29Move the partial_correlations module to gn3.computations...* Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi * Move the partial_correlations.py module to the gn3.computations module, since it contains the computations for partial correlations. Frederick Muriuki Muriithi
2021-10-25mypy:Incompatible types in assignment fixAlexander Kabui