Age | Commit message (Expand) | Author |
2023-09-05 | Fix C3001: Lambda expression assigned to a variable...Signed-off-by: Munyoki Kilyungi <me@bonfacemunyoki.com>
| Munyoki Kilyungi |
2022-12-21 | gn3: (gn3.random -> gn3.chancy): Rename module to avoid conflicts....Rename the `gn3.random` module to gn3.chancy to avoid conflicts with Python's
`random` module.
* gn3/random.py -> gn3/chancy.py: rename module
* gn3/commands.py: update import
* gn3/computations/partial_correlations.py: update import
* gn3/computations/qtlreaper.py: update import
* gn3/computations/rust_correlation.py: update import
* gn3/db/correlations.py: update import
* gn3/db/traits.py: update import
* gn3/heatmaps.py: update import
* tests/integration/conftest.py: update import
| Frederick Muriuki Muriithi |
2022-05-30 | Pass trait data as args to `fix_strains` and fix some bugs...The `fix_strains` function works on the trait data, not the basic trait
info. This commit fixes the arguments passed to the function, and also some
bugs in the function.
| Frederick Muriuki Muriithi |
2022-05-24 | New script to compute partial correlations...* Add a new script to compute the partial correlations against:
- a select list of traits, or
- an entire dataset
depending on the specified subcommand. This new script is meant to supercede
the `scripts/partial_correlations.py` script.
* Fix the check for errors
* Reorganise the order of arguments for the
`partial_correlations_with_target_traits` function: move the `method`
argument before the `target_trait_names` argument so that the common
arguments in the partial correlation computation functions share the same
order.
| Frederick Muriuki Muriithi |
2022-05-21 | Use multiprocessing to improve performance | Frederick Muriuki Muriithi |
2022-05-21 | Process primary, target and control traits in a single iteration...Rework the code to process the traits in a single iteration to improve
performance.
| Frederick Muriuki Muriithi |
2022-05-21 | Return generator object rather than tuples...Return generator objects rather than pre-computed tuples to reduce the number
of iterations needed to process the data, and thus improve the performance of
the system somewhat.
| Frederick Muriuki Muriithi |
2022-05-06 | Fix linting and typing errors | Frederick Muriuki Muriithi |
2022-05-06 | Hook up pcorrs with target traits computations...Enable the endpoint to actually compute partial correlations with selected
target traits rather than against an entire dataset.
Fix some issues caused by recent refactor that broke pcorrs against a dataset
| Frederick Muriuki Muriithi |
2022-05-05 | Compute partial correlation with selected traits...Compute partial correlations against a selection of traits rather than against
an entire dataset.
| Frederick Muriuki Muriithi |
2022-05-05 | Extract common error checking. Rename function....* Extract the common error checking code into a separate function
* Rename the function to make its use clearer
| Frederick Muriuki Muriithi |
2022-05-03 | Refactor: Remove unnecessary loop...Remove an unnecessary looping construct to help with speeding up the partial
correlations somewhat.
| Frederick Muriuki Muriithi |
2022-03-11 | Fix typing errors | Frederick Muriuki Muriithi |
2022-03-08 | Remove unused function and its tests | Frederick Muriuki Muriithi |
2022-02-24 | gn3: Explicitly specify UTF-8 to be the file encoding....When the encoding is not specified explicitly, the system default encoding is
used. This is not recommended.
* gn3/computations/ctl.py (call_ctl_script),
gn3/computations/gemma.py (generate_pheno_txt_file),
gn3/computations/parsers.py (parse_genofile),
gn3/computations/partial_correlations.py (partial_correlations_fast),
gn3/computations/rqtl.py (process_rqtl_output, process_perm_output),
gn3/computations/wgcna.py (dump_wgcna_data, call_wgcna_script),
gn3/fs_helpers.py (jsonfile_to_dict): Explicitly specify UTF-8 to be the file
encoding.
*
tests/unit/computations/test_gemma.py (TestGemma.test_generate_pheno_txt_file),
tests/unit/computations/test_wgcna.py (TestWgcna.test_create_json_file): Test
for call to open with encoding='utf-8' argument.
| Arun Isaac |
2022-02-21 | Fix minor issues introduced while fixing linting errors | Frederick Muriuki Muriithi |
2022-02-21 | Fix a myriad of linter issues...* Use `with` in place of plain `open`
* Use f-strings in place of `str.format()`
* Remove string interpolation from queries - provide data as query parameters
* other minor fixes
| Frederick Muriuki Muriithi |
2022-02-21 | Test partial corrs API with mix of existing and non-existing control traits...Test that the partial correlations endpoint handles a mix of existing and
non-existing control traits gracefully and issues a warning to the user.
Summary of changes:
* gn3/computations/partial_correlations.py: Issue a warning for all
non-existing control traits
* gn3/db/partial_correlations.py: update queries - use `INNER JOIN` for tables
instead of comma-separated list of tables
* tests/integration/conftest.py: Add `db_conn` fixture to provide a database
connection to the tests. This will probably be changed in the future to
connect to a temporary database for tests.
* tests/integration/test_partial_correlations.py: Add test to check for
correct behaviour with a mix of existing and non-existing control traits
| Frederick Muriuki Muriithi |
2022-02-19 | Test partial corrs endpoint with non-existing control traits...Test that if the endpoint is queried and not a single one of the control
traits exists in the database, then the endpoint will respond with a
404 (not-found) status code.
Summary of changes:
* gn3/computations/partial_correlations.py: Check whether any control trait is
found. If none is found, return "not-found" message.
* gn3/db/partial_correlations.py: Fix bug in Geno query.
* tests/integration/test_partial_correlations.py: Add test for non-existing
control traits. Rename function to make it clearer what it is testing
for. Remove obsoleted comments.
| Frederick Muriuki Muriithi |
2022-02-18 | Test partial correlations endpoint with non-existent primary traits...Test that the partial correlations endpoint responds with an appropriate
"not-found" message and the corresponding 404 status code in the case where a
request is made and the primary trait requested for does not exist in the
database.
Summary of the changes in each file:
* gn3/api/correlation.py: generalise the building of the response
* gn3/computations/partial_correlations.py: return with a "not-found" if the
primary trait does not exist in the database
* gn3/db/partial_correlations.py: Fix a number of bugs that led to exceptions
in the case that the primary trait did not exist
* pytest.ini: register a `slow` pytest marker
* tests/integration/test_partial_correlations.py: Add a new test to check for
an appropriate 404 response in case of a primary trait that does not exist
in the database.
| Frederick Muriuki Muriithi |
2022-02-17 | Add property tests for `dictify_by_samples`...Add property tests using pytest and hypothesis to test that the expected
properties hold for the
`gn3.computations.partial_correlations.dictify_by_samples`
function.
| Frederick Muriuki Muriithi |
2022-02-08 | Merge iterations to remove unnecessary computations...Do all the work in a single iteration to avoid unnecessary iterations that
hamper performance.
| Frederick Muriuki Muriithi |
2022-02-08 | Remove multiprocessing for stability...Web servers are long-running processes, and python is not very good at
cleaning up after itself especially in forked processes - this leads to memory
errors in the web-server after a while.
This commit removes the use of multiprocessing to avoid such failures.
| Frederick Muriuki Muriithi |
2022-02-08 | Give sorting functions more descriptive names | Frederick Muriuki Muriithi |
2022-02-08 | Use multiprocessing to speed up computation...This commit refactors the code to make it possible to use multiprocessing to
speed up the computation of the partial correlations.
The major refactor is to move the `__compute_trait_info__` function to the
top-level of the module, and provide to it all the other necessary context via
the new args.
| Frederick Muriuki Muriithi |
2022-02-08 | Remove unnecessary computation...In Python3 when slicing,
seq[:min(some_val, len(seq))] == seq[:some_val]
because Python3 will just return a copy of the entire sequence if `some_val`
happens to be larger/greater than the length of the sequence.
This commit removes the unnecessary call to `min()`
| Frederick Muriuki Muriithi |
2022-01-10 | Use the correct letter case for the keys...* Use the correct case for the keys inorder to retrieve the correct values.
| Frederick Muriuki Muriithi |
2022-01-10 | Indicate that string is an f-string...* The string had the f-string syntax to format the values to be inserted into
the string, but was missing the 'f' before the opening quotes to signify to
python that this was an f-string. This commit fixes that.
| Frederick Muriuki Muriithi |
2022-01-10 | Remove all pairs with 'None' as the value...* Remove all key-value pairs whose value is None.
| Frederick Muriuki Muriithi |
2022-01-10 | Replace unoptimised function with optimised one...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* Replace unoptimised function with one optimised to give better performance.
The optimisation done here is to fetch multiple items/traits from the
database per query, rather than the original form, which fetched a single
item/trait from the database per query.
| Frederick Muriuki Muriithi |
2022-01-10 | Convert NaN to None...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
Comment:
https://github.com/genenetwork/genenetwork3/pull/67#issuecomment-1000828159
* Convert NaN values to None to avoid possible bugs with the string replace
method used before.
| Frederick Muriuki Muriithi |
2022-01-10 | Rework database functions to fetch multiple items...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* In an attempt to optimise the performance of the partial correlations
feature, this commit reworks some database access functions to fetch
multiple items from the database, per query, unlike their original forms
which would fetch a single item per query.
This reduces queries to the database, and should hopefully improve the
responsiveness of the partial correlations feature.
| Frederick Muriuki Muriithi |
2021-12-24 | Fix typing errors | Frederick Muriuki Muriithi |
2021-12-24 | Fix sorting...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* Update the sorting algorithm, for literature and tissue correlations so that
it sorts the results by the correlation value first then by the p-value
next.
| Frederick Muriuki Muriithi |
2021-12-24 | Return the correlation method used...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* Return the correlation method used
| Frederick Muriuki Muriithi |
2021-12-24 | Reduce the total amount of data to be output...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* There is a lot of data that is not necessary in the final result. This
commit removes that data, retaining only data relevant for the display.
| Frederick Muriuki Muriithi |
2021-12-24 | Add dataset type to the results...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* The dataset type is relevant for the display of the data, therefore, this
commit presents the dataset type as part of the results.
| Frederick Muriuki Muriithi |
2021-12-17 | Add "success" status to final computation results...Issue: https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
| Frederick Muriuki Muriithi |
2021-12-14 | linting: Fix obvious linting issues | Frederick Muriuki Muriithi |
2021-12-14 | mypy: ignore some imports and errors...* Ignore some missing library stubs
* Ignore some typing errors
* Fix obvious typing errors
| Frederick Muriuki Muriithi |
2021-12-14 | TO REVERT: Add logging to see data frame | Frederick Muriuki Muriithi |
2021-12-14 | Remove any items with less than 3 samples...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* pingouin raises an exception whenever one attempts to use it to compute the
partial correlation with data that has less than 3 samples.
| Frederick Muriuki Muriithi |
2021-12-14 | Fix dataset: use target dataset not primary...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* Use the target dataset to load the target traits, not the primary trait's
dataset, since they might differ.
| Frederick Muriuki Muriithi |
2021-12-13 | Provide missing function...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* Import the missing function.
| Frederick Muriuki Muriithi |
2021-12-13 | Fix the removal of controls for corresponding Nones in targets...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* Fix the code, so that it removes all control values, whose corresponding
target values are None, without throwing an error.
| Frederick Muriuki Muriithi |
2021-12-13 | Return the primary and control traits in addition to results...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* In addition to the partial correlation results, this commit enables the
return of the chosen primary trait and the selected control traits. This
data is required for presentation on the results page.
| Frederick Muriuki Muriithi |
2021-12-13 | Run partial correlations against chosen database...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* Run the partial correlations against the database that the user selects, and
not the one from which the primary trait is. This was a bug in the code.
| Frederick Muriuki Muriithi |
2021-12-09 | Prevent error on no result. Fix indexing...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* If the dataset name is not found, don't cause an exception, instead, return
the provided search name.
* Use the correct inner object
| Frederick Muriuki Muriithi |
2021-12-08 | Provide group from primary trait...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
* From the collections page, the group is not present, so this commit
retrieves the group value from the primary trait.
| Frederick Muriuki Muriithi |
2021-11-29 | Fix linting errors...Issue:
https://github.com/genenetwork/gn-gemtext-threads/blob/main/topics/gn1-migration-to-gn2/partial-correlations.gmi
| Frederick Muriuki Muriithi |