| Age | Commit message (Expand) | Author |
|---|
| 2022-03-12 | Fill in empty values in csv text with: "x"...* gn3/csvcmp.py (fill_csv): Update this function to allow empty lists to be
filled with the default value(set in the args).
* tests/unit/test_csvcmp.py (test_fill_csv): Update test to capture above.
| BonfaceKilz |
| 2022-03-12 | Fetch id's separately for the insertion edge-case...* gn3/db/sample_data (get_sample_data_ids): Add an extra condition that caters
for inserts: during inserts, joins won't work when fetching the strain_id,
publishdata_id, and strain_name. In this case, just create 2 separate queries
to do that work.
| BonfaceKilz |
| 2022-03-12 | Extract a strain name given a csv string and it's header...* gn3/csvcmp.py (extract_strain_name): New function.
* gn3/db/sample_data (delete_sample_data): Use the aforementioned function.
(insert_sample_data): Ditto.
* tests/unit/test_csvcmp: Test cases for above.
| BonfaceKilz |
| 2022-03-12 | Allow deleting case-attribute data during deletion...* gn3/db/sample_data.py (delete_sample_data): Modify this function to allow
deleting case-attribute values.
| BonfaceKilz |
| 2022-03-12 | Allow inserting case-attribute data during inserts...* gn3/db/sample_data.py (insert_sample_data): Modify this function to allow
inserting case-attribute values.
| BonfaceKilz |
| 2022-03-12 | Fetch InbredSetId...* gn3/db/sample_data.py (get_sample_data_ids): Extend to also fetch
InbredSetId.
(update_sample_data): Discard the returned value of InbredSetId.
(delete_sample_data): Ditto.
| BonfaceKilz |
| 2022-03-12 | Create a new function for retrieving strain_id and publishdata_id...* gn3/db/sample_data.py: Import Any, Tuple.
(get_sample_data_ids): New function that fetches the strain_id and
publishdata_id of a given data point.
(update_sample_data): Use `get_sample_data_ids`.
(delete_sample_data): Ditto.
(insert_sample_data): Ditto.
| BonfaceKilz |
| 2022-03-12 | Move operations on sample_data to it's own module | BonfaceKilz |
| 2022-03-12 | Don't add extra key "Column" to dict if there are no changes...gn3/csvcmp.py (csv_diff): If the diff is empty, don't add an extra key
"Column" to the dictionary.
tests/unit/test_csvcmp (test_csv_diff_only_column_change): Add test-case for
the above.
| BonfaceKilz |
| 2022-03-12 | Fill CSV text if there are non-even rows...Should you try to run `csvdiff` against 2 csv files with either file having a
non-even columns, there will be an error. As such, the csv files need to be
"filled" before running `csvdiff`.
* gn3/csvcmp (csv_diff): For non-even rows in the csv files, fill the csv
rows.
| BonfaceKilz |
| 2022-03-12 | Create new method for filling csv with a default value...* gn3/csvcmp.py (fill_csv): Given a csv text with uneven or incomplete fields
whole length are less than width, fill them with a value which defaults to
"x".
* tests/unit/test_csvcmp.py (test_fill_csv): Test cases for the above.
| BonfaceKilz |
| 2022-03-12 | Replace "all" with "and"...* gn3/csvcmp.py (remove_insignificant_edits): "all" evaluates all elements and
throws an error if when `abs(float(x) - float(y)) < epsilon` is processed. Use
"and" instead because of it's short-circuiting behaviour.
| BonfaceKilz |
| 2022-03-12 | Store columns in the output dict...When inserting, deleting, or editing case-attributes, we need the column
headers in order to be able to know identify the attribute of interest.
* gn3/csvcmp.py (csv_diff): Add extra "Column" key in returned dict.
| BonfaceKilz |
| 2022-03-12 | Add methods for working with csv data...gn3/csvcmp.py: New file
(create_dirs_if_not_exists): From a list of dirs, create them if they don't
exist.
(remove_insignificant_edits): Given a dict with a "Modification" key, remove
edits with "delta < ε".
(csv_diff): Generate a csv_diff using the "csvdiff" tool packaged in guix.
tests/unit/test_csvcmp.py: Add some tests for "gn3/csvcmp.py"
| BonfaceKilz |
| 2022-03-12 | db: Fix error in SQL query...* gn3/db/traits.py (get_trait_csv_sample_data): Update SQL to fix runtime
errors.
| BonfaceKilz |
| 2022-03-12 | Fix pylint error | BonfaceKilz |
| 2022-03-12 | Append case attributes to csv data if they exist | BonfaceKilz |
| 2022-03-12 | db: Extend csv query to fetch case attributes...* gn3/db/traits.py (get_trait_csv_sample_data): Fetch case attribute data if
it exists.
| BonfaceKilz |
| 2022-03-12 | Revert "db: Fetch correct sample data"...This reverts commit 710769e84b3bc6a2bdd66effdbac0659272ed511.
| BonfaceKilz |
| 2022-03-11 | Fix typing errors | Frederick Muriuki Muriithi |
| 2022-03-11 | Fix some linting issues | Frederick Muriuki Muriithi |
| 2022-03-08 | Remove unused function and its tests | Frederick Muriuki Muriithi |
| 2022-03-08 | Fix tests, and issues caught by tests...Fix some issues caught by tests due to changes introducing the hand-off of the
partial correlations computations to an external process
Fix some issues due to the changes that introduce context managers for
database connections
Update some tests to take the above two changes into consideration
| Frederick Muriuki Muriithi |
| 2022-03-08 | Create database connections within context managers...Use the `with` context manager to open database connections, so as to ensure
that those connections are closed once the call is completed. This hopefully
avoids the 'too many connections' error
| Frederick Muriuki Muriithi |
| 2022-03-04 | Automatically decode Redis strings | Frederick Muriuki Muriithi |
| 2022-03-03 | Add endpoint for checking state of external processes...Long-running computations are handed off to external processes. This avoids
timeouts in the webserver, and also reduces chances of instability of the
webserver.
The results of these long-running computations are needed eventually, so this
commit provides a way to check for the state of the computation, and the
results if any.
| Frederick Muriuki Muriithi |
| 2022-03-03 | Run partial correlations in an external process...Run the partial correlations code in an external python process decoupling it
from the server and making it asynchronous.
Summary of changes:
* gn3/api/correlation.py:
- Remove response processing code
- Queue partial corrs processing
- Create new endpoint to get results
* gn3/commands.py
- Compose the pcorrs command to be run in an external process
- Enable running of subprocess commands with list args
* gn3/responses/__init__.py: new module indicator file
* gn3/responses/pcorrs_responses.py: Hold response processing code extracted
from ~gn3.api.correlations.py~ file
* scripts/partial_correlations.py: CLI script to process the pcorrs
* sheepdog/worker.py:
- Add the *genenetwork3* path at the beginning of the ~sys.path~ list to
override any GN3 in the site-packages
- Add any environment variables to be set for the command to be run
| Frederick Muriuki Muriithi |
| 2022-02-25 | Fix issue where 0's were treated as False for the primary trait in...correlations
In the original version of the if statement* I believe it was
interpreted as "if a_val and (b_val is not None)". This caused
values of 0 for a_val (the primary trait's values) to be evaluated as
False.
I changed it to compare both a_val and b_val to None. This seems to have
fixed the issue.
* if (a_val and b_val is not None)
| zsloan |
| 2022-02-24 | gn3: computations: Call Popen with context manager....Context managers should be preferred when allocating resources.
* gn3/computations/wgcna.py (stream_cmd_output): Call Popen with context
manager.
| Arun Isaac |
| 2022-02-24 | gn3: Explicitly specify UTF-8 to be the file encoding....When the encoding is not specified explicitly, the system default encoding is
used. This is not recommended.
* gn3/computations/ctl.py (call_ctl_script),
gn3/computations/gemma.py (generate_pheno_txt_file),
gn3/computations/parsers.py (parse_genofile),
gn3/computations/partial_correlations.py (partial_correlations_fast),
gn3/computations/rqtl.py (process_rqtl_output, process_perm_output),
gn3/computations/wgcna.py (dump_wgcna_data, call_wgcna_script),
gn3/fs_helpers.py (jsonfile_to_dict): Explicitly specify UTF-8 to be the file
encoding.
*
tests/unit/computations/test_gemma.py (TestGemma.test_generate_pheno_txt_file),
tests/unit/computations/test_wgcna.py (TestWgcna.test_create_json_file): Test
for call to open with encoding='utf-8' argument.
| Arun Isaac |
| 2022-02-21 | Fix minor issues introduced while fixing linting errors | Frederick Muriuki Muriithi |
| 2022-02-21 | Fix a myriad of linter issues...* Use `with` in place of plain `open`
* Use f-strings in place of `str.format()`
* Remove string interpolation from queries - provide data as query parameters
* other minor fixes
| Frederick Muriuki Muriithi |
| 2022-02-21 | Test partial corrs API with mix of existing and non-existing control traits...Test that the partial correlations endpoint handles a mix of existing and
non-existing control traits gracefully and issues a warning to the user.
Summary of changes:
* gn3/computations/partial_correlations.py: Issue a warning for all
non-existing control traits
* gn3/db/partial_correlations.py: update queries - use `INNER JOIN` for tables
instead of comma-separated list of tables
* tests/integration/conftest.py: Add `db_conn` fixture to provide a database
connection to the tests. This will probably be changed in the future to
connect to a temporary database for tests.
* tests/integration/test_partial_correlations.py: Add test to check for
correct behaviour with a mix of existing and non-existing control traits
| Frederick Muriuki Muriithi |
| 2022-02-19 | Test partial corrs endpoint with non-existing control traits...Test that if the endpoint is queried and not a single one of the control
traits exists in the database, then the endpoint will respond with a
404 (not-found) status code.
Summary of changes:
* gn3/computations/partial_correlations.py: Check whether any control trait is
found. If none is found, return "not-found" message.
* gn3/db/partial_correlations.py: Fix bug in Geno query.
* tests/integration/test_partial_correlations.py: Add test for non-existing
control traits. Rename function to make it clearer what it is testing
for. Remove obsoleted comments.
| Frederick Muriuki Muriithi |
| 2022-02-18 | Remove code trying to query non-existent `TempFreeze` table...The code was migrated from GN1 with a faulty assumption that all trait types
have a corresponding `*Freeze` table in the database. This assumption is not
true for the `Temp` traits.
This commit removes the buggy code.
| Frederick Muriuki Muriithi |
| 2022-02-18 | Test partial correlations endpoint with non-existent primary traits...Test that the partial correlations endpoint responds with an appropriate
"not-found" message and the corresponding 404 status code in the case where a
request is made and the primary trait requested for does not exist in the
database.
Summary of the changes in each file:
* gn3/api/correlation.py: generalise the building of the response
* gn3/computations/partial_correlations.py: return with a "not-found" if the
primary trait does not exist in the database
* gn3/db/partial_correlations.py: Fix a number of bugs that led to exceptions
in the case that the primary trait did not exist
* pytest.ini: register a `slow` pytest marker
* tests/integration/test_partial_correlations.py: Add a new test to check for
an appropriate 404 response in case of a primary trait that does not exist
in the database.
| Frederick Muriuki Muriithi |
| 2022-02-17 | Test partial correlations endpoint with missing data in POST request...Add a test for the partial correlations endpoint, with:
- no data in the request
- missing items in the data
Fix the bugs caught by the test
| Frederick Muriuki Muriithi |
| 2022-02-17 | Add property tests for `dictify_by_samples`...Add property tests using pytest and hypothesis to test that the expected
properties hold for the
`gn3.computations.partial_correlations.dictify_by_samples`
function.
| Frederick Muriuki Muriithi |
| 2022-02-08 | Merge iterations to remove unnecessary computations...Do all the work in a single iteration to avoid unnecessary iterations that
hamper performance.
| Frederick Muriuki Muriithi |
| 2022-02-08 | Remove multiprocessing for stability...Web servers are long-running processes, and python is not very good at
cleaning up after itself especially in forked processes - this leads to memory
errors in the web-server after a while.
This commit removes the use of multiprocessing to avoid such failures.
| Frederick Muriuki Muriithi |
| 2022-02-08 | Give sorting functions more descriptive names | Frederick Muriuki Muriithi |
| 2022-02-08 | Use multiprocessing to speed up computation...This commit refactors the code to make it possible to use multiprocessing to
speed up the computation of the partial correlations.
The major refactor is to move the `__compute_trait_info__` function to the
top-level of the module, and provide to it all the other necessary context via
the new args.
| Frederick Muriuki Muriithi |
| 2022-02-08 | Remove unnecessary computation...In Python3 when slicing,
seq[:min(some_val, len(seq))] == seq[:some_val]
because Python3 will just return a copy of the entire sequence if `some_val`
happens to be larger/greater than the length of the sequence.
This commit removes the unnecessary call to `min()`
| Frederick Muriuki Muriithi |
| 2022-02-08 | db: traits: Enable deletion of individual sample values...If a user replaces an individual value with an "x", delete that date entry
from the respective table. Deletion here is the only option since by default
the Nstrain, PublishData and PublishSE don't accept null values. Note that
deleting all 3 values is equivalent to removing the sample from the CSV file.
* gn3/db/traits.py (update_sample_data): If a value is "x", delete it from the
respective table.
| BonfaceKilz |
| 2022-02-08 | db: traits: Allow insertions of "0" in resp. sample values tables...When editing values from "x" to "0"(or any other value) when editing data, an
"update" statement was being run; thereby no new value was being inserted. To
the end user, modifying an "x" value to something else meant that no value was
being inserted. This commit fixes that by doing an insert whenever a change
from "x" to "0" is performed.
* gn3/db/traits.py (update_sample_data): Add insert statements whenever an
"update" statement returns a 0 row-count.
| BonfaceKilz |
| 2022-02-08 | db: traits: Insert "count" instead of "error" in "NStrain" table | BonfaceKilz |
| 2022-02-03 | db: Fetch correct sample data | BonfaceKilz |
| 2022-02-02 | Remove PublishFreeze from retrieve_publish_trait_info query...The PublishFreeze table isn't necessary in phenotype queries, since
PublishFreeze.Id = InbredSet.Id (for the purposes of identifying traits,
at least)
| zsloan |
| 2022-02-02 | Fix bug where sample values of 0 were being treated as False...In line 91 of gn3/db/traits.py, there was an if statement "if
record[key] else 'x'" that was treating values of 0 as False, so I
changed it to explicitly check that values aren't None
| zsloan |
| 2022-02-02 | Fix retrieve_publish_trait_data query...The PublishFreeeze table is actually unnecessary for this query, since
the group ID (inbred_set_id) should be passed in and that ID is in the
PublishXRef table (so no neeed to join with PublishFreeze)
| zsloan |
