| Age | Commit message (Expand) | Author |
|---|
| 2022-03-12 | Compose csv-diff command within single quotes...* gn3/csvcmp.py (csv_diff): Use single quotes. There was a change in 6d39c92
that broke this.
| BonfaceKilz |
| 2022-03-12 | Delete noisy "print" statement | BonfaceKilz |
| 2022-03-12 | Store the first element as strain_id | BonfaceKilz |
| 2022-03-12 | Append the strain name when extracting "actions"...* gn3/db/sample_data.py (__extract_actions): During updates, make sure that
the strain name is part of the returned string when extracting "actions".
* tests/unit/db/test_sample_data.py: Add test cases for the above.
| BonfaceKilz |
| 2022-03-12 | Apply auto-pep8 to sample_data.py and it's test file | BonfaceKilz |
| 2022-03-12 | Add missing return type-annotations...* tests/unit/db/test_sample_data.py (delete_sample_data): Add missing return
type for type annotations.
(insert_sample_data): Ditto.
| BonfaceKilz |
| 2022-03-12 | Update how data is updated by re-using existing functions...* gn3/db/sample_data.py (get_sample_data_ids): Re-use "delete_sample_data" and
"insert_sample_data" when updating data; and also add logic for updating
modified data.
* tests/unit/db/test_sample_data.py: Add tests for the above.
| BonfaceKilz |
| 2022-03-12 | Create action dict that's created when updating data...* gn3/db/sample_data.py (__extract_actions): An update on a vector of data can
contain: inserts, deletes and updates. This functions extracts these actions
during an update.
* tests/unit/db/test_sample_data.py (test_extract_actions): Add test-case for
the above.
| BonfaceKilz |
| 2022-03-12 | Add test cases for inserting and deleting data...* tests/unit/db/test_sample_data.py (test_insert_sample_data): Test inserting
data.
(test_delete_sample_data): Test deleting data.
| BonfaceKilz |
| 2022-03-12 | Remove check for inserted data when inserting individual data...* gn3/db/sample_data.py (insert_sample_data)[__insert_data]: Move check to the
main body. With this check here, you have 3 redundant checks. For a successful
insert, it will insert the first value to the `PublishData` table and ignore the
rest of the inserts.
| BonfaceKilz |
| 2022-03-12 | Make `_map` a constant...* gn3/db/sample_data.py: Now constant, `_MAP`.
(delete_sample_data)[__delete_data]: Replace `_map` with `_MAP`.
(insert_sample_data)[__insert_data]: Ditto.
| BonfaceKilz |
| 2022-03-12 | Fix faulty SQL query string when deleting case-attributes | BonfaceKilz |
| 2022-03-12 | Explicitly get CaseAttributeId and fix broken sql query...* gn3/db/sample_data.py (insert_sample_data): Use correct query string. Also,
use CaseAttributeId to determine whether case-attributes were inserted. If so,
do not attempt an insert.
| BonfaceKilz |
| 2022-03-12 | Remove duplicate params...* gn3/db/sample_data.py (insert_sample_data)[__insert_case_attribute]: Remove
extra parameters.
| BonfaceKilz |
| 2022-03-12 | Remove dead code | BonfaceKilz |
| 2022-03-12 | Check whether publish data already exists before inserting...* gn3/db/sample_data.py (insert_sample_data): If data already exists in the
table, do not attempt an insert; otherwise, an error will be generated.
| BonfaceKilz |
| 2022-03-12 | Fill in empty values in csv text with: "x"...* gn3/csvcmp.py (fill_csv): Update this function to allow empty lists to be
filled with the default value(set in the args).
* tests/unit/test_csvcmp.py (test_fill_csv): Update test to capture above.
| BonfaceKilz |
| 2022-03-12 | Remove test cases related to sample data...Most of this functions were moved to their own module.
| BonfaceKilz |
| 2022-03-12 | Fetch id's separately for the insertion edge-case...* gn3/db/sample_data (get_sample_data_ids): Add an extra condition that caters
for inserts: during inserts, joins won't work when fetching the strain_id,
publishdata_id, and strain_name. In this case, just create 2 separate queries
to do that work.
| BonfaceKilz |
| 2022-03-12 | Extract a strain name given a csv string and it's header...* gn3/csvcmp.py (extract_strain_name): New function.
* gn3/db/sample_data (delete_sample_data): Use the aforementioned function.
(insert_sample_data): Ditto.
* tests/unit/test_csvcmp: Test cases for above.
| BonfaceKilz |
| 2022-03-12 | Allow deleting case-attribute data during deletion...* gn3/db/sample_data.py (delete_sample_data): Modify this function to allow
deleting case-attribute values.
| BonfaceKilz |
| 2022-03-12 | Allow inserting case-attribute data during inserts...* gn3/db/sample_data.py (insert_sample_data): Modify this function to allow
inserting case-attribute values.
| BonfaceKilz |
| 2022-03-12 | Fetch InbredSetId...* gn3/db/sample_data.py (get_sample_data_ids): Extend to also fetch
InbredSetId.
(update_sample_data): Discard the returned value of InbredSetId.
(delete_sample_data): Ditto.
| BonfaceKilz |
| 2022-03-12 | Create a new function for retrieving strain_id and publishdata_id...* gn3/db/sample_data.py: Import Any, Tuple.
(get_sample_data_ids): New function that fetches the strain_id and
publishdata_id of a given data point.
(update_sample_data): Use `get_sample_data_ids`.
(delete_sample_data): Ditto.
(insert_sample_data): Ditto.
| BonfaceKilz |
| 2022-03-12 | Move operations on sample_data to it's own module | BonfaceKilz |
| 2022-03-12 | Don't add extra key "Column" to dict if there are no changes...gn3/csvcmp.py (csv_diff): If the diff is empty, don't add an extra key
"Column" to the dictionary.
tests/unit/test_csvcmp (test_csv_diff_only_column_change): Add test-case for
the above.
| BonfaceKilz |
| 2022-03-12 | Test edges cases for csv files when running csvdiff...* tests/unit/test_csvcmp.py (test_csv_diff): Delete it.
(test_csv_diff_same_columns): Test csv_diff against csv texts with the same
columns.
(test_csv_diff_different_columns): Test csv texts against csv texts with
different varying columns.
| BonfaceKilz |
| 2022-03-12 | Fill CSV text if there are non-even rows...Should you try to run `csvdiff` against 2 csv files with either file having a
non-even columns, there will be an error. As such, the csv files need to be
"filled" before running `csvdiff`.
* gn3/csvcmp (csv_diff): For non-even rows in the csv files, fill the csv
rows.
| BonfaceKilz |
| 2022-03-12 | Create new method for filling csv with a default value...* gn3/csvcmp.py (fill_csv): Given a csv text with uneven or incomplete fields
whole length are less than width, fill them with a value which defaults to
"x".
* tests/unit/test_csvcmp.py (test_fill_csv): Test cases for the above.
| BonfaceKilz |
| 2022-03-12 | Replace "all" with "and"...* gn3/csvcmp.py (remove_insignificant_edits): "all" evaluates all elements and
throws an error if when `abs(float(x) - float(y)) < epsilon` is processed. Use
"and" instead because of it's short-circuiting behaviour.
| BonfaceKilz |
| 2022-03-12 | Store columns in the output dict...When inserting, deleting, or editing case-attributes, we need the column
headers in order to be able to know identify the attribute of interest.
* gn3/csvcmp.py (csv_diff): Add extra "Column" key in returned dict.
| BonfaceKilz |
| 2022-03-12 | Add methods for working with csv data...gn3/csvcmp.py: New file
(create_dirs_if_not_exists): From a list of dirs, create them if they don't
exist.
(remove_insignificant_edits): Given a dict with a "Modification" key, remove
edits with "delta < ε".
(csv_diff): Generate a csv_diff using the "csvdiff" tool packaged in guix.
tests/unit/test_csvcmp.py: Add some tests for "gn3/csvcmp.py"
| BonfaceKilz |
| 2022-03-12 | db: Fix error in SQL query...* gn3/db/traits.py (get_trait_csv_sample_data): Update SQL to fix runtime
errors.
| BonfaceKilz |
| 2022-03-12 | Fix pylint error | BonfaceKilz |
| 2022-03-12 | Append case attributes to csv data if they exist | BonfaceKilz |
| 2022-03-12 | db: Extend csv query to fetch case attributes...* gn3/db/traits.py (get_trait_csv_sample_data): Fetch case attribute data if
it exists.
| BonfaceKilz |
| 2022-03-12 | Revert "db: Fetch correct sample data"...This reverts commit 710769e84b3bc6a2bdd66effdbac0659272ed511.
| BonfaceKilz |
| 2022-03-11 | Fix typing errors | Frederick Muriuki Muriithi |
| 2022-03-11 | Fix some linting issues | Frederick Muriuki Muriithi |
| 2022-03-08 | Remove unused function and its tests | Frederick Muriuki Muriithi |
| 2022-03-08 | Fix tests, and issues caught by tests...Fix some issues caught by tests due to changes introducing the hand-off of the
partial correlations computations to an external process
Fix some issues due to the changes that introduce context managers for
database connections
Update some tests to take the above two changes into consideration
| Frederick Muriuki Muriithi |
| 2022-03-08 | Create database connections within context managers...Use the `with` context manager to open database connections, so as to ensure
that those connections are closed once the call is completed. This hopefully
avoids the 'too many connections' error
| Frederick Muriuki Muriithi |
| 2022-03-04 | Automatically decode Redis strings | Frederick Muriuki Muriithi |
| 2022-03-04 | Document basic partial correlations dependencies...* Document the things that the partial correlations feature depends on to work
| Frederick Muriuki Muriithi |
| 2022-03-03 | Add endpoint for checking state of external processes...Long-running computations are handed off to external processes. This avoids
timeouts in the webserver, and also reduces chances of instability of the
webserver.
The results of these long-running computations are needed eventually, so this
commit provides a way to check for the state of the computation, and the
results if any.
| Frederick Muriuki Muriithi |
| 2022-03-03 | Do incremental backoff if there are no jobs on the queue...Since the worker polls a queue for jobs, it can get into a busy poll. This was
the reason that there was a delay of 0.1 seconds between each poll instance.
This commit takes this a little further by doing an incremental backoff, where
it waits longer and longer after each poll where it does not find a job
available, up to an arbitrary maximum.
| Frederick Muriuki Muriithi |
| 2022-03-03 | Run partial correlations in an external process...Run the partial correlations code in an external python process decoupling it
from the server and making it asynchronous.
Summary of changes:
* gn3/api/correlation.py:
- Remove response processing code
- Queue partial corrs processing
- Create new endpoint to get results
* gn3/commands.py
- Compose the pcorrs command to be run in an external process
- Enable running of subprocess commands with list args
* gn3/responses/__init__.py: new module indicator file
* gn3/responses/pcorrs_responses.py: Hold response processing code extracted
from ~gn3.api.correlations.py~ file
* scripts/partial_correlations.py: CLI script to process the pcorrs
* sheepdog/worker.py:
- Add the *genenetwork3* path at the beginning of the ~sys.path~ list to
override any GN3 in the site-packages
- Add any environment variables to be set for the command to be run
| Frederick Muriuki Muriithi |
| 2022-02-25 | Fix issue where 0's were treated as False for the primary trait in...correlations
In the original version of the if statement* I believe it was
interpreted as "if a_val and (b_val is not None)". This caused
values of 0 for a_val (the primary trait's values) to be evaluated as
False.
I changed it to compare both a_val and b_val to None. This seems to have
fixed the issue.
* if (a_val and b_val is not None)
| zsloan |
| 2022-02-24 | gn3: computations: Call Popen with context manager....Context managers should be preferred when allocating resources.
* gn3/computations/wgcna.py (stream_cmd_output): Call Popen with context
manager.
| Arun Isaac |
| 2022-02-24 | gn3: Explicitly specify UTF-8 to be the file encoding....When the encoding is not specified explicitly, the system default encoding is
used. This is not recommended.
* gn3/computations/ctl.py (call_ctl_script),
gn3/computations/gemma.py (generate_pheno_txt_file),
gn3/computations/parsers.py (parse_genofile),
gn3/computations/partial_correlations.py (partial_correlations_fast),
gn3/computations/rqtl.py (process_rqtl_output, process_perm_output),
gn3/computations/wgcna.py (dump_wgcna_data, call_wgcna_script),
gn3/fs_helpers.py (jsonfile_to_dict): Explicitly specify UTF-8 to be the file
encoding.
*
tests/unit/computations/test_gemma.py (TestGemma.test_generate_pheno_txt_file),
tests/unit/computations/test_wgcna.py (TestWgcna.test_create_json_file): Test
for call to open with encoding='utf-8' argument.
| Arun Isaac |
