aboutsummaryrefslogtreecommitdiff
AgeCommit message (Expand)Author
2022-03-12README: Replace "unit-test" instructions with "pytest"BonfaceKilz
2022-03-12Compose csv-diff command within single quotes...* gn3/csvcmp.py (csv_diff): Use single quotes. There was a change in 6d39c92 that broke this. BonfaceKilz
2022-03-12Delete noisy "print" statementBonfaceKilz
2022-03-12Store the first element as strain_idBonfaceKilz
2022-03-12Append the strain name when extracting "actions"...* gn3/db/sample_data.py (__extract_actions): During updates, make sure that the strain name is part of the returned string when extracting "actions". * tests/unit/db/test_sample_data.py: Add test cases for the above. BonfaceKilz
2022-03-12Apply auto-pep8 to sample_data.py and it's test fileBonfaceKilz
2022-03-12Add missing return type-annotations...* tests/unit/db/test_sample_data.py (delete_sample_data): Add missing return type for type annotations. (insert_sample_data): Ditto. BonfaceKilz
2022-03-12Update how data is updated by re-using existing functions...* gn3/db/sample_data.py (get_sample_data_ids): Re-use "delete_sample_data" and "insert_sample_data" when updating data; and also add logic for updating modified data. * tests/unit/db/test_sample_data.py: Add tests for the above. BonfaceKilz
2022-03-12Create action dict that's created when updating data...* gn3/db/sample_data.py (__extract_actions): An update on a vector of data can contain: inserts, deletes and updates. This functions extracts these actions during an update. * tests/unit/db/test_sample_data.py (test_extract_actions): Add test-case for the above. BonfaceKilz
2022-03-12Add test cases for inserting and deleting data...* tests/unit/db/test_sample_data.py (test_insert_sample_data): Test inserting data. (test_delete_sample_data): Test deleting data. BonfaceKilz
2022-03-12Remove check for inserted data when inserting individual data...* gn3/db/sample_data.py (insert_sample_data)[__insert_data]: Move check to the main body. With this check here, you have 3 redundant checks. For a successful insert, it will insert the first value to the `PublishData` table and ignore the rest of the inserts. BonfaceKilz
2022-03-12Make `_map` a constant...* gn3/db/sample_data.py: Now constant, `_MAP`. (delete_sample_data)[__delete_data]: Replace `_map` with `_MAP`. (insert_sample_data)[__insert_data]: Ditto. BonfaceKilz
2022-03-12Fix faulty SQL query string when deleting case-attributesBonfaceKilz
2022-03-12Explicitly get CaseAttributeId and fix broken sql query...* gn3/db/sample_data.py (insert_sample_data): Use correct query string. Also, use CaseAttributeId to determine whether case-attributes were inserted. If so, do not attempt an insert. BonfaceKilz
2022-03-12Remove duplicate params...* gn3/db/sample_data.py (insert_sample_data)[__insert_case_attribute]: Remove extra parameters. BonfaceKilz
2022-03-12Remove dead codeBonfaceKilz
2022-03-12Check whether publish data already exists before inserting...* gn3/db/sample_data.py (insert_sample_data): If data already exists in the table, do not attempt an insert; otherwise, an error will be generated. BonfaceKilz
2022-03-12Fill in empty values in csv text with: "x"...* gn3/csvcmp.py (fill_csv): Update this function to allow empty lists to be filled with the default value(set in the args). * tests/unit/test_csvcmp.py (test_fill_csv): Update test to capture above. BonfaceKilz
2022-03-12Remove test cases related to sample data...Most of this functions were moved to their own module. BonfaceKilz
2022-03-12Fetch id's separately for the insertion edge-case...* gn3/db/sample_data (get_sample_data_ids): Add an extra condition that caters for inserts: during inserts, joins won't work when fetching the strain_id, publishdata_id, and strain_name. In this case, just create 2 separate queries to do that work. BonfaceKilz
2022-03-12Extract a strain name given a csv string and it's header...* gn3/csvcmp.py (extract_strain_name): New function. * gn3/db/sample_data (delete_sample_data): Use the aforementioned function. (insert_sample_data): Ditto. * tests/unit/test_csvcmp: Test cases for above. BonfaceKilz
2022-03-12Allow deleting case-attribute data during deletion...* gn3/db/sample_data.py (delete_sample_data): Modify this function to allow deleting case-attribute values. BonfaceKilz
2022-03-12Allow inserting case-attribute data during inserts...* gn3/db/sample_data.py (insert_sample_data): Modify this function to allow inserting case-attribute values. BonfaceKilz
2022-03-12Fetch InbredSetId...* gn3/db/sample_data.py (get_sample_data_ids): Extend to also fetch InbredSetId. (update_sample_data): Discard the returned value of InbredSetId. (delete_sample_data): Ditto. BonfaceKilz
2022-03-12Create a new function for retrieving strain_id and publishdata_id...* gn3/db/sample_data.py: Import Any, Tuple. (get_sample_data_ids): New function that fetches the strain_id and publishdata_id of a given data point. (update_sample_data): Use `get_sample_data_ids`. (delete_sample_data): Ditto. (insert_sample_data): Ditto. BonfaceKilz
2022-03-12Move operations on sample_data to it's own moduleBonfaceKilz
2022-03-12Don't add extra key "Column" to dict if there are no changes...gn3/csvcmp.py (csv_diff): If the diff is empty, don't add an extra key "Column" to the dictionary. tests/unit/test_csvcmp (test_csv_diff_only_column_change): Add test-case for the above. BonfaceKilz
2022-03-12Test edges cases for csv files when running csvdiff...* tests/unit/test_csvcmp.py (test_csv_diff): Delete it. (test_csv_diff_same_columns): Test csv_diff against csv texts with the same columns. (test_csv_diff_different_columns): Test csv texts against csv texts with different varying columns. BonfaceKilz
2022-03-12Fill CSV text if there are non-even rows...Should you try to run `csvdiff` against 2 csv files with either file having a non-even columns, there will be an error. As such, the csv files need to be "filled" before running `csvdiff`. * gn3/csvcmp (csv_diff): For non-even rows in the csv files, fill the csv rows. BonfaceKilz
2022-03-12Create new method for filling csv with a default value...* gn3/csvcmp.py (fill_csv): Given a csv text with uneven or incomplete fields whole length are less than width, fill them with a value which defaults to "x". * tests/unit/test_csvcmp.py (test_fill_csv): Test cases for the above. BonfaceKilz
2022-03-12Replace "all" with "and"...* gn3/csvcmp.py (remove_insignificant_edits): "all" evaluates all elements and throws an error if when `abs(float(x) - float(y)) < epsilon` is processed. Use "and" instead because of it's short-circuiting behaviour. BonfaceKilz
2022-03-12Store columns in the output dict...When inserting, deleting, or editing case-attributes, we need the column headers in order to be able to know identify the attribute of interest. * gn3/csvcmp.py (csv_diff): Add extra "Column" key in returned dict. BonfaceKilz
2022-03-12Add methods for working with csv data...gn3/csvcmp.py: New file (create_dirs_if_not_exists): From a list of dirs, create them if they don't exist. (remove_insignificant_edits): Given a dict with a "Modification" key, remove edits with "delta < ε". (csv_diff): Generate a csv_diff using the "csvdiff" tool packaged in guix. tests/unit/test_csvcmp.py: Add some tests for "gn3/csvcmp.py" BonfaceKilz
2022-03-12db: Fix error in SQL query...* gn3/db/traits.py (get_trait_csv_sample_data): Update SQL to fix runtime errors. BonfaceKilz
2022-03-12Fix pylint errorBonfaceKilz
2022-03-12Append case attributes to csv data if they existBonfaceKilz
2022-03-12db: Extend csv query to fetch case attributes...* gn3/db/traits.py (get_trait_csv_sample_data): Fetch case attribute data if it exists. BonfaceKilz
2022-03-12Revert "db: Fetch correct sample data"...This reverts commit 710769e84b3bc6a2bdd66effdbac0659272ed511. BonfaceKilz
2022-03-11Fix typing errorsFrederick Muriuki Muriithi
2022-03-11Fix some linting issuesFrederick Muriuki Muriithi
2022-03-08Remove unused function and its testsFrederick Muriuki Muriithi
2022-03-08Fix tests, and issues caught by tests...Fix some issues caught by tests due to changes introducing the hand-off of the partial correlations computations to an external process Fix some issues due to the changes that introduce context managers for database connections Update some tests to take the above two changes into consideration Frederick Muriuki Muriithi
2022-03-08Create database connections within context managers...Use the `with` context manager to open database connections, so as to ensure that those connections are closed once the call is completed. This hopefully avoids the 'too many connections' error Frederick Muriuki Muriithi
2022-03-04Automatically decode Redis stringsFrederick Muriuki Muriithi
2022-03-04Document basic partial correlations dependencies...* Document the things that the partial correlations feature depends on to work Frederick Muriuki Muriithi
2022-03-03Add endpoint for checking state of external processes...Long-running computations are handed off to external processes. This avoids timeouts in the webserver, and also reduces chances of instability of the webserver. The results of these long-running computations are needed eventually, so this commit provides a way to check for the state of the computation, and the results if any. Frederick Muriuki Muriithi
2022-03-03Do incremental backoff if there are no jobs on the queue...Since the worker polls a queue for jobs, it can get into a busy poll. This was the reason that there was a delay of 0.1 seconds between each poll instance. This commit takes this a little further by doing an incremental backoff, where it waits longer and longer after each poll where it does not find a job available, up to an arbitrary maximum. Frederick Muriuki Muriithi
2022-03-03Run partial correlations in an external process...Run the partial correlations code in an external python process decoupling it from the server and making it asynchronous. Summary of changes: * gn3/api/correlation.py: - Remove response processing code - Queue partial corrs processing - Create new endpoint to get results * gn3/commands.py - Compose the pcorrs command to be run in an external process - Enable running of subprocess commands with list args * gn3/responses/__init__.py: new module indicator file * gn3/responses/pcorrs_responses.py: Hold response processing code extracted from ~gn3.api.correlations.py~ file * scripts/partial_correlations.py: CLI script to process the pcorrs * sheepdog/worker.py: - Add the *genenetwork3* path at the beginning of the ~sys.path~ list to override any GN3 in the site-packages - Add any environment variables to be set for the command to be run Frederick Muriuki Muriithi
2022-02-25Fix issue where 0's were treated as False for the primary trait in...correlations In the original version of the if statement* I believe it was interpreted as "if a_val and (b_val is not None)". This caused values of 0 for a_val (the primary trait's values) to be evaluated as False. I changed it to compare both a_val and b_val to None. This seems to have fixed the issue. * if (a_val and b_val is not None) zsloan
2022-02-24gn3: computations: Call Popen with context manager....Context managers should be preferred when allocating resources. * gn3/computations/wgcna.py (stream_cmd_output): Call Popen with context manager. Arun Isaac