aboutsummaryrefslogtreecommitdiff
path: root/gn3
AgeCommit message (Collapse)Author
2022-03-12Append the strain name when extracting "actions"BonfaceKilz
* gn3/db/sample_data.py (__extract_actions): During updates, make sure that the strain name is part of the returned string when extracting "actions". * tests/unit/db/test_sample_data.py: Add test cases for the above.
2022-03-12Apply auto-pep8 to sample_data.py and it's test fileBonfaceKilz
2022-03-12Add missing return type-annotationsBonfaceKilz
* tests/unit/db/test_sample_data.py (delete_sample_data): Add missing return type for type annotations. (insert_sample_data): Ditto.
2022-03-12Update how data is updated by re-using existing functionsBonfaceKilz
* gn3/db/sample_data.py (get_sample_data_ids): Re-use "delete_sample_data" and "insert_sample_data" when updating data; and also add logic for updating modified data. * tests/unit/db/test_sample_data.py: Add tests for the above.
2022-03-12Create action dict that's created when updating dataBonfaceKilz
* gn3/db/sample_data.py (__extract_actions): An update on a vector of data can contain: inserts, deletes and updates. This functions extracts these actions during an update. * tests/unit/db/test_sample_data.py (test_extract_actions): Add test-case for the above.
2022-03-12Remove check for inserted data when inserting individual dataBonfaceKilz
* gn3/db/sample_data.py (insert_sample_data)[__insert_data]: Move check to the main body. With this check here, you have 3 redundant checks. For a successful insert, it will insert the first value to the `PublishData` table and ignore the rest of the inserts.
2022-03-12Make `_map` a constantBonfaceKilz
* gn3/db/sample_data.py: Now constant, `_MAP`. (delete_sample_data)[__delete_data]: Replace `_map` with `_MAP`. (insert_sample_data)[__insert_data]: Ditto.
2022-03-12Fix faulty SQL query string when deleting case-attributesBonfaceKilz
2022-03-12Explicitly get CaseAttributeId and fix broken sql queryBonfaceKilz
* gn3/db/sample_data.py (insert_sample_data): Use correct query string. Also, use CaseAttributeId to determine whether case-attributes were inserted. If so, do not attempt an insert.
2022-03-12Remove duplicate paramsBonfaceKilz
* gn3/db/sample_data.py (insert_sample_data)[__insert_case_attribute]: Remove extra parameters.
2022-03-12Remove dead codeBonfaceKilz
2022-03-12Check whether publish data already exists before insertingBonfaceKilz
* gn3/db/sample_data.py (insert_sample_data): If data already exists in the table, do not attempt an insert; otherwise, an error will be generated.
2022-03-12Fill in empty values in csv text with: "x"BonfaceKilz
* gn3/csvcmp.py (fill_csv): Update this function to allow empty lists to be filled with the default value(set in the args). * tests/unit/test_csvcmp.py (test_fill_csv): Update test to capture above.
2022-03-12Fetch id's separately for the insertion edge-caseBonfaceKilz
* gn3/db/sample_data (get_sample_data_ids): Add an extra condition that caters for inserts: during inserts, joins won't work when fetching the strain_id, publishdata_id, and strain_name. In this case, just create 2 separate queries to do that work.
2022-03-12Extract a strain name given a csv string and it's headerBonfaceKilz
* gn3/csvcmp.py (extract_strain_name): New function. * gn3/db/sample_data (delete_sample_data): Use the aforementioned function. (insert_sample_data): Ditto. * tests/unit/test_csvcmp: Test cases for above.
2022-03-12Allow deleting case-attribute data during deletionBonfaceKilz
* gn3/db/sample_data.py (delete_sample_data): Modify this function to allow deleting case-attribute values.
2022-03-12Allow inserting case-attribute data during insertsBonfaceKilz
* gn3/db/sample_data.py (insert_sample_data): Modify this function to allow inserting case-attribute values.
2022-03-12Fetch InbredSetIdBonfaceKilz
* gn3/db/sample_data.py (get_sample_data_ids): Extend to also fetch InbredSetId. (update_sample_data): Discard the returned value of InbredSetId. (delete_sample_data): Ditto.
2022-03-12Create a new function for retrieving strain_id and publishdata_idBonfaceKilz
* gn3/db/sample_data.py: Import Any, Tuple. (get_sample_data_ids): New function that fetches the strain_id and publishdata_id of a given data point. (update_sample_data): Use `get_sample_data_ids`. (delete_sample_data): Ditto. (insert_sample_data): Ditto.
2022-03-12Move operations on sample_data to it's own moduleBonfaceKilz
2022-03-12Don't add extra key "Column" to dict if there are no changesBonfaceKilz
gn3/csvcmp.py (csv_diff): If the diff is empty, don't add an extra key "Column" to the dictionary. tests/unit/test_csvcmp (test_csv_diff_only_column_change): Add test-case for the above.
2022-03-12Fill CSV text if there are non-even rowsBonfaceKilz
Should you try to run `csvdiff` against 2 csv files with either file having a non-even columns, there will be an error. As such, the csv files need to be "filled" before running `csvdiff`. * gn3/csvcmp (csv_diff): For non-even rows in the csv files, fill the csv rows.
2022-03-12Create new method for filling csv with a default valueBonfaceKilz
* gn3/csvcmp.py (fill_csv): Given a csv text with uneven or incomplete fields whole length are less than width, fill them with a value which defaults to "x". * tests/unit/test_csvcmp.py (test_fill_csv): Test cases for the above.
2022-03-12Replace "all" with "and"BonfaceKilz
* gn3/csvcmp.py (remove_insignificant_edits): "all" evaluates all elements and throws an error if when `abs(float(x) - float(y)) < epsilon` is processed. Use "and" instead because of it's short-circuiting behaviour.
2022-03-12Store columns in the output dictBonfaceKilz
When inserting, deleting, or editing case-attributes, we need the column headers in order to be able to know identify the attribute of interest. * gn3/csvcmp.py (csv_diff): Add extra "Column" key in returned dict.
2022-03-12Add methods for working with csv dataBonfaceKilz
gn3/csvcmp.py: New file (create_dirs_if_not_exists): From a list of dirs, create them if they don't exist. (remove_insignificant_edits): Given a dict with a "Modification" key, remove edits with "delta < ε". (csv_diff): Generate a csv_diff using the "csvdiff" tool packaged in guix. tests/unit/test_csvcmp.py: Add some tests for "gn3/csvcmp.py"
2022-03-12db: Fix error in SQL queryBonfaceKilz
* gn3/db/traits.py (get_trait_csv_sample_data): Update SQL to fix runtime errors.
2022-03-12Fix pylint errorBonfaceKilz
2022-03-12Append case attributes to csv data if they existBonfaceKilz
2022-03-12db: Extend csv query to fetch case attributesBonfaceKilz
* gn3/db/traits.py (get_trait_csv_sample_data): Fetch case attribute data if it exists.
2022-03-12Revert "db: Fetch correct sample data"BonfaceKilz
This reverts commit 710769e84b3bc6a2bdd66effdbac0659272ed511.
2022-03-11Fix typing errorsFrederick Muriuki Muriithi
2022-03-11Fix some linting issuesFrederick Muriuki Muriithi
2022-03-08Remove unused function and its testsFrederick Muriuki Muriithi
2022-03-08Fix tests, and issues caught by testsFrederick Muriuki Muriithi
Fix some issues caught by tests due to changes introducing the hand-off of the partial correlations computations to an external process Fix some issues due to the changes that introduce context managers for database connections Update some tests to take the above two changes into consideration
2022-03-08Create database connections within context managersFrederick Muriuki Muriithi
Use the `with` context manager to open database connections, so as to ensure that those connections are closed once the call is completed. This hopefully avoids the 'too many connections' error
2022-03-04Automatically decode Redis stringsFrederick Muriuki Muriithi
2022-03-03Add endpoint for checking state of external processesFrederick Muriuki Muriithi
Long-running computations are handed off to external processes. This avoids timeouts in the webserver, and also reduces chances of instability of the webserver. The results of these long-running computations are needed eventually, so this commit provides a way to check for the state of the computation, and the results if any.
2022-03-03Run partial correlations in an external processFrederick Muriuki Muriithi
Run the partial correlations code in an external python process decoupling it from the server and making it asynchronous. Summary of changes: * gn3/api/correlation.py: - Remove response processing code - Queue partial corrs processing - Create new endpoint to get results * gn3/commands.py - Compose the pcorrs command to be run in an external process - Enable running of subprocess commands with list args * gn3/responses/__init__.py: new module indicator file * gn3/responses/pcorrs_responses.py: Hold response processing code extracted from ~gn3.api.correlations.py~ file * scripts/partial_correlations.py: CLI script to process the pcorrs * sheepdog/worker.py: - Add the *genenetwork3* path at the beginning of the ~sys.path~ list to override any GN3 in the site-packages - Add any environment variables to be set for the command to be run
2022-02-25Fix issue where 0's were treated as False for the primary trait inzsloan
correlations In the original version of the if statement* I believe it was interpreted as "if a_val and (b_val is not None)". This caused values of 0 for a_val (the primary trait's values) to be evaluated as False. I changed it to compare both a_val and b_val to None. This seems to have fixed the issue. * if (a_val and b_val is not None)
2022-02-24gn3: computations: Call Popen with context manager.Arun Isaac
Context managers should be preferred when allocating resources. * gn3/computations/wgcna.py (stream_cmd_output): Call Popen with context manager.
2022-02-24gn3: Explicitly specify UTF-8 to be the file encoding.Arun Isaac
When the encoding is not specified explicitly, the system default encoding is used. This is not recommended. * gn3/computations/ctl.py (call_ctl_script), gn3/computations/gemma.py (generate_pheno_txt_file), gn3/computations/parsers.py (parse_genofile), gn3/computations/partial_correlations.py (partial_correlations_fast), gn3/computations/rqtl.py (process_rqtl_output, process_perm_output), gn3/computations/wgcna.py (dump_wgcna_data, call_wgcna_script), gn3/fs_helpers.py (jsonfile_to_dict): Explicitly specify UTF-8 to be the file encoding. * tests/unit/computations/test_gemma.py (TestGemma.test_generate_pheno_txt_file), tests/unit/computations/test_wgcna.py (TestWgcna.test_create_json_file): Test for call to open with encoding='utf-8' argument.
2022-02-21Fix minor issues introduced while fixing linting errorsFrederick Muriuki Muriithi
2022-02-21Fix a myriad of linter issuesFrederick Muriuki Muriithi
* Use `with` in place of plain `open` * Use f-strings in place of `str.format()` * Remove string interpolation from queries - provide data as query parameters * other minor fixes
2022-02-21Test partial corrs API with mix of existing and non-existing control traitsFrederick Muriuki Muriithi
Test that the partial correlations endpoint handles a mix of existing and non-existing control traits gracefully and issues a warning to the user. Summary of changes: * gn3/computations/partial_correlations.py: Issue a warning for all non-existing control traits * gn3/db/partial_correlations.py: update queries - use `INNER JOIN` for tables instead of comma-separated list of tables * tests/integration/conftest.py: Add `db_conn` fixture to provide a database connection to the tests. This will probably be changed in the future to connect to a temporary database for tests. * tests/integration/test_partial_correlations.py: Add test to check for correct behaviour with a mix of existing and non-existing control traits
2022-02-19Test partial corrs endpoint with non-existing control traitsFrederick Muriuki Muriithi
Test that if the endpoint is queried and not a single one of the control traits exists in the database, then the endpoint will respond with a 404 (not-found) status code. Summary of changes: * gn3/computations/partial_correlations.py: Check whether any control trait is found. If none is found, return "not-found" message. * gn3/db/partial_correlations.py: Fix bug in Geno query. * tests/integration/test_partial_correlations.py: Add test for non-existing control traits. Rename function to make it clearer what it is testing for. Remove obsoleted comments.
2022-02-18Remove code trying to query non-existent `TempFreeze` tableFrederick Muriuki Muriithi
The code was migrated from GN1 with a faulty assumption that all trait types have a corresponding `*Freeze` table in the database. This assumption is not true for the `Temp` traits. This commit removes the buggy code.
2022-02-18Test partial correlations endpoint with non-existent primary traitsFrederick Muriuki Muriithi
Test that the partial correlations endpoint responds with an appropriate "not-found" message and the corresponding 404 status code in the case where a request is made and the primary trait requested for does not exist in the database. Summary of the changes in each file: * gn3/api/correlation.py: generalise the building of the response * gn3/computations/partial_correlations.py: return with a "not-found" if the primary trait does not exist in the database * gn3/db/partial_correlations.py: Fix a number of bugs that led to exceptions in the case that the primary trait did not exist * pytest.ini: register a `slow` pytest marker * tests/integration/test_partial_correlations.py: Add a new test to check for an appropriate 404 response in case of a primary trait that does not exist in the database.
2022-02-17Test partial correlations endpoint with missing data in POST requestFrederick Muriuki Muriithi
Add a test for the partial correlations endpoint, with: - no data in the request - missing items in the data Fix the bugs caught by the test
2022-02-17Add property tests for `dictify_by_samples`Frederick Muriuki Muriithi
Add property tests using pytest and hypothesis to test that the expected properties hold for the `gn3.computations.partial_correlations.dictify_by_samples` function.