aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2022-03-08Create database connections within context managersFrederick Muriuki Muriithi
Use the `with` context manager to open database connections, so as to ensure that those connections are closed once the call is completed. This hopefully avoids the 'too many connections' error
2022-03-04Automatically decode Redis stringsFrederick Muriuki Muriithi
2022-03-04Document basic partial correlations dependenciesFrederick Muriuki Muriithi
* Document the things that the partial correlations feature depends on to work
2022-03-03Add endpoint for checking state of external processesFrederick Muriuki Muriithi
Long-running computations are handed off to external processes. This avoids timeouts in the webserver, and also reduces chances of instability of the webserver. The results of these long-running computations are needed eventually, so this commit provides a way to check for the state of the computation, and the results if any.
2022-03-03Do incremental backoff if there are no jobs on the queueFrederick Muriuki Muriithi
Since the worker polls a queue for jobs, it can get into a busy poll. This was the reason that there was a delay of 0.1 seconds between each poll instance. This commit takes this a little further by doing an incremental backoff, where it waits longer and longer after each poll where it does not find a job available, up to an arbitrary maximum.
2022-03-03Run partial correlations in an external processFrederick Muriuki Muriithi
Run the partial correlations code in an external python process decoupling it from the server and making it asynchronous. Summary of changes: * gn3/api/correlation.py: - Remove response processing code - Queue partial corrs processing - Create new endpoint to get results * gn3/commands.py - Compose the pcorrs command to be run in an external process - Enable running of subprocess commands with list args * gn3/responses/__init__.py: new module indicator file * gn3/responses/pcorrs_responses.py: Hold response processing code extracted from ~gn3.api.correlations.py~ file * scripts/partial_correlations.py: CLI script to process the pcorrs * sheepdog/worker.py: - Add the *genenetwork3* path at the beginning of the ~sys.path~ list to override any GN3 in the site-packages - Add any environment variables to be set for the command to be run
2022-02-25Fix issue where 0's were treated as False for the primary trait inzsloan
correlations In the original version of the if statement* I believe it was interpreted as "if a_val and (b_val is not None)". This caused values of 0 for a_val (the primary trait's values) to be evaluated as False. I changed it to compare both a_val and b_val to None. This seems to have fixed the issue. * if (a_val and b_val is not None)
2022-02-24gn3: computations: Call Popen with context manager.Arun Isaac
Context managers should be preferred when allocating resources. * gn3/computations/wgcna.py (stream_cmd_output): Call Popen with context manager.
2022-02-24gn3: Explicitly specify UTF-8 to be the file encoding.Arun Isaac
When the encoding is not specified explicitly, the system default encoding is used. This is not recommended. * gn3/computations/ctl.py (call_ctl_script), gn3/computations/gemma.py (generate_pheno_txt_file), gn3/computations/parsers.py (parse_genofile), gn3/computations/partial_correlations.py (partial_correlations_fast), gn3/computations/rqtl.py (process_rqtl_output, process_perm_output), gn3/computations/wgcna.py (dump_wgcna_data, call_wgcna_script), gn3/fs_helpers.py (jsonfile_to_dict): Explicitly specify UTF-8 to be the file encoding. * tests/unit/computations/test_gemma.py (TestGemma.test_generate_pheno_txt_file), tests/unit/computations/test_wgcna.py (TestWgcna.test_create_json_file): Test for call to open with encoding='utf-8' argument.
2022-02-21Fix minor issues introduced while fixing linting errorsFrederick Muriuki Muriithi
2022-02-21Fix a myriad of linter issuesFrederick Muriuki Muriithi
* Use `with` in place of plain `open` * Use f-strings in place of `str.format()` * Remove string interpolation from queries - provide data as query parameters * other minor fixes
2022-02-21Add test to ensure samples are read correctly from .geno filesFrederick Muriuki Muriithi
* tests/unit/db/test_genotypes2.py: New file
2022-02-21Test partial corrs API with mix of existing and non-existing control traitsFrederick Muriuki Muriithi
Test that the partial correlations endpoint handles a mix of existing and non-existing control traits gracefully and issues a warning to the user. Summary of changes: * gn3/computations/partial_correlations.py: Issue a warning for all non-existing control traits * gn3/db/partial_correlations.py: update queries - use `INNER JOIN` for tables instead of comma-separated list of tables * tests/integration/conftest.py: Add `db_conn` fixture to provide a database connection to the tests. This will probably be changed in the future to connect to a temporary database for tests. * tests/integration/test_partial_correlations.py: Add test to check for correct behaviour with a mix of existing and non-existing control traits
2022-02-19Test partial corrs endpoint with non-existing control traitsFrederick Muriuki Muriithi
Test that if the endpoint is queried and not a single one of the control traits exists in the database, then the endpoint will respond with a 404 (not-found) status code. Summary of changes: * gn3/computations/partial_correlations.py: Check whether any control trait is found. If none is found, return "not-found" message. * gn3/db/partial_correlations.py: Fix bug in Geno query. * tests/integration/test_partial_correlations.py: Add test for non-existing control traits. Rename function to make it clearer what it is testing for. Remove obsoleted comments.
2022-02-18Remove erroneous test based on wrong assumptionsFrederick Muriuki Muriithi
Related to commit 75dcfe295af57b16428c586cc11dbaa827a5feba This commit removes the related test that was checking for the wrong thing.
2022-02-18Remove code trying to query non-existent `TempFreeze` tableFrederick Muriuki Muriithi
The code was migrated from GN1 with a faulty assumption that all trait types have a corresponding `*Freeze` table in the database. This assumption is not true for the `Temp` traits. This commit removes the buggy code.
2022-02-18Test partial correlations endpoint with non-existent primary traitsFrederick Muriuki Muriithi
Test that the partial correlations endpoint responds with an appropriate "not-found" message and the corresponding 404 status code in the case where a request is made and the primary trait requested for does not exist in the database. Summary of the changes in each file: * gn3/api/correlation.py: generalise the building of the response * gn3/computations/partial_correlations.py: return with a "not-found" if the primary trait does not exist in the database * gn3/db/partial_correlations.py: Fix a number of bugs that led to exceptions in the case that the primary trait did not exist * pytest.ini: register a `slow` pytest marker * tests/integration/test_partial_correlations.py: Add a new test to check for an appropriate 404 response in case of a primary trait that does not exist in the database.
2022-02-17Test partial correlations endpoint with missing data in POST requestFrederick Muriuki Muriithi
Add a test for the partial correlations endpoint, with: - no data in the request - missing items in the data Fix the bugs caught by the test
2022-02-17Create client fixture for integration testsFrederick Muriuki Muriithi
Create a client fixture to help with the integration tests
2022-02-17Use pytest's "mark" feature to categorise testsFrederick Muriuki Muriithi
Use pytest's `mark` feature to explicitly categorise the tests and run them per category
2022-02-17Add property tests for `dictify_by_samples`Frederick Muriuki Muriithi
Add property tests using pytest and hypothesis to test that the expected properties hold for the `gn3.computations.partial_correlations.dictify_by_samples` function.
2022-02-12Provide custom class to run testsFrederick Muriuki Muriithi
2022-02-11sql: Remove database mapping code.Arun Isaac
These tools have been greatly improved and moved to a new home at https://git.genenetwork.org/arunisaac/dump-genenetwork-database * sql/map-database.sh, sql/schema-from-in-db-documentation.org, sql/schema-original.sql, sql/schema.png, sql/schema.sql, sql/schema.svg: New files.
2022-02-11Add scaffolding to run different types of testsFrederick Muriuki Muriithi
Enable running commands: - `python3 setup.py unit_check`: run the unit tests - `python3 setup.py integration_check`: run integration tests - `python3 setup.py performance_check`: run performance tests
2022-02-11Quote shell variables to prevent globbingFrederick Muriuki Muriithi
Quote the shell variables to prevent globbing and word splitting. Deactivate this check for the specific lines that require intentional word splitting
2022-02-09github: Remove GitHub actions.Arun Isaac
We have our own laminar CI and no longer need GitHub actions. * .github/workflows: Delete directory.
2022-02-09Remove obsolete `PublishFreeze` tableFrederick Muriuki Muriithi
2022-02-08Merge iterations to remove unnecessary computationsFrederick Muriuki Muriithi
Do all the work in a single iteration to avoid unnecessary iterations that hamper performance.
2022-02-08Remove multiprocessing for stabilityFrederick Muriuki Muriithi
Web servers are long-running processes, and python is not very good at cleaning up after itself especially in forked processes - this leads to memory errors in the web-server after a while. This commit removes the use of multiprocessing to avoid such failures.
2022-02-08Give sorting functions more descriptive namesFrederick Muriuki Muriithi
2022-02-08Use multiprocessing to speed up computationFrederick Muriuki Muriithi
This commit refactors the code to make it possible to use multiprocessing to speed up the computation of the partial correlations. The major refactor is to move the `__compute_trait_info__` function to the top-level of the module, and provide to it all the other necessary context via the new args.
2022-02-08Remove unnecessary computationFrederick Muriuki Muriithi
In Python3 when slicing, seq[:min(some_val, len(seq))] == seq[:some_val] because Python3 will just return a copy of the entire sequence if `some_val` happens to be larger/greater than the length of the sequence. This commit removes the unnecessary call to `min()`
2022-02-08db: traits: Enable deletion of individual sample valuesBonfaceKilz
If a user replaces an individual value with an "x", delete that date entry from the respective table. Deletion here is the only option since by default the Nstrain, PublishData and PublishSE don't accept null values. Note that deleting all 3 values is equivalent to removing the sample from the CSV file. * gn3/db/traits.py (update_sample_data): If a value is "x", delete it from the respective table.
2022-02-08db: traits: Allow insertions of "0" in resp. sample values tablesBonfaceKilz
When editing values from "x" to "0"(or any other value) when editing data, an "update" statement was being run; thereby no new value was being inserted. To the end user, modifying an "x" value to something else meant that no value was being inserted. This commit fixes that by doing an insert whenever a change from "x" to "0" is performed. * gn3/db/traits.py (update_sample_data): Add insert statements whenever an "update" statement returns a 0 row-count.
2022-02-08db: traits: Insert "count" instead of "error" in "NStrain" tableBonfaceKilz
2022-02-03db: Fetch correct sample dataBonfaceKilz
2022-02-02Fix R/qtl covar bugzsloan
The rqtl_wrapper script was throwing an error when only a single categorical covariate was used. This is apparently because "covars[,name]" throws an error in such a situation. Using just "covars" in such a situation prevents the error. So I just added an if statement checking the number of covariates. There might be some better way to deal with this in R, but this is the best I could come up with.
2022-02-02Remove PublishFreeze from retrieve_publish_trait_info queryzsloan
The PublishFreeze table isn't necessary in phenotype queries, since PublishFreeze.Id = InbredSet.Id (for the purposes of identifying traits, at least)
2022-02-02Fix bug where sample values of 0 were being treated as Falsezsloan
In line 91 of gn3/db/traits.py, there was an if statement "if record[key] else 'x'" that was treating values of 0 as False, so I changed it to explicitly check that values aren't None
2022-02-02Fix retrieve_publish_trait_data queryzsloan
The PublishFreeeze table is actually unnecessary for this query, since the group ID (inbred_set_id) should be passed in and that ID is in the PublishXRef table (so no neeed to join with PublishFreeze)
2022-02-02response object error fixAlexander Kabui
2022-02-02pep8 formattingAlexander Kabui
2022-02-02remove commentsAlexander Kabui
2022-02-02return 401 on request failAlexander Kabui
2022-02-02refactor code for invoking ctl scriptAlexander Kabui
2022-02-02pep8 formattingAlexander Kabui
2022-02-02match case string input to booleanAlexander Kabui
2022-02-02fix for parametric inputAlexander Kabui
2022-02-02use user input significanceAlexander Kabui
2022-02-02new line fixAlexander Kabui