aboutsummaryrefslogtreecommitdiff
path: root/wqflask/base/data_set.py
AgeCommit message (Collapse)Author
2022-09-20Decompose file into separate modulesrefactor-data_set_py-20220920Frederick Muriuki Muriithi
To ease future refactors on the code, decompose the file into a module with multiple modules that can be refactored semi-independently.
2022-09-12Fix query: Add comma to separate columns.Frederick Muriuki Muriithi
2022-09-12Fix query: remove wrongly-placed commaFrederick Muriuki Muriithi
2022-09-12Check db resultset before indexing into itFrederick Muriuki Muriithi
The assumption that the resultset will never be empty was leading to a lot of errors.
2022-09-09Fix bug in retrieve_other_nameszsloan
After a recent change, it always tries to unpack 6 values from the query, but it should only be unpacking 4 values if not type ProbeSet (so it was throwing an error for anything not ProbeSet)
2022-09-09Fetch sample values correctlyMunyoki Kilyungi
* wqflask/base/data_set.py (MrnaAssayDataSet.retrieve_sample_data): Fix indentation.
2022-09-09Filter out Samples without valuesFrederick Muriuki Muriithi
2022-09-09Replace `g.db.execute` with `cursor.execute`Frederick Muriuki Muriithi
2022-09-08Remove redundant cursor.fetchone()Munyoki Kilyungi
* wqflask/base/data_set.py (DataSet.retrieve_other_names): Call cursor.fetchone() only once.
2022-09-08Fix sql queriesMunyoki Kilyungi
"%s" should only be used outside table names and column names otherwise a string literal will be inserted thereby leading to errors in the sql statements. * wqflask/base/data_set.py (geno_mrna_confidentiality): Use f-strings for table/columns/clause. * wqflask/base/trait.py (retrieve_trait_info): Ditto. * wqflask/wqflask/gsearch.py (GSearch.__init__): Ditto. * wqflask/wqflask/interval_analyst/GeneUtil.py (loadGenes): Ditto. * wqflask/wqflask/snp_browser/snp_browser.py (SnpBrowser.get_browser_results): Ditto.
2022-09-08Add missing "," in cursor.execute()Munyoki Kilyungi
* wqflask/base/data_set.py (DatasetGroup.get_mapping_methods): Fix failing query by adding a missing ",".
2022-09-08Remove intermediate variables that are only used onceMunyoki Kilyungi
* wqflask/base/data_set.py (DatasetGroup.__init__): Remove "query" variable. (DataSet.chunk_dataset): Fix indentation. * wqflask/wqflask/search_results.py (get_GO_symbols): Remove "this_term" variable. * wqflask/wqflask/views.py (search_page): Remove "the_search" variable.
2022-09-08Delete dead importsMunyoki Kilyungi
* wqflask/base/data_set.py: Remove un-used imports. * wqflask/base/trait.py: Ditto. wqflask/tests/wqflask/show_trait/test_show_trait.py: Ditto. * wqflask/wqflask/show_trait/show_trait.py: Ditto. * wqflask/wqflask/views: Ditto.
2022-09-08Replace g.db with database_connection() context managerMunyoki Kilyungi
* wqflask/base/data_set.py (DatasetType.set_dataset_key): Replace g.db with "database_connection()". (DatasetGroup.get_mapping_methods): Ditto. (DataSet.get_accession_id): Ditto. (DataSet.retrieve_other_names): Rework "if ... else" logic to get rid of "query" and "_vars" variables. Also, nest "try ... except" block within "database_connection". (DataSet.chunk_dataset): Replace g.db with "database_connection()". (DataSet.get_probeset_data): Ditto. (DataSet.get_trait_data): Ditto. (PhenotypeDataSet.get_trait_info): Ditto. (PhenotypeDataSet.retrieve_sample_data): Ditto. (GenotypeDataSet.retrieve_sample_data): Ditto. (MrnaAssayDataSet.get_trait_info): Ditto. (MrnaAssayDataSet.retrieve_sample_data): Ditto. (MrnaAssayDataSet.retrieve_genes): Ditto. (TempDataSet): Ditto. (geno_mrna_confidentiality): Ditto. (query_table_timestamp): Ditto. * wqflask/wqflask/api/router.py: Replace "flask.g" with database_connection import. (get_species_list): Replace g.db with "database_connection()". (get_species_info): Ditto. (get_groups_list): Ditto. (get_group_info): Ditto. (get_datasets_for_group): Ditto. (get_dataset_info): Ditto. (fetch_traits): Ditto. (all_sample_data): Ditto. (trait_sample_data): Ditto. (get_trait_info): Ditto. (get_dataset_trait_ids): Ditto. (get_samplelist): Ditto. (get_group_id_from_dataset): Ditto. (get_group_id): Ditto.
2022-09-08Merge pull request #727 from Alexanderlacuna/feature/use-textfilesAlexander Kabui
use text files for Probeset
2022-09-07catch general errors this will recreate the fileAlexander_Kabui
2022-09-06Fix error with dataset menu query in data_set.pyzsloan
I'm only using g.db.execute here because using cursor is throwing an error and I don't want to change the way this query is currently written since I didn't write it (I believe it was initially written using g.db.execute and whoever wrote it just forgot to put the g.db.execute, since previously it just called "fetchall")
2022-09-06Set up the self.* variables from query resultsFrederick Muriuki Muriithi
Set up the appropriate self.* variables from the results of running the appropriate query.
2022-09-06Use dataset name if dataset group name is not set.Frederick Muriuki Muriithi
2022-08-31Delete unused modulesMunyoki Kilyungi
* wqflask/base/data_set.py: Delete "menu_main" import. * wqflask/db/call.py: Delete it. * wqflask/db/gn_server.py: Ditto. * wqflask/wqflask/submit_bnw.py: Ditto.
2022-08-31Replace fetchall, fetchone, fetch1 with database_connectionMunyoki Kilyungi
* wqflask/base/data_set.py: Replace "db.call" import with "database_connection". (create_datasets_list): Use "database_connection" to fetch data. (DatasetGroup.__init__): Ditto. (DataSet.retrieve_other_names): Ditto. (PhenotypeDataSet.setup): Remove query escaping in string and format the string. (GenotypeDataSet.setup): Ditto. (MrnaAssayDataSet.setup): Ditto. * wqflask/db/webqtlDatabaseFunction.py: Remove db.call import. (retrieve_species): Use database_connection() to fetch data. (retrieve_species_id): Ditto.
2022-08-31Remove "with Bench ..." callsMunyoki Kilyungi
"with Bench" instruments how long a function takes and generates time reports on as INFO logs. This should be done on a developer server. Should the log level be low enough, this bench marks will generate a lot of noise. Instrumentation should be done during development. * wqflask/base/data_set.py (create_datasets_list): Remove "with Bench...". * wqflask/db/call.py (fetchone): Ditto. (fetchall): Ditto. (gn_server): Ditto. * wqflask/wqflask/gsearch.py (GSearch.__init__): Ditto. * wqflask/wqflask/marker_regression/display_mapping_results.py (DisplayMappingResults.__init__): Ditto. * wqflask/wqflask/marker_regression/run_mapping.py (RunMapping.__init__): Ditto. * wqflask/wqflask/update_search_results.py (GSearch.__init__): Ditto. * wqflask/wqflask/views.py (search_page): Ditto. (heatmap_page): Ditto. (mapping_results_page): Ditto.
2022-08-31Remove usage of "logger" and un-necessary comments wrt the sameMunyoki Kilyungi
Logging is used to introspect variables or notify the commencement of a given operation. Logging should only be used to log errors. Also, most of the logging is either "logger.debug" or "logger.info"; and this won't show up in production/testing since we need a logging level above "WARNING" for them to show up. * wqflask/base/data_set.py (create_datasets_list): Remove logger. (Markers.add_pvalues): Ditto. (DataSet.retrieve_other_names): Ditto. * wqflask/base/mrna_assay_tissue_data.py: Ditto. * wqflask/base/webqtlCaseData.py: Ditto. * wqflask/db/call.py (fetch1): Ditto. (gn_server): Ditto. * wqflask/db/gn_server.py: Ditto. * wqflask/maintenance/set_resource_defaults.py: Ditto. * wqflask/utility/Plot.py (find_outliers): Ditto. * wqflask/utility/gen_geno_ob.py: Ditto. * wqflask/utility/helper_functions.py: Ditto. * wqflask/utility/pillow_utils.py: Ditto. * wqflask/utility/redis_tools.py: Ditto. * wqflask/wqflask/api/gen_menu.py (get_groups): Ditto. * wqflask/wqflask/api/mapping.py: Ditto. * wqflask/wqflask/api/router.py (get_dataset_info): Ditto. * wqflask/wqflask/collect.py (report_change): Ditto. * wqflask/wqflask/correlation/corr_scatter_plot.py: Ditto. * wqflask/wqflask/ctl/ctl_analysis.py (CTL): Ditto. (CTL.__init__): Ditto. (CTL.run_analysis): Ditto. (CTL.process_results): Ditto. * wqflask/wqflask/db_info.py: Ditto. * wqflask/wqflask/do_search.py (DoSearch.execute): Ditto. (DoSearch.mescape): Ditto. (DoSearch.get_search): Ditto. (MrnaAssaySearch.run_combined): Ditto. (MrnaAssaySearch.run): Ditto. (PhenotypeSearch.run_combined): Ditto. (GenotypeSearch.get_where_clause): Ditto. (LrsSearch.get_where_clause): Ditto. (MeanSearch.run): Ditto. (RangeSearch.get_where_clause): Ditto. (PvalueSearch.run): Ditto. * wqflask/wqflask/docs.py: Ditto. * wqflask/wqflask/export_traits.py: Ditto. * wqflask/wqflask/external_tools/send_to_bnw.py: Ditto. * wqflask/wqflask/external_tools/send_to_geneweaver.py: Ditto. * wqflask/wqflask/external_tools/send_to_webgestalt.py: Ditto. * wqflask/wqflask/gsearch.py (GSearch.__init__): Ditto. * wqflask/wqflask/heatmap/heatmap.py: Ditto. * wqflask/wqflask/marker_regression/display_mapping_results.py (DisplayMappingResults): Ditto. * wqflask/wqflask/marker_regression/gemma_mapping.py: Ditto. * wqflask/wqflask/marker_regression/plink_mapping.py (run_plink): Ditto. * wqflask/wqflask/marker_regression/qtlreaper_mapping.py (run_reaper): Ditto. * wqflask/wqflask/marker_regression/rqtl_mapping.py: Ditto. * wqflask/wqflask/marker_regression/run_mapping.py (RunMapping.__init__): Ditto. * wqflask/wqflask/parser.py (parse): Ditto. * wqflask/wqflask/search_results.py (SearchResultPage.__init__): Ditto. * wqflask/wqflask/update_search_results.py (GSearch.__init__): Ditto. * wqflask/wqflask/user_login.py (send_email): Ditto. (logout): Ditto. (forgot_password_submit): Ditto. (password_reset): Ditto. (password_reset_step2): Ditto. (register): Ditto. * wqflask/wqflask/user_session.py (create_signed_cookie): Ditto.
2022-07-12Include genotypes for F1s/reference/non-reference strains when fetchingzsloan
sample data for genotype traits
2022-04-14Add group code to loading pages when applicablezsloan
This adds the group code to phenotype traits on loading pages, and also sets the group code as an attribute of the dataset.group class.
2022-02-21Fix incorrect dataset trait data cachingzsloan
Trait data caching wasn't working correctly because it didn't account for the samplelist, causing caching to work incorrect in any situation where the target dataset's samplelist wasn't the same as that of the trait being correlated against. Trait data is stored as a dictionary where the keys are trait IDs and values are *lists* of sample values. This means that the caching needs to account for the exact same set of samples; otherwise you'll end up with samples being mismatched (since "the third sample with a value" for one dataset's trait might not be the same as "the third sample with a value" for another dataset's trait). To fix this, I added the samplelist to the functions that generate and fetch the hash file. This will require more cache files, though, so this should probably be reexamined later to make the code work with only a single cache file for each dataset.
2021-11-25Delete noisy loggingBonfaceKilz
2021-11-23resolve merge conflictsAlexander Kabui
2021-11-23pep8 formatting remove dead variablesAlexander Kabui
2021-11-17refactor sql query & date formattingAlexander Kabui
2021-11-17replace redis fetch for cached timestamp with a queryAlexander Kabui
2021-11-17isolate SQL_URI parse to a functionAlexander Kabui
2021-11-17use a dynamic value for the db_nameAlexander Kabui
2021-11-17temp disable query cachingAlexander Kabui
2021-11-16use comprehension list;fix typoAlexander Kabui
2021-11-16pep8 formatting remove debug statementsAlexander Kabui
2021-11-16test generate new filesAlexander Kabui
2021-11-16refactor:fix on the query :modify cache pointAlexander Kabui
2021-11-04check for null timestampAlexander Kabui
2021-11-04store table timestamp on redisAlexander Kabui
2021-11-04init function to get the table timestampAlexander Kabui
2021-11-04pep8 formatting ,minor fixesAlexander Kabui
2021-11-04generate filename hashAlexander Kabui
2021-11-04fetch cached resultsAlexander Kabui
2021-11-04function to cache sample resultsAlexander Kabui
2021-08-17Merge pull request #598 from zsloan/feature/add_filter_by_study_sampleszsloan
Feature/add filter by study samples
2021-08-13base: data_set: Remove unnecessary comments and logging statementsBonfaceKilz
2021-08-12Return empty list instead of None in get_study_samplelistszsloan
2021-08-12Fix the way the study_sample_lists path is set and checkedzsloan
2021-08-12add function for reading in JSON file that lists sample lists unique to each ↵zsloan
study within a group (in this case only BXD Longevity for now)
generated by cgit v1.2.3 (git 2.44.0) at 2024-10-21 01:12:50 +0000