diff options
193 files changed, 11590 insertions, 2091 deletions
diff --git a/docs/dev/background_jobs.md b/docs/dev/background_jobs.md new file mode 100644 index 0000000..1a41636 --- /dev/null +++ b/docs/dev/background_jobs.md @@ -0,0 +1,62 @@ +# Background Jobs + +We run background jobs for long-running processes, e.g. quality-assurance checks +across multiple huge files, inserting huge data to databases, etc. The system +needs to keep track of the progress of these jobs and communicate the state to +the user whenever the user requests. + +This details some thoughts on how to handle these jobs, especially in failure +conditions. + +We currently use Redis[^redis] to keep track of the state of the background +processes. + +Every background job started will have a Redis[^redis] key with the prefix `gn-uploader:jobs` + +## Users + +Currently (2024-10-23T13:29UTC-05:00), we do not track the user that started the job. Moving forward, we will track this information. + +We could have the keys be something like, `gn-uploader:jobs:<user-id>:<job-id>`. + +Another option is track any particular users jobs with a key of the form +`gn-uploader:users:<user-id>:jobs` and in that case, have the job keys take the +form `gn-uploader:jobs:<job-id>`. I (@fredmanglis) favour this option over +having the user's ID in the jobs keys directly, since it provides a way to +interact with **ALL** the jobs without indirecting through each specific user. +This is a useful ability to have, especially for system administrative tasks. + +## Multiprocessing Within Jobs + +Some jobs, e.g. quality-assurance jobs, can run multiple threads/processes +themselves. This brings up a problem because Redis[^redis] does not allow +parallel access to a key, especially for writing. + +We also do not want to create bottlenecks by writing to the same key from +multiple threads/processes. + +The design I have currently come up with, that might work is as follows: + +- At any point just before where multiple threads/processes are started, a list + of new keys, each of which will collect the output from a single thread, will + be built. +- These keys are recorded in the parent's redis key data +- The threads/processes are started and do whatever they need, pushing their + outputs to the appropriate keys within redis. + +The new keys for the children threads/processe could build on the theme + + +## Fetching Jobs Status + +Different jobs could have different ways of requirements for handling/processing +their outputs, and those of any children they might spawn. The system will need +to provide a way to pass in the correct function/code to process the outputs at +the point where the job status is requested. + +This implies that we need to track the type of job in order to be able to select +the correct code for processing such output. + +## Links + +- [^redis]: https://redis.io/ diff --git a/docs/dev/quality_assurance_on_csv_files.md b/docs/dev/quality_assurance_on_csv_files.md new file mode 100644 index 0000000..02d63c9 --- /dev/null +++ b/docs/dev/quality_assurance_on_csv_files.md @@ -0,0 +1,52 @@ +# Quality Assurance/Control on CSV Files + +## Abbreviations + +- CSV files: Character-separated-values files — these are data files structured in a table-like format, with a specific character chosen as the column/field separator. The comma (,) is the most common field separator used by most csv files. It is, however, possible to encounter files with other characters separating the values. + +## General Pattern + +A general pattern has emerged when performing quality assurance on the data in +CSV files — the pseudocode below shows the general pattern: + +```python +def qc_function(filepath, …): + open(filepath, …) + + headers = read_first_line(…) + perform_qc_on_headings(headers, …) + + for each subsequent line in file: + perform_qc_on_first_column(line, …) + + for each subsequent field in line: + perform_qc_on_field(field, …) +``` + +We want to list the errors found in each file, so it makes sense for the `perform_qc_on*` functions in the pseudocode above to return the list of errors found for each file. + +The actual quality assurance done on the headers, first column of data rows, and the fields can differ from one type of file to the next, but the structure remains relatively unchanged. + +This implies we could make use of a higher-order function that contains the general structure with the actual qc steps passed in as functions that are called in the higher-order structuring function. This gives something like: + +```python +def qc_function(filepath, headers_qc, first_column_qc, data_qc, …): + for line in file: + if line is a comment line: + skip line and continue iteration + if line is first non-comment line: + line is the header line + call headers_qc on fields in this line + if line is not first non-comment line: + line is data line + call first_column_qc on first field of line + call data_qc on each of the subsequent fields of the line + + collect and return errors +``` + +## Improvements + +- Read the file in a separate generator function +- Parallelize QC if many files are present +- Add logging/output for user update (how do we do this correctly?) @@ -40,4 +40,7 @@ ignore_missing_imports = True ignore_missing_imports = True [mypy-flask_session.*] +ignore_missing_imports = True + +[mypy-gn_libs.*] ignore_missing_imports = True
\ No newline at end of file diff --git a/quality_control/checks.py b/quality_control/checks.py index bdfd12b..bb05e31 100644 --- a/quality_control/checks.py +++ b/quality_control/checks.py @@ -52,12 +52,15 @@ def decimal_places_pattern(mini: int, maxi: Optional[int] = None) -> re.Pattern: + r")$" ) -def decimal_points_error(filename: str,# pylint: disable=[too-many-arguments] - lineno: int, - field: str, - value: str, - mini: int, - maxi: Optional[int] = None) -> Optional[InvalidValue]: +def decimal_points_error( + # pylint: disable=[too-many-arguments, too-many-positional-arguments] + filename: str, + lineno: int, + field: str, + value: str, + mini: int, + maxi: Optional[int] = None +) -> Optional[InvalidValue]: """ Check that 'value' in a decimal number with the appropriate decimal places. """ diff --git a/quality_control/parsing.py b/quality_control/parsing.py index f1d21fc..7a8185d 100644 --- a/quality_control/parsing.py +++ b/quality_control/parsing.py @@ -104,23 +104,22 @@ def collect_errors( if line_number == 1: consistent_columns_checker = make_column_consistency_checker( filename, line) - for error in __process_errors__( - filename, line_number, line, - partial(header_errors, strains=strains), - errors): - yield error + yield from __process_errors__( + filename, line_number, line, + partial(header_errors, strains=strains), + errors) if line_number != 1: - col_consistency_error = consistent_columns_checker(line_number, line) + col_consistency_error = consistent_columns_checker(# pylint: disable=[possibly-used-before-assignment] + line_number, line) if col_consistency_error: yield col_consistency_error - for error in __process_errors__( + yield from __process_errors__( filename, line_number, line, ( average_errors if filetype == FileType.AVERAGE else se_errors), - errors): - yield error + errors) if update_progress: update_progress(line_number, line) diff --git a/r_qtl/r_qtl2.py b/r_qtl/r_qtl2.py index 9da4081..06175ce 100644 --- a/r_qtl/r_qtl2.py +++ b/r_qtl/r_qtl2.py @@ -16,7 +16,11 @@ from r_qtl.exceptions import InvalidFormat, MissingFileException FILE_TYPES = ( "geno", "founder_geno", "pheno", "covar", "phenocovar", "gmap", "pmap", - "phenose") + "phenose", "phenonum") + +__CONTROL_FILE_ERROR_MESSAGE__ = ( + "The zipped bundle that was provided does not contain a valid control file " + "in either JSON or YAML format.") def __special_file__(filename): @@ -72,6 +76,8 @@ def transpose_csv( def __read_by_line__(_path): with open(_path, "r", encoding="utf8") as infile: for line in infile: + if line.startswith("#"): + continue yield line transposed_data= (f"{linejoinerfn(items)}\n" for items in zip(*( @@ -112,7 +118,7 @@ def __control_data_from_zipfile__(zfile: ZipFile) -> dict: or filename.endswith(".json")))) num_files = len(files) if num_files == 0: - raise InvalidFormat("Expected a json or yaml control file.") + raise InvalidFormat(__CONTROL_FILE_ERROR_MESSAGE__) if num_files > 1: raise InvalidFormat("Found more than one possible control file.") @@ -129,7 +135,6 @@ def __control_data_from_zipfile__(zfile: ZipFile) -> dict: else yaml.safe_load(zfile.read(files[0]))) } - def __control_data_from_dirpath__(dirpath: Path): """Load control data from a given directory path.""" files = tuple(path for path in dirpath.iterdir() @@ -137,7 +142,7 @@ def __control_data_from_dirpath__(dirpath: Path): and (path.suffix in (".yaml", ".json")))) num_files = len(files) if num_files == 0: - raise InvalidFormat("Expected a json or yaml control file.") + raise InvalidFormat(__CONTROL_FILE_ERROR_MESSAGE__) if num_files > 1: raise InvalidFormat("Found more than one possible control file.") @@ -182,7 +187,7 @@ def control_data(control_src: Union[Path, ZipFile]) -> dict: r_qtl.exceptions.InvalidFormat """ def __cleanup__(cdata): - return { + _cdata = { **cdata, **dict((filetype, ([cdata[filetype]] if isinstance(cdata[filetype], str) @@ -190,6 +195,14 @@ def control_data(control_src: Union[Path, ZipFile]) -> dict: ) for filetype in (typ for typ in cdata.keys() if typ in FILE_TYPES)) } + if "na.string" in _cdata:# handle common error in file. + _cdata = { + **cdata, + "na.strings": list(set( + _cdata["na.string"] + _cdata["na.strings"])) + } + + return _cdata if isinstance(control_src, ZipFile): return __cleanup__(__control_data_from_zipfile__(control_src)) @@ -200,8 +213,8 @@ def control_data(control_src: Union[Path, ZipFile]) -> dict: if control_src.is_dir(): return __cleanup__(__control_data_from_dirpath__(control_src)) raise InvalidFormat( - "Expects either a zipfile.ZipFile object or a pathlib.Path object " - "pointing to a directory containing the R/qtl2 bundle.") + "Expects either a zipped bundle of files or a path-like object " + "pointing to the zipped R/qtl2 bundle.") def replace_na_strings(cdata, val): @@ -549,3 +562,43 @@ def load_samples(zipfilepath: Union[str, Path], pass return tuple(samples) + + + +def read_text_file(filepath: Union[str, Path]) -> Iterator[str]: + """Read the raw text from a text file.""" + with open(filepath, "r", encoding="utf8") as _file: + for line in _file: + yield line + + +def read_csv_file(filepath: Union[str, Path], + separator: str = ",", + comment_char: str = "#") -> Iterator[tuple[str, ...]]: + """Read a file as a csv file. This does not process the N/A values.""" + for line in read_text_file(filepath): + if line.startswith(comment_char): + continue + yield tuple(field.strip() for field in line.split(separator)) + + +def read_csv_file_headers( + filepath: Union[str, Path], + transposed: bool, + separator: str = ",", + comment_char: str = "#" +) -> tuple[str, ...]: + """Read the 'true' headers of a CSV file.""" + headers = tuple() + for line in read_text_file(filepath): + if line.startswith(comment_char): + continue + + line = tuple(field.strip() for field in line.split(separator)) + if not transposed: + return line + + headers = headers + (line[0],) + continue + + return headers diff --git a/r_qtl/r_qtl2_qc.py b/r_qtl/r_qtl2_qc.py index 7b26b50..2d9e9a8 100644 --- a/r_qtl/r_qtl2_qc.py +++ b/r_qtl/r_qtl2_qc.py @@ -95,7 +95,7 @@ def missing_files(bundlesrc: Union[Path, ZipFile]) -> tuple[tuple[str, str], ... "pointing to a directory containing the R/qtl2 bundle.") -def validate_bundle(zfile: ZipFile): +def validate_bundle(zfile: Union[Path, ZipFile]): """Ensure the R/qtl2 bundle is valid.""" missing = missing_files(zfile) if len(missing) > 0: diff --git a/scripts/cli_parser.py b/scripts/cli_parser.py index 308ee4b..0c91c5e 100644 --- a/scripts/cli_parser.py +++ b/scripts/cli_parser.py @@ -19,6 +19,13 @@ def init_cli_parser(program: str, description: Optional[str] = None) -> Argument type=int, default=86400, help="How long to keep any redis keys around.") + parser.add_argument( + "--loglevel", + type=str, + default="INFO", + choices=["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL", + "debug", "info", "warning", "error", "critical"], + help="The severity of events to track with the logger.") return parser def add_global_data_arguments(parser: ArgumentParser) -> ArgumentParser: diff --git a/scripts/insert_data.py b/scripts/insert_data.py index 4b2e5f3..67038f8 100644 --- a/scripts/insert_data.py +++ b/scripts/insert_data.py @@ -10,11 +10,11 @@ from typing import Tuple, Iterator import MySQLdb as mdb from redis import Redis from MySQLdb.cursors import DictCursor +from gn_libs.mysqldb import database_connection from functional_tools import take from quality_control.file_utils import open_file -from uploader.db_utils import database_connection from uploader.check_connections import check_db, check_redis # Set up logging diff --git a/scripts/insert_samples.py b/scripts/insert_samples.py index e3577b6..742c4ae 100644 --- a/scripts/insert_samples.py +++ b/scripts/insert_samples.py @@ -3,11 +3,12 @@ import sys import logging import pathlib import argparse +import traceback import MySQLdb as mdb from redis import Redis +from gn_libs.mysqldb import database_connection -from uploader.db_utils import database_connection from uploader.check_connections import check_db, check_redis from uploader.species.models import species_by_id from uploader.population.models import population_by_id @@ -73,6 +74,7 @@ def insert_samples(conn: mdb.Connection,# pylint: disable=[too-many-arguments] print("Samples upload successfully completed.") return 0 + if __name__ == "__main__": def cli_args(): @@ -127,7 +129,7 @@ if __name__ == "__main__": def main(): """Run script to insert samples into the database.""" - + status_code = 1 # Exit with an Exception args = cli_args() check_db(args.databaseuri) check_redis(args.redisuri) @@ -137,13 +139,19 @@ if __name__ == "__main__": with (Redis.from_url(args.redisuri, decode_responses=True) as rconn, database_connection(args.databaseuri) as dbconn): - return insert_samples(dbconn, - rconn, - args.speciesid, - args.populationid, - args.samplesfile, - args.separator, - args.firstlineheading, - args.quotechar) + + try: + status_code = insert_samples(dbconn, + rconn, + args.speciesid, + args.populationid, + args.samplesfile, + args.separator, + args.firstlineheading, + args.quotechar) + except Exception as _exc: + print(traceback.format_exc(), file=sys.stderr) + + return status_code sys.exit(main()) diff --git a/scripts/load_phenotypes_to_db.py b/scripts/load_phenotypes_to_db.py new file mode 100644 index 0000000..5ce37f3 --- /dev/null +++ b/scripts/load_phenotypes_to_db.py @@ -0,0 +1,518 @@ +import sys +import uuid +import json +import logging +import argparse +import datetime +from pathlib import Path +from zipfile import ZipFile +from typing import Any, Union +from urllib.parse import urljoin +from functools import reduce, partial + +from MySQLdb.cursors import Cursor, DictCursor + +from gn_libs import jobs, mysqldb, sqlite3, monadic_requests as mrequests + +from r_qtl import r_qtl2 as rqtl2 +from uploader.species.models import species_by_id +from uploader.population.models import population_by_species_and_id +from uploader.samples.models import samples_by_species_and_population +from uploader.phenotypes.models import ( + dataset_by_id, + save_phenotypes_data, + create_new_phenotypes, + quick_save_phenotypes_data) +from uploader.publications.models import ( + create_new_publications, + fetch_publication_by_id) + +from scripts.rqtl2.bundleutils import build_line_joiner, build_line_splitter + +logging.basicConfig( + format="%(asctime)s — %(filename)s:%(lineno)s — %(levelname)s: %(message)s") +logger = logging.getLogger(__name__) + + + +def __replace_na_strings__(line, na_strings): + return ((None if value in na_strings else value) for value in line) + + +def save_phenotypes( + cursor: mysqldb.Connection, + control_data: dict[str, Any], + filesdir: Path +) -> tuple[dict, ...]: + """Read `phenofiles` and save the phenotypes therein.""" + ## TODO: Replace with something like this: ## + # phenofiles = control_data["phenocovar"] + control_data.get( + # "gn-metadata", {}).get("pheno", []) + # + # This is meant to load (and merge) data from the "phenocovar" and + # "gn-metadata -> pheno" files into a single collection of phenotypes. + phenofiles = tuple(filesdir.joinpath(_file) for _file in control_data["phenocovar"]) + if len(phenofiles) <= 0: + return tuple() + + if control_data["phenocovar_transposed"]: + logger.info("Undoing transposition of the files rows and columns.") + phenofiles = ( + rqtl2.transpose_csv_with_rename( + _file, + build_line_splitter(control_data), + build_line_joiner(control_data)) + for _file in phenofiles) + + _headers = rqtl2.read_csv_file_headers(phenofiles[0], + control_data["phenocovar_transposed"], + control_data["sep"], + control_data["comment.char"]) + return create_new_phenotypes( + cursor, + (dict(zip(_headers, + __replace_na_strings__(line, control_data["na.strings"]))) + for filecontent + in (rqtl2.read_csv_file(path, + separator=control_data["sep"], + comment_char=control_data["comment.char"]) + for path in phenofiles) + for idx, line in enumerate(filecontent) + if idx != 0)) + + +def __fetch_next_dataid__(conn: mysqldb.Connection) -> int: + """Fetch the next available DataId value from the database.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT MAX(DataId) AS CurrentMaxDataId FROM PublishXRef") + return int(cursor.fetchone()["CurrentMaxDataId"]) + 1 + + +def __row_to_dataitems__( + sample_row: dict, + dataidmap: dict, + pheno_name2id: dict[str, int], + samples: dict +) -> tuple[dict, ...]: + samplename = sample_row["id"] + + return ({ + "phenotype_id": dataidmap[pheno_name2id[phenoname]]["phenotype_id"], + "data_id": dataidmap[pheno_name2id[phenoname]]["data_id"], + "sample_name": samplename, + "sample_id": samples[samplename]["Id"], + "value": phenovalue + } for phenoname, phenovalue in sample_row.items() if phenoname != "id") + + +def __build_dataitems__( + filetype, + phenofiles, + control_data, + samples, + dataidmap, + pheno_name2id +): + _headers = rqtl2.read_csv_file_headers( + phenofiles[0], + False, # Any transposed files have been un-transposed by this point + control_data["sep"], + control_data["comment.char"]) + _filescontents = ( + rqtl2.read_csv_file(path, + separator=control_data["sep"], + comment_char=control_data["comment.char"]) + for path in phenofiles) + _linescontents = ( + __row_to_dataitems__( + dict(zip(("id",) + _headers[1:], + __replace_na_strings__(line, control_data["na.strings"]))), + dataidmap, + pheno_name2id, + samples) + for linenum, line in (enumline for filecontent in _filescontents + for enumline in enumerate(filecontent)) + if linenum > 0) + return (item for items in _linescontents + for item in items + if item["value"] is not None) + + +def save_numeric_data( + conn: mysqldb.Connection, + dataidmap: dict, + pheno_name2id: dict[str, int], + samples: tuple[dict, ...], + control_data: dict, + filesdir: Path, + filetype: str, + table: str +): + """Read data from files and save to the database.""" + phenofiles = tuple( + filesdir.joinpath(_file) for _file in control_data[filetype]) + if len(phenofiles) <= 0: + return tuple() + + if control_data[f"{filetype}_transposed"]: + logger.info("Undoing transposition of the files rows and columns.") + phenofiles = tuple( + rqtl2.transpose_csv_with_rename( + _file, + build_line_splitter(control_data), + build_line_joiner(control_data)) + for _file in phenofiles) + + try: + logger.debug("Attempt quick save with `LOAD … INFILE`.") + return quick_save_phenotypes_data( + conn, + table, + __build_dataitems__( + filetype, + phenofiles, + control_data, + samples, + dataidmap, + pheno_name2id), + filesdir) + except Exception as _exc: + logger.debug("Could not use `LOAD … INFILE`, using raw query", + exc_info=True) + import time;time.sleep(60) + return save_phenotypes_data( + conn, + table, + __build_dataitems__( + filetype, + phenofiles, + control_data, + samples, + dataidmap, + pheno_name2id)) + + +save_pheno_data = partial(save_numeric_data, + filetype="pheno", + table="PublishData") + + +save_phenotypes_se = partial(save_numeric_data, + filetype="phenose", + table="PublishSE") + + +save_phenotypes_n = partial(save_numeric_data, + filetype="phenonum", + table="NStrain") + + +def cross_reference_phenotypes_publications_and_data( + conn: mysqldb.Connection, xref_data: tuple[dict, ...] +): + """Crossreference the phenotypes, publication and data.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute("SELECT MAX(Id) CurrentMaxId FROM PublishXRef") + _nextid = int(cursor.fetchone()["CurrentMaxId"]) + 1 + _params = tuple({**row, "xref_id": _id} + for _id, row in enumerate(xref_data, start=_nextid)) + cursor.executemany( + ("INSERT INTO PublishXRef(" + "Id, InbredSetId, PhenotypeId, PublicationId, DataId, comments" + ") " + "VALUES (" + "%(xref_id)s, %(population_id)s, %(phenotype_id)s, " + "%(publication_id)s, %(data_id)s, 'Upload of new data.'" + ")"), + _params) + return _params + return tuple() + + +def update_auth(authserver, token, species, population, dataset, xrefdata): + """Grant the user access to their data.""" + # TODO Call into the auth server to: + # 1. Link the phenotypes with a user group + # - fetch group: http://localhost:8081/auth/user/group + # - link data to group: http://localhost:8081/auth/data/link/phenotype + # - *might need code update in gn-auth: remove restriction, perhaps* + # 2. Create resource (perhaps?) + # - Get resource categories: http://localhost:8081/auth/resource/categories + # - Create a new resource: http://localhost:80host:8081/auth/resource/create + # - single resource for all phenotypes + # - resource name from user, species, population, dataset, datetime? + # - User will have "ownership" of resource by default + # 3. Link data to the resource: http://localhost:8081/auth/resource/data/link + # - Update code to allow linking multiple items in a single request + _tries = 0 # TODO use this to limit how many tries before quiting and bailing + _delay = 1 + headers = { + "Authorization": f"Bearer {token}", + "Content-Type": "application/json" + } + def authserveruri(endpoint): + return urljoin(authserver, endpoint) + + def __fetch_user_details__(): + logger.debug("… Fetching user details") + return mrequests.get( + authserveruri("/auth/user/"), + headers=headers + ) + + def __link_data__(user): + logger.debug("… linking uploaded data to user's group") + return mrequests.post( + authserveruri("/auth/data/link/phenotype"), + headers=headers, + json={ + "species_name": species["Name"], + "group_id": user["group"]["group_id"], + "selected": [ + { + "SpeciesId": species["SpeciesId"], + "InbredSetId": population["Id"], + "PublishFreezeId": dataset["Id"], + "dataset_name": dataset["Name"], + "dataset_fullname": dataset["FullName"], + "dataset_shortname": dataset["ShortName"], + "PublishXRefId": item["xref_id"] + } + for item in xrefdata + ], + "using-raw-ids": "on" + }).then(lambda ld_results: (user, ld_results)) + + def __fetch_phenotype_category_details__(user, linkeddata): + logger.debug("… fetching phenotype category details") + return mrequests.get( + authserveruri("/auth/resource/categories"), + headers=headers + ).then( + lambda categories: ( + user, + linkeddata, + next(category for category in categories + if category["resource_category_key"] == "phenotype")) + ) + + def __create_resource__(user, linkeddata, category): + logger.debug("… creating authorisation resource object") + now = datetime.datetime.now().isoformat() + return mrequests.post( + authserveruri("/auth/resource/create"), + headers=headers, + json={ + "resource_category": category["resource_category_id"], + "resource_name": (f"{user['email']}—{dataset['Name']}—{now}—" + f"{len(xrefdata)} phenotypes"), + "public": "off" + }).then(lambda cr_results: (user, linkeddata, cr_results)) + + def __attach_data_to_resource__(user, linkeddata, resource): + logger.debug("… attaching data to authorisation resource object") + return mrequests.post( + authserveruri("/auth/resource/data/link"), + headers=headers, + json={ + "dataset_type": "phenotype", + "resource_id": resource["resource_id"], + "data_link_ids": [ + item["data_link_id"] for item in linkeddata["traits"]] + }).then(lambda attc: (user, linkeddata, resource, attc)) + + def __handle_error__(resp): + logger.error("ERROR: Updating the authorisation for the data failed.") + logger.debug( + "ERROR: The response from the authorisation server was:\n\t%s", + resp.json()) + return 1 + + def __handle_success__(val): + logger.info( + "The authorisation for the data has been updated successfully.") + return 0 + + return __fetch_user_details__().then(__link_data__).then( + lambda result: __fetch_phenotype_category_details__(*result) + ).then( + lambda result: __create_resource__(*result) + ).then( + lambda result: __attach_data_to_resource__(*result) + ).either(__handle_error__, __handle_success__) + + +def load_data(conn: mysqldb.Connection, job: dict) -> int: + """Load the data attached in the given job.""" + _job_metadata = job["metadata"] + # Steps + # 0. Read data from the files: can be multiple files per type + # + _species = species_by_id(conn, int(_job_metadata["species_id"])) + _population = population_by_species_and_id( + conn, + _species["SpeciesId"], + int(_job_metadata["population_id"])) + _dataset = dataset_by_id( + conn, + _species["SpeciesId"], + _population["Id"], + int(_job_metadata["dataset_id"])) + # 1. Just retrive the publication: Don't create publications for now. + _publication = fetch_publication_by_id( + conn, int(_job_metadata.get("publication_id", "0"))) or {"Id": 0} + # 2. Save all new phenotypes: + # -> return phenotype IDs + bundle = Path(_job_metadata["bundle_file"]) + _control_data = rqtl2.control_data(bundle) + logger.info("Extracting the zipped bundle of files.") + _outdir = Path(bundle.parent, f"bundle_{bundle.stem}") + with ZipFile(str(bundle), "r") as zfile: + _files = rqtl2.extract(zfile, _outdir) + logger.info("Saving new phenotypes.") + _phenos = save_phenotypes(conn, _control_data, _outdir) + def __build_phenos_maps__(accumulator, current): + dataid, row = current + return ({ + **accumulator[0], + row["phenotype_id"]: { + "population_id": _population["Id"], + "phenotype_id": row["phenotype_id"], + "data_id": dataid, + "publication_id": _publication["Id"], + } + }, { + **accumulator[1], + row["id"]: row["phenotype_id"] + }) + dataidmap, pheno_name2id = reduce( + __build_phenos_maps__, + enumerate(_phenos, start=__fetch_next_dataid__(conn)), + ({},{})) + # 3. a. Fetch the strain names and IDS: create name->ID map + samples = { + row["Name"]: row + for row in samples_by_species_and_population( + conn, _species["SpeciesId"], _population["Id"])} + # b. Save all the data items (DataIds are vibes), return new IDs + logger.info("Saving new phenotypes data.") + _num_data_rows = save_pheno_data(conn=conn, + dataidmap=dataidmap, + pheno_name2id=pheno_name2id, + samples=samples, + control_data=_control_data, + filesdir=_outdir) + logger.info("Saved %s new phenotype data rows.", _num_data_rows) + # 4. Cross-reference Phenotype, Publication, and PublishData in PublishXRef + logger.info("Cross-referencing new phenotypes to their data and publications.") + _xrefs = cross_reference_phenotypes_publications_and_data( + conn, tuple(dataidmap.values())) + # 5. If standard errors and N exist, save them too + # (use IDs returned in `3. b.` above). + logger.info("Saving new phenotypes standard errors.") + _num_se_rows = save_phenotypes_se(conn=conn, + dataidmap=dataidmap, + pheno_name2id=pheno_name2id, + samples=samples, + control_data=_control_data, + filesdir=_outdir) + logger.info("Saved %s new phenotype standard error rows.", _num_se_rows) + + logger.info("Saving new phenotypes sample counts.") + _num_n_rows = save_phenotypes_n(conn=conn, + dataidmap=dataidmap, + pheno_name2id=pheno_name2id, + samples=samples, + control_data=_control_data, + filesdir=_outdir) + logger.info("Saved %s new phenotype sample counts rows.", _num_n_rows) + return (_species, _population, _dataset, _xrefs) + + +if __name__ == "__main__": + def parse_args(): + """Setup command-line arguments.""" + parser = argparse.ArgumentParser( + prog="load_phenotypes_to_db", + description="Process the phenotypes' data and load it into the database.") + parser.add_argument("db_uri", type=str, help="MariaDB/MySQL connection URL") + parser.add_argument( + "jobs_db_path", type=Path, help="Path to jobs' SQLite database.") + parser.add_argument("job_id", type=uuid.UUID, help="ID of the running job") + parser.add_argument( + "--log-level", + type=str, + help="Determines what is logged out.", + choices=("debug", "info", "warning", "error", "critical"), + default="info") + return parser.parse_args() + + def setup_logging(log_level: str): + """Setup logging for the script.""" + logger.setLevel(log_level) + logging.getLogger("uploader.phenotypes.models").setLevel(log_level) + + + def main(): + """Entry-point for this script.""" + args = parse_args() + setup_logging(args.log_level.upper()) + + with (mysqldb.database_connection(args.db_uri) as conn, + conn.cursor(cursorclass=DictCursor) as cursor, + sqlite3.connection(args.jobs_db_path) as jobs_conn): + job = jobs.job(jobs_conn, args.job_id) + + # Lock the PublishXRef/PublishData/PublishSE/NStrain here: Why? + # The `DataId` values are sequential, but not auto-increment + # Can't convert `PublishXRef`.`DataId` to AUTO_INCREMENT. + # `SELECT MAX(DataId) FROM PublishXRef;` + # How do you check for a table lock? + # https://oracle-base.com/articles/mysql/mysql-identify-locked-tables + # `SHOW OPEN TABLES LIKE 'Publish%';` + _db_tables_ = ( + "Species", + "InbredSet", + "Strain", + "StrainXRef", + "Publication", + "Phenotype", + "PublishXRef", + "PublishFreeze", + "PublishData", + "PublishSE", + "NStrain") + + logger.debug( + ("Locking database tables for the connection:" + + "".join("\n\t- %s" for _ in _db_tables_) + "\n"), + *_db_tables_) + cursor.execute(# Lock the tables to avoid race conditions + "LOCK TABLES " + ", ".join( + f"{_table} WRITE" for _table in _db_tables_)) + + db_results = load_data(conn, job) + jobs.update_metadata( + jobs_conn, + args.job_id, + "xref_ids", + json.dumps([xref["xref_id"] for xref in db_results[3]])) + + logger.info("Unlocking all database tables.") + cursor.execute("UNLOCK TABLES") + + # Update authorisations (break this down) — maybe loop until it works? + logger.info("Updating authorisation.") + _job_metadata = job["metadata"] + return update_auth(_job_metadata["authserver"], + _job_metadata["token"], + *db_results) + + + try: + sys.exit(main()) + except Exception as _exc: + logger.debug("Data loading failed… Halting!", + exc_info=True) + sys.exit(1) diff --git a/scripts/phenotypes_bulk_edit.py b/scripts/phenotypes_bulk_edit.py new file mode 100644 index 0000000..cee5f4e --- /dev/null +++ b/scripts/phenotypes_bulk_edit.py @@ -0,0 +1,266 @@ +import sys +import uuid +import logging +import argparse +from pathlib import Path +from typing import Iterator +from functools import reduce + +from MySQLdb.cursors import DictCursor + +from gn_libs import jobs, mysqldb, sqlite3 + +from uploader.phenotypes.models import phenotypes_data_by_ids +from uploader.phenotypes.misc import phenotypes_data_differences +from uploader.phenotypes.views import BULK_EDIT_COMMON_FIELDNAMES + +import uploader.publications.pubmed as pmed +from uploader.publications.misc import publications_differences +from uploader.publications.models import ( + update_publications, fetch_phenotype_publications) + +logging.basicConfig( + format="%(asctime)s — %(filename)s:%(lineno)s — %(levelname)s: %(message)s") +logger = logging.getLogger(__name__) + + +def check_ids(conn, ids: tuple[tuple[int, int], ...]) -> bool: + """Verify that all the `UniqueIdentifier` values are valid.""" + logger.info("Checking the 'UniqueIdentifier' values.") + with conn.cursor(cursorclass=DictCursor) as cursor: + paramstr = ",".join(["(%s, %s)"] * len(ids)) + cursor.execute( + "SELECT PhenotypeId AS phenotype_id, Id AS xref_id " + "FROM PublishXRef " + f"WHERE (PhenotypeId, Id) IN ({paramstr})", + tuple(item for row in ids for item in row)) + mysqldb.debug_query(cursor, logger) + found = tuple((row["phenotype_id"], row["xref_id"]) + for row in cursor.fetchall()) + + not_found = tuple(item for item in ids if item not in found) + if len(not_found) == 0: + logger.info("All 'UniqueIdentifier' are valid.") + return True + + for item in not_found: + logger.error(f"Invalid 'UniqueIdentifier' value: phId:%s::xrId:%s", item[0], item[1]) + + return False + + +def check_for_mandatory_fields(): + """Verify that mandatory fields have values.""" + pass + + +def __fetch_phenotypes__(conn, ids: tuple[int, ...]) -> tuple[dict, ...]: + """Fetch basic (non-numeric) phenotypes data from the database.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + paramstr = ",".join(["%s"] * len(ids)) + cursor.execute(f"SELECT * FROM Phenotype WHERE Id IN ({paramstr}) " + "ORDER BY Id ASC", + ids) + return tuple(dict(row) for row in cursor.fetchall()) + + +def descriptions_differences(file_data, db_data) -> dict[str, str]: + """Compute differences in the descriptions.""" + logger.info("Computing differences in phenotype descriptions.") + assert len(file_data) == len(db_data), "The counts of phenotypes differ!" + description_columns = ("Pre_publication_description", + "Post_publication_description", + "Original_description", + "Pre_publication_abbreviation", + "Post_publication_abbreviation") + diff = tuple() + for file_row, db_row in zip(file_data, db_data): + assert file_row["phenotype_id"] == db_row["Id"] + inner_diff = { + key: file_row[key] + for key in description_columns + if not file_row[key] == db_row[key] + } + if bool(inner_diff): + diff = diff + ({ + "phenotype_id": file_row["phenotype_id"], + **inner_diff + },) + + return diff + + +def update_descriptions(): + """Update descriptions in the database""" + logger.info("Updating descriptions") + # Compute differences between db data and uploaded file + # Only run query for changed descriptions + pass + + +def link_publications(): + """Link phenotypes to relevant publications.""" + logger.info("Linking phenotypes to publications.") + # Create publication if PubMed_ID doesn't exist in db + pass + + +def update_values(): + """Update the phenotype values.""" + logger.info("Updating phenotypes values.") + # Compute differences between db data and uploaded file + # Only run query for changed data + pass + + +def parse_args(): + parser = argparse.ArgumentParser( + prog="Phenotypes Bulk-Edit Processor", + description="Process the bulk-edits to phenotype data and descriptions.") + parser.add_argument("db_uri", type=str, help="MariaDB/MySQL connection URL") + parser.add_argument( + "jobs_db_path", type=Path, help="Path to jobs' SQLite database.") + parser.add_argument("job_id", type=uuid.UUID, help="ID of the running job") + parser.add_argument( + "--log-level", + type=str, + help="Determines what is logged out.", + choices=("debug", "info", "warning", "error", "critical"), + default="info") + return parser.parse_args() + + +def read_file(filepath: Path) -> Iterator[str]: + """Read the file, one line at a time.""" + with filepath.open(mode="r", encoding="utf-8") as infile: + count = 0 + headers = None + for line in infile: + if line.startswith("#"): # ignore comments + continue; + + fields = line.strip().split("\t") + if count == 0: + headers = fields + count = count + 1 + continue + + _dict = dict(zip( + headers, + ((None if item.strip() == "" else item.strip()) + for item in fields))) + _pheno, _xref = _dict.pop("UniqueIdentifier").split("::") + _dict = { + key: ((float(val) if bool(val) else val) + if key not in BULK_EDIT_COMMON_FIELDNAMES + else val) + for key, val in _dict.items() + } + _dict["phenotype_id"] = int(_pheno.split(":")[1]) + _dict["xref_id"] = int(_xref.split(":")[1]) + if _dict["PubMed_ID"] is not None: + _dict["PubMed_ID"] = int(_dict["PubMed_ID"]) + + yield _dict + count = count + 1 + + +def run(conn, job): + """Process the data and update it.""" + file_contents = tuple(sorted(read_file(Path(job["metadata"]["edit-file"])), + key=lambda item: item["phenotype_id"])) + pheno_ids, pheno_xref_ids, pubmed_ids = reduce( + lambda coll, curr: ( + coll[0] + (curr["phenotype_id"],), + coll[1] + ((curr["phenotype_id"], curr["xref_id"]),), + coll[2].union(set([curr["PubMed_ID"]]))), + file_contents, + (tuple(), tuple(), set([None]))) + check_ids(conn, pheno_xref_ids) + check_for_mandatory_fields() + # stop running here if any errors are found. + + ### Compute differences + logger.info("Computing differences.") + # 1. Basic Phenotype data differences + # a. Descriptions differences + _desc_diff = descriptions_differences( + file_contents, __fetch_phenotypes__(conn, pheno_ids)) + logger.debug("DESCRIPTIONS DIFFERENCES: %s", _desc_diff) + + # b. Publications differences + _db_publications = fetch_phenotype_publications(conn, pheno_xref_ids) + logger.debug("DB PUBLICATIONS: %s", _db_publications) + + _pubmed_map = { + (int(row["PubMed_ID"]) if bool(row["PubMed_ID"]) else None): f"{row['phenotype_id']}::{row['xref_id']}" + for row in file_contents + } + _pub_id_map = { + f"{pub['PhenotypeId']}::{pub['xref_id']}": pub["PublicationId"] + for pub in _db_publications + } + + _new_publications = update_publications( + conn, tuple({ + **pub, "publication_id": _pub_id_map[_pubmed_map[pub["pubmed_id"]]] + } for pub in pmed.fetch_publications(tuple( + pubmed_id for pubmed_id in pubmed_ids + if pubmed_id not in + tuple(row["PubMed_ID"] for row in _db_publications))))) + _pub_diff = publications_differences( + file_contents, _db_publications, { + row["PubMed_ID" if "PubMed_ID" in row else "pubmed_id"]: row[ + "PublicationId" if "PublicationId" in row else "publication_id"] + for row in _db_publications + _new_publications}) + logger.debug("Publications diff: %s", _pub_diff) + # 2. Data differences + _db_pheno_data = phenotypes_data_by_ids(conn, tuple({ + "population_id": job["metadata"]["population-id"], + "phenoid": row[0], + "xref_id": row[1] + } for row in pheno_xref_ids)) + + data_diff = phenotypes_data_differences( + ({ + "phenotype_id": row["phenotype_id"], + "xref_id": row["xref_id"], + "data": { + key:val for key,val in row.items() + if key not in BULK_EDIT_COMMON_FIELDNAMES + [ + "phenotype_id", "xref_id"] + } + } for row in file_contents), + ({ + **row, + "PhenotypeId": row["Id"], + "data": { + dataitem["StrainName"]: dataitem + for dataitem in row["data"].values() + } + } for row in _db_pheno_data)) + logger.debug("Data differences: %s", data_diff) + ### END: Compute differences + update_descriptions() + link_publications() + update_values() + return 0 + + +def main(): + """Entry-point for this script.""" + args = parse_args() + logger.setLevel(args.log_level.upper()) + logger.debug("Arguments: %s", args) + + logging.getLogger("uploader.phenotypes.misc").setLevel(args.log_level.upper()) + logging.getLogger("uploader.phenotypes.models").setLevel(args.log_level.upper()) + logging.getLogger("uploader.publications.models").setLevel(args.log_level.upper()) + + with (mysqldb.database_connection(args.db_uri) as conn, + sqlite3.connection(args.jobs_db_path) as jobs_conn): + return run(conn, jobs.job(jobs_conn, args.job_id)) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/process_rqtl2_bundle.py b/scripts/process_rqtl2_bundle.py index 20cfd3b..8b7a0fb 100644 --- a/scripts/process_rqtl2_bundle.py +++ b/scripts/process_rqtl2_bundle.py @@ -2,6 +2,7 @@ import sys import uuid import json +import argparse import traceback from typing import Any from pathlib import Path @@ -10,6 +11,7 @@ from logging import Logger, getLogger, StreamHandler import MySQLdb as mdb from redis import Redis +from gn_libs.mysqldb import database_connection from functional_tools import take @@ -18,7 +20,6 @@ import r_qtl.r_qtl2_qc as rqc import r_qtl.exceptions as rqe from uploader import jobs -from uploader.db_utils import database_connection from uploader.check_connections import check_db, check_redis from scripts.cli_parser import init_cli_parser @@ -93,11 +94,14 @@ def process_bundle(dbconn: mdb.Connection, if has_geno_file(thejob): logger.info("Processing geno files.") genoexit = install_genotypes( + rconn, dbconn, - meta["speciesid"], - meta["populationid"], - meta["geno-dataset-id"], - Path(meta["rqtl2-bundle-file"]), + f"{rprefix}:{jobid}", + argparse.Namespace( + speciesid=meta["speciesid"], + populationid=meta["populationid"], + datasetid=meta["geno-dataset-id"], + rqtl2bundle=Path(meta["rqtl2-bundle-file"])), logger) if genoexit != 0: raise Exception("Processing 'geno' file failed.") @@ -108,11 +112,14 @@ def process_bundle(dbconn: mdb.Connection, if has_pheno_file(thejob): phenoexit = install_pheno_files( + rconn, dbconn, - meta["speciesid"], - meta["platformid"], - meta["probe-dataset-id"], - Path(meta["rqtl2-bundle-file"]), + f"{rprefix}:{jobid}", + argparse.Namespace( + speciesid=meta["speciesid"], + platformid=meta["platformid"], + dataset_id=meta["probe-dataset-id"], + rqtl2bundle=Path(meta["rqtl2-bundle-file"])), logger) if phenoexit != 0: raise Exception("Processing 'pheno' file failed.") diff --git a/scripts/qc.py b/scripts/qc.py index 6de051f..b00f4c1 100644 --- a/scripts/qc.py +++ b/scripts/qc.py @@ -5,14 +5,14 @@ import mimetypes from typing import Union, Callable from argparse import ArgumentParser +from gn_libs.mysqldb import database_connection + from functional_tools import take from quality_control.utils import make_progress_calculator from quality_control.errors import InvalidValue, DuplicateHeading from quality_control.parsing import FileType, strain_names, collect_errors -from uploader.db_utils import database_connection - from .cli_parser import init_cli_parser diff --git a/scripts/qc_on_rqtl2_bundle.py b/scripts/qc_on_rqtl2_bundle.py index fc95d13..9f9248c 100644 --- a/scripts/qc_on_rqtl2_bundle.py +++ b/scripts/qc_on_rqtl2_bundle.py @@ -12,12 +12,12 @@ from typing import Union, Sequence, Callable, Iterator import MySQLdb as mdb from redis import Redis +from gn_libs.mysqldb import database_connection from quality_control.errors import InvalidValue from quality_control.checks import decimal_points_error from uploader import jobs -from uploader.db_utils import database_connection from uploader.check_connections import check_db, check_redis from r_qtl import r_qtl2 as rqtl2 diff --git a/scripts/redis_logger.py b/scripts/redis_logger.py index 2ae682b..d3fde5f 100644 --- a/scripts/redis_logger.py +++ b/scripts/redis_logger.py @@ -1,5 +1,6 @@ """Utilities to log to redis for our worker scripts.""" import logging +from typing import Optional from redis import Redis @@ -26,6 +27,26 @@ class RedisLogger(logging.Handler): self.redisconnection.rpush(self.messageslistname, self.format(record)) self.redisconnection.expire(self.messageslistname, self.expiry) +class RedisMessageListHandler(logging.Handler): + """Send messages to specified redis list.""" + def __init__(self, + rconn: Redis, + fullyqualifiedkey: str, + loglevel: int = logging.NOTSET, + expiry: Optional[int] = 86400): + super().__init__(loglevel) + self.redisconnection = rconn + self.fullyqualifiedkey = fullyqualifiedkey + self.expiry = expiry + + def emit(self, record): + """Log out to specified `fullyqualifiedkey`.""" + self.redisconnection.rpush(self.fullyqualifiedkey, self.format(record)) + if bool(self.expiry): + self.redisconnection.expire(self.fullyqualifiedkey, self.expiry) + else: + self.redisconnection.persist(self.fullyqualifiedkey) + def setup_redis_logger(rconn: Redis, fullyqualifiedjobid: str, job_messagelist: str, diff --git a/scripts/rqtl2/bundleutils.py b/scripts/rqtl2/bundleutils.py new file mode 100644 index 0000000..17faa7c --- /dev/null +++ b/scripts/rqtl2/bundleutils.py @@ -0,0 +1,44 @@ +"""Common utilities to operate in R/qtl2 bundles.""" +from typing import Union, Callable + +def build_line_splitter(cdata: dict) -> Callable[[str], tuple[Union[str, None], ...]]: + """Build and return a function to use to split data in the files. + + Parameters + ---------- + cdata: A dict holding the control information included with the R/qtl2 + bundle. + + Returns + ------- + A function that takes a string and return a tuple of strings. + """ + separator = cdata["sep"] + na_strings = cdata["na.strings"] + def __splitter__(line: str) -> tuple[Union[str, None], ...]: + return tuple( + item if item not in na_strings else None + for item in + (field.strip() for field in line.strip().split(separator))) + return __splitter__ + + +def build_line_joiner(cdata: dict) -> Callable[[tuple[Union[str, None], ...]], str]: + """Build and return a function to use to split data in the files. + + Parameters + ---------- + cdata: A dict holding the control information included with the R/qtl2 + bundle. + + Returns + ------- + A function that takes a string and return a tuple of strings. + """ + separator = cdata["sep"] + na_strings = cdata["na.strings"] + def __joiner__(row: tuple[Union[str, None], ...]) -> str: + return separator.join( + (na_strings[0] if item is None else item) + for item in row) + return __joiner__ diff --git a/scripts/rqtl2/cli_parser.py b/scripts/rqtl2/cli_parser.py index bcc7a4f..9bb60a3 100644 --- a/scripts/rqtl2/cli_parser.py +++ b/scripts/rqtl2/cli_parser.py @@ -2,12 +2,22 @@ from pathlib import Path from argparse import ArgumentParser -def add_common_arguments(parser: ArgumentParser) -> ArgumentParser: - """Add common arguments to the CLI parser.""" - parser.add_argument("datasetid", - type=int, - help="The dataset to which the data belongs.") +def add_bundle_argument(parser: ArgumentParser) -> ArgumentParser: + """Add the `rqtl2bundle` argument.""" parser.add_argument("rqtl2bundle", type=Path, help="Path to R/qtl2 bundle zip file.") return parser + + +def add_datasetid_argument(parser: ArgumentParser) -> ArgumentParser: + """Add the `datasetid` argument.""" + parser.add_argument("datasetid", + type=int, + help="The dataset to which the data belongs.") + return parser + + +def add_common_arguments(parser: ArgumentParser) -> ArgumentParser: + """Add common arguments to the CLI parser.""" + return add_bundle_argument(add_datasetid_argument(parser)) diff --git a/scripts/rqtl2/entry.py b/scripts/rqtl2/entry.py index b7fb68e..e0e00e7 100644 --- a/scripts/rqtl2/entry.py +++ b/scripts/rqtl2/entry.py @@ -1,38 +1,58 @@ """Build common script-entry structure.""" -from logging import Logger +import sys +import logging from typing import Callable from argparse import Namespace +from logging import StreamHandler from redis import Redis from MySQLdb import Connection +from gn_libs.mysqldb import database_connection from uploader import jobs -from uploader.db_utils import database_connection from uploader.check_connections import check_db, check_redis from scripts.redis_logger import setup_redis_logger -def build_main(args: Namespace, - run_fn: Callable[[Connection, Namespace], int], - logger: Logger, - loglevel: str = "INFO") -> Callable[[],int]: +def build_main( + args: Namespace, + run_fn: Callable[ + [Redis, Connection, str, Namespace, logging.Logger], + int + ], + logger: logging.Logger +) -> Callable[[],int]: """Build a function to be used as an entry-point for scripts.""" def main(): - check_db(args.databaseuri) - check_redis(args.redisuri) - if not args.rqtl2bundle.exists(): - logger.error("File not found: '%s'.", args.rqtl2bundle) - return 2 - with (Redis.from_url(args.redisuri, decode_responses=True) as rconn, database_connection(args.databaseuri) as dbconn): - fqjobid = jobs.job_key(jobs.jobsnamespace(), args.jobid) - logger.addHandler(setup_redis_logger( - rconn, - fqjobid, - f"{fqjobid}:log-messages", - args.redisexpiry)) - logger.setLevel(loglevel) - return run_fn(dbconn, args) + logger.setLevel(args.loglevel.upper()) + fqjobid = jobs.job_key(args.redisprefix, args.jobid) + + try: + rconn.hset(fqjobid, "status", "started") + logger.addHandler(setup_redis_logger( + rconn, + fqjobid, + f"{fqjobid}:log-messages", + args.redisexpiry)) + logger.addHandler(StreamHandler(stream=sys.stderr)) + + check_db(args.databaseuri) + check_redis(args.redisuri) + if not args.rqtl2bundle.exists(): + logger.error("File not found: '%s'.", args.rqtl2bundle) + return 2 + + returncode = run_fn(rconn, dbconn, fqjobid, args) + if returncode == 0: + rconn.hset(fqjobid, "status", "completed:success") + return returncode + rconn.hset(fqjobid, "status", "completed:error") + return returncode + except Exception as _exc:# pylint: disable=[broad-except] + logger.error("The process failed!", exc_info=True) + rconn.hset(fqjobid, "status", "completed:error") + return 4 return main diff --git a/scripts/rqtl2/install_genotypes.py b/scripts/rqtl2/install_genotypes.py index 6b89142..8762655 100644 --- a/scripts/rqtl2/install_genotypes.py +++ b/scripts/rqtl2/install_genotypes.py @@ -1,12 +1,13 @@ """Load genotypes from R/qtl2 bundle into the database.""" import sys +import argparse import traceback -from pathlib import Path from zipfile import ZipFile from functools import reduce from typing import Iterator, Optional -from logging import Logger, getLogger, StreamHandler +from logging import Logger, getLogger +from redis import Redis import MySQLdb as mdb from MySQLdb.cursors import DictCursor @@ -19,6 +20,8 @@ from scripts.rqtl2.entry import build_main from scripts.rqtl2.cli_parser import add_common_arguments from scripts.cli_parser import init_cli_parser, add_global_data_arguments +__MODULE__ = "scripts.rqtl2.install_genotypes" + def insert_markers( dbconn: mdb.Connection, speciesid: int, @@ -183,15 +186,16 @@ def cross_reference_genotypes( cursor.executemany(insertquery, insertparams) return cursor.rowcount -def install_genotypes(#pylint: disable=[too-many-arguments, too-many-locals] +def install_genotypes(#pylint: disable=[too-many-locals] + rconn: Redis,#pylint: disable=[unused-argument] dbconn: mdb.Connection, - speciesid: int, - populationid: int, - datasetid: int, - rqtl2bundle: Path, + fullyqualifiedjobid: str,#pylint: disable=[unused-argument] + args: argparse.Namespace, logger: Logger = getLogger(__name__) ) -> int: """Load any existing genotypes into the database.""" + (speciesid, populationid, datasetid, rqtl2bundle) = ( + args.speciesid, args.populationid, args.datasetid, args.rqtl2bundle) count = 0 with ZipFile(str(rqtl2bundle.absolute()), "r") as zfile: try: @@ -253,15 +257,5 @@ if __name__ == "__main__": return parser.parse_args() - thelogger = getLogger("install_genotypes") - thelogger.addHandler(StreamHandler(stream=sys.stderr)) - main = build_main( - cli_args(), - lambda dbconn, args: install_genotypes(dbconn, - args.speciesid, - args.populationid, - args.datasetid, - args.rqtl2bundle), - thelogger, - "INFO") + main = build_main(cli_args(), install_genotypes, __MODULE__) sys.exit(main()) diff --git a/scripts/rqtl2/install_phenos.py b/scripts/rqtl2/install_phenos.py index b5cab8e..9059cd6 100644 --- a/scripts/rqtl2/install_phenos.py +++ b/scripts/rqtl2/install_phenos.py @@ -1,11 +1,12 @@ """Load pheno from R/qtl2 bundle into the database.""" import sys +import argparse import traceback -from pathlib import Path from zipfile import ZipFile from functools import reduce -from logging import Logger, getLogger, StreamHandler +from logging import Logger, getLogger +from redis import Redis import MySQLdb as mdb from MySQLdb.cursors import DictCursor @@ -18,6 +19,8 @@ from r_qtl import r_qtl2_qc as rqc from functional_tools import take +__MODULE__ = "scripts.rqtl2.install_phenos" + def insert_probesets(dbconn: mdb.Connection, platformid: int, phenos: tuple[str, ...]) -> int: @@ -93,14 +96,15 @@ def cross_reference_probeset_data(dbconn: mdb.Connection, } for row in dataids)) return cursor.rowcount -def install_pheno_files(#pylint: disable=[too-many-arguments, too-many-locals] +def install_pheno_files(#pylint: disable=[too-many-locals] + rconn: Redis,#pylint: disable=[unused-argument] dbconn: mdb.Connection, - speciesid: int, - platformid: int, - datasetid: int, - rqtl2bundle: Path, + fullyqualifiedjobid: str,#pylint: disable=[unused-argument] + args: argparse.Namespace, logger: Logger = getLogger()) -> int: """Load data in `pheno` files and other related files into the database.""" + (speciesid, platformid, datasetid, rqtl2bundle) = ( + args.speciesid, args.platformid, args.datasetid, args.rqtl2bundle) with ZipFile(str(rqtl2bundle), "r") as zfile: try: rqc.validate_bundle(zfile) @@ -155,16 +159,5 @@ if __name__ == "__main__": return parser.parse_args() - thelogger = getLogger("install_phenos") - thelogger.addHandler(StreamHandler(stream=sys.stderr)) - main = build_main( - cli_args(), - lambda dbconn, args: install_pheno_files(dbconn, - args.speciesid, - args.platformid, - args.datasetid, - args.rqtl2bundle, - thelogger), - thelogger, - "DEBUG") + main = build_main(cli_args(), install_pheno_files, __MODULE__) sys.exit(main()) diff --git a/scripts/rqtl2/phenotypes_qc.py b/scripts/rqtl2/phenotypes_qc.py new file mode 100644 index 0000000..5c89ca0 --- /dev/null +++ b/scripts/rqtl2/phenotypes_qc.py @@ -0,0 +1,516 @@ +"""Run quality control on phenotypes-specific files in the bundle.""" +import sys +import uuid +import json +import shutil +import logging +import tempfile +import contextlib +from pathlib import Path +from logging import Logger +from zipfile import ZipFile +from argparse import Namespace +import multiprocessing as mproc +from functools import reduce, partial +from typing import Union, Iterator, Callable, Optional, Sequence + +import MySQLdb as mdb +from redis import Redis + +from r_qtl import r_qtl2 as rqtl2 +from r_qtl import r_qtl2_qc as rqc +from r_qtl import exceptions as rqe +from r_qtl.fileerrors import InvalidValue + +from functional_tools import chain + +from quality_control.checks import decimal_places_pattern + +from uploader.files import sha256_digest_over_file +from uploader.samples.models import samples_by_species_and_population + +from scripts.rqtl2.entry import build_main +from scripts.redis_logger import RedisMessageListHandler +from scripts.rqtl2.cli_parser import add_bundle_argument +from scripts.cli_parser import init_cli_parser, add_global_data_arguments +from scripts.rqtl2.bundleutils import build_line_joiner, build_line_splitter + +__MODULE__ = "scripts.rqtl2.phenotypes_qc" +logging.basicConfig( + format=("%(asctime)s - %(levelname)s %(name)s: " + "(%(pathname)s: %(lineno)d) %(message)s")) +logger = logging.getLogger(__MODULE__) + +def validate(phenobundle: Path, logger: Logger) -> dict: + """Check that the bundle is generally valid""" + try: + rqc.validate_bundle(phenobundle) + except rqe.RQTLError as rqtlerr: + # logger.error("Bundle file validation failed!", exc_info=True) + return { + "skip": True, + "logger": logger, + "phenobundle": phenobundle, + "errors": (" ".join(rqtlerr.args),) + } + return { + "errors": tuple(), + "skip": False, + "phenobundle": phenobundle, + "logger": logger + } + + +def check_for_mandatory_pheno_keys( + phenobundle: Path, + logger: Logger, + **kwargs +) -> dict: + """Check that the mandatory keys exist for phenotypes.""" + if kwargs.get("skip", False): + return { + **kwargs, + "logger": logger, + "phenobundle": phenobundle + } + + _mandatory_keys = ("pheno", "phenocovar") + _cdata = rqtl2.read_control_file(phenobundle) + _errors = kwargs.get("errors", tuple()) + tuple( + f"Expected '{key}' file(s) are not declared in the bundle." + for key in _mandatory_keys if key not in _cdata.keys()) + return { + **kwargs, + "logger": logger, + "phenobundle": phenobundle, + "errors": _errors, + "skip": len(_errors) > 0 + } + + +def check_for_averages_files( + phenobundle: Path, + logger: Logger, + **kwargs +) -> dict: + """Check that averages files appear together""" + if kwargs.get("skip", False): + return { + **kwargs, + "logger": logger, + "phenobundle": phenobundle + } + + _together = (("phenose", "phenonum"), ("phenonum", "phenose")) + _cdata = rqtl2.read_control_file(phenobundle) + _errors = kwargs.get("errors", tuple()) + tuple( + f"'{first}' is defined in the control file but there is no " + f"corresponding '{second}'" + for first, second in _together + if ((first in _cdata.keys()) and (second not in _cdata.keys()))) + return { + **kwargs, + "logger": logger, + "phenobundle": phenobundle, + "errors": _errors, + "skip": len(_errors) > 0 + } + + +def extract_bundle( + bundle: Path, workdir: Path, jobid: uuid.UUID +) -> tuple[Path, tuple[Path, ...]]: + """Extract the bundle.""" + with ZipFile(bundle) as zfile: + extractiondir = workdir.joinpath( + f"{str(jobid)}-{sha256_digest_over_file(bundle)}-{bundle.name}") + return extractiondir, rqtl2.extract(zfile, extractiondir) + + +def undo_transpose(filetype: str, cdata: dict, extractiondir): + """Undo transposition of all files of type `filetype` in thebundle.""" + if len(cdata.get(filetype, [])) > 0 and cdata.get(f"{filetype}_transposed", False): + files = (extractiondir.joinpath(_file) for _file in cdata[filetype]) + for _file in files: + rqtl2.transpose_csv_with_rename( + _file, + build_line_splitter(cdata), + build_line_joiner(cdata)) + + +@contextlib.contextmanager +def redis_logger( + redisuri: str, loggername: str, filename: str, fqkey: str +) -> Iterator[logging.Logger]: + """Build a Redis message-list logger.""" + rconn = Redis.from_url(redisuri, decode_responses=True) + logger = logging.getLogger(loggername) + logger.propagate = False + handler = RedisMessageListHandler( + rconn, + fullyqualifiedkey(fqkey, filename))#type: ignore[arg-type] + handler.setFormatter(logging.getLogger().handlers[0].formatter) + logger.addHandler(handler) + try: + yield logger + finally: + rconn.close() + + +def push_error(rconn: Redis, fqkey: str, error: InvalidValue) -> InvalidValue: + """Persist the error in redis.""" + rconn.rpush(fqkey, json.dumps(error._asdict())) + return error + + +def file_fqkey(prefix: str, section: str, filepath: Path) -> str: + """Build a files fully-qualified key in a consistent manner""" + return f"{prefix}:{section}:{filepath.name}" + + +def qc_phenocovar_file( + filepath: Path, + redisuri, + fqkey: str, + separator: str, + comment_char: str): + """Check that `phenocovar` files are structured correctly.""" + with (redis_logger( + redisuri, + f"{__MODULE__}.qc_phenocovar_file", + filepath.name, + f"{fqkey}:logs") as logger, + Redis.from_url(redisuri, decode_responses=True) as rconn): + print("Running QC on file: ", filepath.name) + _csvfile = rqtl2.read_csv_file(filepath, separator, comment_char) + _headings = tuple(heading.lower() for heading in next(_csvfile)) + _errors: tuple[InvalidValue, ...] = tuple() + save_error = partial( + push_error, rconn, file_fqkey(fqkey, "errors", filepath)) + for heading in ("description", "units"): + if heading not in _headings: + _errors = (save_error(InvalidValue( + filepath.name, + "header row", + "-", + "-", + (f"File {filepath.name} is missing the {heading} heading " + "in the header line."))),) + + def collect_errors(errors_and_linecount, line): + _errs, _lc = errors_and_linecount + logger.info("Testing record '%s'", line[0]) + if len(line) != len(_headings): + _errs = _errs + (save_error(InvalidValue( + filepath.name, + line[0], + "-", + "-", + (f"Record {_lc} in file {filepath.name} has a different " + "number of columns than the number of headings"))),) + _line = dict(zip(_headings, line)) + if not bool(_line.get("description")): + _errs = _errs + ( + save_error(InvalidValue(filepath.name, + _line[_headings[0]], + "description", + _line.get("description"), + "The description is not provided!")),) + + rconn.hset(file_fqkey(fqkey, "metadata", filepath), + mapping={ + "status": "checking", + "linecount": _lc+1, + "total-errors": len(_errs) + }) + return _errs, _lc+1 + + _errors, _linecount = reduce(collect_errors, _csvfile, (_errors, 1)) + rconn.hset(file_fqkey(fqkey, "metadata", filepath), + mapping={ + "status": "completed", + "linecount": _linecount, + "total-errors": len(_errors) + }) + return {filepath.name: {"errors": _errors, "linecount": _linecount}} + + +def merge_dicts(*dicts): + """Merge multiple dicts into a single one.""" + return reduce(lambda merged, dct: {**merged, **dct}, dicts, {}) + + +def decimal_points_error(# pylint: disable=[too-many-arguments] + filename: str, + rowtitle: str, + coltitle: str, + cellvalue: str, + message: str, + decimal_places: int = 1 +) -> Optional[InvalidValue]: + """Returns an error if the value does not meet the checks.""" + if not bool(decimal_places_pattern(decimal_places).match(cellvalue)): + return InvalidValue(filename, rowtitle, coltitle, cellvalue, message) + return None + + +def integer_error( + filename: str, + rowtitle: str, + coltitle: str, + cellvalue: str, + message: str +) -> Optional[InvalidValue]: + """Returns an error if the value does not meet the checks.""" + try: + value = int(cellvalue) + if value <= 0: + raise ValueError("Must be a non-zero, positive number.") + return None + except ValueError as _verr: + return InvalidValue(filename, rowtitle, coltitle, cellvalue, message) + + +def qc_pheno_file(# pylint: disable=[too-many-locals, too-many-arguments] + filepath: Path, + redisuri: str, + fqkey: str, + samples: tuple[str, ...], + phenonames: tuple[str, ...], + separator: str, + comment_char: str, + na_strings: Sequence[str], + error_fn: Callable = decimal_points_error +): + """Run QC/QA on a `pheno` file.""" + with (redis_logger( + redisuri, + f"{__MODULE__}.qc_pheno_file", + filepath.name, + f"{fqkey}:logs") as logger, + Redis.from_url(redisuri, decode_responses=True) as rconn): + print("Running QC on file: ", filepath.name) + save_error = partial( + push_error, rconn, file_fqkey(fqkey, "errors", filepath)) + _csvfile = rqtl2.read_csv_file(filepath, separator, comment_char) + _headings: tuple[str, ...] = tuple( + # select lowercase for comparison purposes + heading.lower() for heading in next(_csvfile)) + _errors: tuple[InvalidValue, ...] = tuple() + + _absent = tuple(pheno for pheno in _headings[1:] if pheno + not in tuple( + # lower to have consistent case with headings for + # comparison + phe.lower() for phe in phenonames)) + if len(_absent) > 0: + _errors = _errors + (save_error(InvalidValue( + filepath.name, + "header row", + "-", + ", ".join(_absent), + ("The following phenotype names do not exist in any of the " + f"provided phenocovar files: ({', '.join(_absent)})"))),) + + def collect_errors(errors_and_linecount, line): + _errs, _lc = errors_and_linecount + logger.debug("Checking row %s", line[0]) + if line[0] not in samples: + _errs = _errs + (save_error(InvalidValue( + filepath.name, + line[0], + _headings[0], + line[0], + (f"The sample named '{line[0]}' does not exist in the database. " + "You will need to upload that first."))),) + + for field, value in zip(_headings[1:], line[1:]): + if value in na_strings: + continue + _err = error_fn( + filepath.name, + line[0], + field, + value) + _errs = _errs + ((save_error(_err),) if bool(_err) else tuple()) + + rconn.hset(file_fqkey(fqkey, "metadata", filepath), + mapping={ + "status": "checking", + "linecount": _lc+1, + "total-errors": len(_errs) + }) + return _errs, _lc+1 + + _errors, _linecount = reduce(collect_errors, _csvfile, (_errors, 1)) + rconn.hset(file_fqkey(fqkey, "metadata", filepath), + mapping={ + "status": "completed", + "linecount": _linecount, + "total-errors": len(_errors) + }) + return {filepath.name: {"errors": _errors, "linecount": _linecount}} + + +def phenotype_names(filepath: Path, + separator: str, + comment_char: str) -> tuple[str, ...]: + """Read phenotype names from `phenocovar` file.""" + return reduce(lambda tpl, line: tpl + (line[0],),#type: ignore[arg-type, return-value] + rqtl2.read_csv_file(filepath, separator, comment_char), + tuple())[1:] + +def fullyqualifiedkey( + prefix: str, + rest: Optional[str] = None +) -> Union[Callable[[str], str], str]: + """Compute fully qualified Redis key.""" + if not bool(rest): + return lambda _rest: f"{prefix}:{_rest}" + return f"{prefix}:{rest}" + +def run_qc(# pylint: disable=[too-many-locals] + rconn: Redis, + dbconn: mdb.Connection, + fullyqualifiedjobid: str, + args: Namespace +) -> int: + """Run quality control checks on the bundle.""" + print("Beginning the quality assurance checks.") + results = check_for_averages_files( + **check_for_mandatory_pheno_keys( + **validate(args.rqtl2bundle, logger))) + errors = results.get("errors", tuple()) + if len(errors) > 0: + logger.error("We found the following errors:\n%s", + "\n".join(f" - {error}" for error in errors)) + return 1 + # Run QC on actual values + # Steps: + # - Extract file to specific directory + extractiondir, *_bundlefiles = extract_bundle( + args.rqtl2bundle, args.workingdir, args.jobid) + + # - For every pheno, phenocovar, phenose, phenonum file, undo + # transposition where relevant + cdata = rqtl2.control_data(extractiondir) + with mproc.Pool(mproc.cpu_count() - 1) as pool: + pool.starmap( + undo_transpose, + ((ftype, cdata, extractiondir) + for ftype in ("pheno", "phenocovar", "phenose", "phenonum"))) + + # - Fetch samples/individuals from database. + print("Fetching samples/individuals from the database.") + samples = tuple(#type: ignore[var-annotated] + item for item in set(reduce( + lambda acc, item: acc + ( + item["Name"], item["Name2"], item["Symbol"], item["Alias"]), + samples_by_species_and_population( + dbconn, args.speciesid, args.populationid), + tuple())) + if bool(item)) + + # - Check that `description` and `units` is present in phenocovar for + # all phenotypes + rconn.hset(fullyqualifiedjobid, + "fully-qualified-keys:phenocovar", + json.dumps(tuple(f"{fullyqualifiedjobid}:phenocovar:{_file}" + for _file in cdata.get("phenocovar", [])))) + with mproc.Pool(mproc.cpu_count() - 1) as pool: + print("Check for errors in 'phenocovar' file(s).") + _phenocovar_qc_res = merge_dicts(*pool.starmap(qc_phenocovar_file, tuple( + (extractiondir.joinpath(_file), + args.redisuri, + f"{fullyqualifiedjobid}:phenocovar", + cdata["sep"], + cdata["comment.char"]) + for _file in cdata.get("phenocovar", [])))) + + # - Check all samples in pheno files exist in database + # - Check all phenotypes in pheno files exist in phenocovar files + # - Check all numeric values in pheno files + phenonames = tuple(set( + name for names in pool.starmap(phenotype_names, tuple( + (extractiondir.joinpath(_file), cdata["sep"], cdata["comment.char"]) + for _file in cdata.get("phenocovar", []))) + for name in names)) + + dec_err_fn = partial(decimal_points_error, message=( + "Expected a non-negative number with at least one decimal " + "place.")) + + print("Check for errors in 'pheno' file(s).") + _pheno_qc_res = merge_dicts(*pool.starmap(qc_pheno_file, tuple(( + extractiondir.joinpath(_file), + args.redisuri, + chain( + "pheno", + fullyqualifiedkey(args.jobid), + fullyqualifiedkey(args.redisprefix)), + samples, + phenonames, + cdata["sep"], + cdata["comment.char"], + cdata["na.strings"], + dec_err_fn + ) for _file in cdata.get("pheno", [])))) + + # - Check the 3 checks above for phenose and phenonum values too + # qc_phenose_files(…) + # qc_phenonum_files(…) + print("Check for errors in 'phenose' file(s).") + _phenose_qc_res = merge_dicts(*pool.starmap(qc_pheno_file, tuple(( + extractiondir.joinpath(_file), + args.redisuri, + chain( + "phenose", + fullyqualifiedkey(args.jobid), + fullyqualifiedkey(args.redisprefix)), + samples, + phenonames, + cdata["sep"], + cdata["comment.char"], + cdata["na.strings"], + dec_err_fn + ) for _file in cdata.get("phenose", [])))) + + print("Check for errors in 'phenonum' file(s).") + _phenonum_qc_res = merge_dicts(*pool.starmap(qc_pheno_file, tuple(( + extractiondir.joinpath(_file), + args.redisuri, + chain( + "phenonum", + fullyqualifiedkey(args.jobid), + fullyqualifiedkey(args.redisprefix)), + samples, + phenonames, + cdata["sep"], + cdata["comment.char"], + cdata["na.strings"], + partial(integer_error, message=( + "Expected a non-negative, non-zero integer value.")) + ) for _file in cdata.get("phenonum", [])))) + + # - Delete all extracted files + shutil.rmtree(extractiondir) + return 0 + + +if __name__ == "__main__": + def cli_args(): + """Process command-line arguments for `install_phenos`""" + parser = add_bundle_argument(add_global_data_arguments(init_cli_parser( + program="PhenotypesQC", + description=( + "Perform Quality Control checks on a phenotypes bundle file")))) + parser.add_argument( + "--workingdir", + default=f"{tempfile.gettempdir()}/phenotypes_qc", + help=("The directory where this script will put its intermediate " + "files."), + type=Path) + return parser.parse_args() + + main = build_main(cli_args(), run_qc, logger) + sys.exit(main()) diff --git a/scripts/validate_file.py b/scripts/validate_file.py index a40d7e7..52e55ec 100644 --- a/scripts/validate_file.py +++ b/scripts/validate_file.py @@ -8,12 +8,12 @@ from zipfile import ZipFile, is_zipfile import jsonpickle from redis import Redis from redis.exceptions import ConnectionError # pylint: disable=[redefined-builtin] +from gn_libs.mysqldb import database_connection from quality_control.utils import make_progress_calculator from quality_control.parsing import FileType, strain_names, collect_errors from uploader import jobs -from uploader.db_utils import database_connection from .cli_parser import init_cli_parser from .qc import add_file_validation_arguments @@ -25,6 +25,7 @@ setup( "redis", "Flask", "pyyaml", + "gn-libs", "gunicorn", "jsonpickle", "mysqlclient"], diff --git a/tests/conftest.py b/tests/conftest.py index 9012221..a716c52 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -2,6 +2,8 @@ import io import os import uuid +import shutil +from pathlib import Path from hashlib import sha256 import redis @@ -46,17 +48,20 @@ def cleanup_redis(redisuri: str, prefix: str): @pytest.fixture(scope="module") def client(): "Fixture for test client" - app = create_app() test_prefix = sha256(f"test:{uuid.uuid4()}".encode("utf8")).hexdigest() - app.config.update({ + tests_work_dir = Path("/tmp/{test_prefix}") + tests_work_dir.mkdir(exist_ok=True) + app = create_app({ "TESTING": True, "GNQC_REDIS_PREFIX": f"{test_prefix}:GNQC", - "JOBS_TTL_SECONDS": 2 * 60 * 60# 2 hours + "JOBS_TTL_SECONDS": 2 * 60 * 60,# 2 hours + "ASYNCHRONOUS_JOBS_SQLITE_DB": f"{tests_work_dir}/jobs.db" }) with app.app_context(): yield app.test_client() cleanup_redis(app.config["REDIS_URL"], test_prefix) + shutil.rmtree(tests_work_dir, ignore_errors=True) @pytest.fixture(scope="module") def db_url(client):#pylint: disable=[redefined-outer-name] diff --git a/tests/r_qtl/test_r_qtl2_control_file.py b/tests/r_qtl/test_r_qtl2_control_file.py index 316307d..5b9fef6 100644 --- a/tests/r_qtl/test_r_qtl2_control_file.py +++ b/tests/r_qtl/test_r_qtl2_control_file.py @@ -16,6 +16,7 @@ __DEFAULTS__ = { "pheno_transposed": False, "covar_transposed": False, "phenocovar_transposed": False, + "phenonum_transposed": False, "gmap_transposed": False, "pmap_transposed": False, "phenose_transposed": False diff --git a/tests/qc_app/__init__.py b/tests/uploader/__init__.py index e69de29..e69de29 100644 --- a/tests/qc_app/__init__.py +++ b/tests/uploader/__init__.py diff --git a/tests/uploader/phenotypes/__init__.py b/tests/uploader/phenotypes/__init__.py new file mode 100644 index 0000000..1e0a932 --- /dev/null +++ b/tests/uploader/phenotypes/__init__.py @@ -0,0 +1 @@ +"""phenotypes tests""" diff --git a/tests/uploader/phenotypes/test_misc.py b/tests/uploader/phenotypes/test_misc.py new file mode 100644 index 0000000..cf475ad --- /dev/null +++ b/tests/uploader/phenotypes/test_misc.py @@ -0,0 +1,387 @@ +"""Test miscellaneous phenotypes functions.""" + +import pytest + +from uploader.phenotypes.misc import phenotypes_data_differences + +__sample_db_phenotypes_data__ = ( + { + "PhenotypeId": 4, + "xref_id": 10001, + "DataId": 8967043, + "data": { + "B6D2F1": {"StrainId": 1, "value": None}, + "C57BL/6J": {"StrainId": 2, "value": None}, + "DBA/2J": {"StrainId": 3, "value": None}, + "BXD1": {"StrainId": 4, "value": 61.4}, + "BXD2": {"StrainId": 5, "value": 49}, + "BXD5": {"StrainId": 6, "value": 62.5}, + "BXD6": {"StrainId": 7, "value": 53.1} + } + }, + { + "PhenotypeId": 10, + "xref_id": 10002, + "DataId": 8967044, + "data": { + "B6D2F1": {"StrainId": 1, "value": None}, + "C57BL/6J": {"StrainId": 2, "value": None}, + "DBA/2J": {"StrainId": 3, "value": None}, + "BXD1": {"StrainId": 4, "value": 54.1}, + "BXD2": {"StrainId": 5, "value": 50.1}, + "BXD5": {"StrainId": 6, "value": 53.3}, + "BXD6": {"StrainId": 7, "value": 55.1} + } + }, + { + "PhenotypeId": 15, + "xref_id": 10003, + "DataId": 8967045, + "data": { + "B6D2F1": {"StrainId": 1, "value": None}, + "C57BL/6J": {"StrainId": 2, "value": None}, + "DBA/2J": {"StrainId": 3, "value": None}, + "BXD1": {"StrainId": 4, "value": 483}, + "BXD2": {"StrainId": 5, "value": 403}, + "BXD5": {"StrainId": 6, "value": 501}, + "BXD6": {"StrainId": 7, "value": 403} + } + }, + { + "PhenotypeId": 20, + "xref_id": 10004, + "DataId": 8967046, + "data": { + "B6D2F1": {"StrainId": 1, "value": None}, + "C57BL/6J": {"StrainId": 2, "value": None}, + "DBA/2J": {"StrainId": 3, "value": None}, + "BXD1": {"StrainId": 4, "value": 49.8}, + "BXD2": {"StrainId": 5, "value": 45.5}, + "BXD5": {"StrainId": 6, "value": 62.9}, + "BXD6": {"StrainId": 7, "value": None} + } + }, + { + "PhenotypeId": 25, + "xref_id": 10005, + "DataId": 8967047, + "data": { + "B6D2F1": {"StrainId": 1, "value": None}, + "C57BL/6J": {"StrainId": 2, "value": None}, + "DBA/2J": {"StrainId": 3, "value": None}, + "BXD1": {"StrainId": 4, "value": 46}, + "BXD2": {"StrainId": 5, "value": 44.9}, + "BXD5": {"StrainId": 6, "value": 52.5}, + "BXD6": {"StrainId": 7, "value": None} + } + }) + + +@pytest.mark.unit_test +@pytest.mark.parametrize( + "filedata,dbdata,expected", + ((tuple(), tuple(), tuple()), # No data + + # No data difference + (({ + "phenotype_id": 4, + "xref_id": 10001, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 61.4, + "BXD2": 49, + "BXD5":62.5, + "BXD6": 53.1 + } + }, + { + "phenotype_id": 10, + "xref_id": 10002, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 54.1, + "BXD2": 50.1, + "BXD5": 53.3, + "BXD6": 55.1 + } + }, + { + "phenotype_id": 15, + "xref_id": 10003, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 483, + "BXD2": 403, + "BXD5": 501, + "BXD6": 403 + } + }, + { + "phenotype_id": 20, + "xref_id": 10004, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 49.8, + "BXD2": 45.5, + "BXD5": 62.9, + "BXD6": None + } + }, + { + "phenotype_id": 25, + "xref_id": 10005, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 46, + "BXD2": 44.9, + "BXD5": 52.5, + "BXD6": None + } + }), + __sample_db_phenotypes_data__, + tuple()), + + # Change values: No deletions + (({ + "phenotype_id": 4, + "xref_id": 10001, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 77.2, + "BXD2": 49, + "BXD5":62.5, + "BXD6": 53.1 + } + }, + { + "phenotype_id": 10, + "xref_id": 10002, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 54.1, + "BXD2": 50.1, + "BXD5": 53.3, + "BXD6": 55.1 + } + }, + { + "phenotype_id": 15, + "xref_id": 10003, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 483, + "BXD2": 403, + "BXD5": 503, + "BXD6": 903 + } + }, + { + "phenotype_id": 20, + "xref_id": 10004, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": 1, + "BXD1": 8, + "BXD2": 9, + "BXD5": 62.9, + "BXD6": None + } + }, + { + "phenotype_id": 25, + "xref_id": 10005, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 46, + "BXD2": 44.9, + "BXD5": 52.5, + "BXD6": None + } + }), + __sample_db_phenotypes_data__, + ({ + "PhenotypeId": 4, + "xref_id": 10001, + "DataId": 8967043, + "StrainId": 4, + "StrainName": "BXD1", + "value": 77.2 + }, + { + "PhenotypeId": 15, + "xref_id": 10003, + "DataId": 8967045, + "StrainId": 6, + "StrainName": "BXD5", + "value": 503 + }, + { + "PhenotypeId": 15, + "xref_id": 10003, + "DataId": 8967045, + "StrainId": 7, + "StrainName": "BXD6", + "value": 903 + }, + { + "PhenotypeId": 20, + "xref_id": 10004, + "DataId": 8967046, + "StrainId": 3, + "StrainName": "DBA/2J", + "value": 1 + }, + { + "PhenotypeId": 20, + "xref_id": 10004, + "DataId": 8967046, + "StrainId": 4, + "StrainName": "BXD1", + "value": 8 + }, + { + "PhenotypeId": 20, + "xref_id": 10004, + "DataId": 8967046, + "StrainId": 5, + "StrainName": "BXD2", + "value": 9 + })), + + # Changes — with deletions + (({ + "phenotype_id": 4, + "xref_id": 10001, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": None, + "BXD2": 49, + "BXD5":62.5, + "BXD6": 53.1 + } + }, + { + "phenotype_id": 10, + "xref_id": 10002, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 54.1, + "BXD2": 50.1, + "BXD5": 53.3, + "BXD6": 55.1 + } + }, + { + "phenotype_id": 15, + "xref_id": 10003, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 483, + "BXD2": 403, + "BXD5": None, + "BXD6": None + } + }, + { + "phenotype_id": 20, + "xref_id": 10004, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": 15, + "BXD1": None, + "BXD2": 24, + "BXD5": 62.9, + "BXD6": None + } + }, + { + "phenotype_id": 25, + "xref_id": 10005, + "data": { + "B6D2F1": None, + "C57BL/6J": None, + "DBA/2J": None, + "BXD1": 46, + "BXD2": 44.9, + "BXD5": 52.5, + "BXD6": None + } + }), + __sample_db_phenotypes_data__, + ({ + "PhenotypeId": 4, + "xref_id": 10001, + "DataId": 8967043, + "StrainId": 4, + "StrainName": "BXD1", + "value": None + }, + { + "PhenotypeId": 15, + "xref_id": 10003, + "DataId": 8967045, + "StrainId": 6, + "StrainName": "BXD5", + "value": None + }, + { + "PhenotypeId": 15, + "xref_id": 10003, + "DataId": 8967045, + "StrainId": 7, + "StrainName": "BXD6", + "value": None + }, + { + "PhenotypeId": 20, + "xref_id": 10004, + "DataId": 8967046, + "StrainId": 3, + "StrainName": "DBA/2J", + "value": 15 + }, + { + "PhenotypeId": 20, + "xref_id": 10004, + "DataId": 8967046, + "StrainId": 4, + "StrainName": "BXD1", + "value": None + }, + { + "PhenotypeId": 20, + "xref_id": 10004, + "DataId": 8967046, + "StrainId": 5, + "StrainName": "BXD2", + "value": 24 + })))) +def test_phenotypes_data_differences(filedata, dbdata, expected): + """Test differences are computed correctly.""" + assert phenotypes_data_differences(filedata, dbdata) == expected diff --git a/tests/uploader/publications/__init__.py b/tests/uploader/publications/__init__.py new file mode 100644 index 0000000..de15e08 --- /dev/null +++ b/tests/uploader/publications/__init__.py @@ -0,0 +1 @@ +"""publications tests""" diff --git a/tests/uploader/publications/test_misc.py b/tests/uploader/publications/test_misc.py new file mode 100644 index 0000000..8c7e567 --- /dev/null +++ b/tests/uploader/publications/test_misc.py @@ -0,0 +1,68 @@ +"""Tests for functions used for bulk editing.""" +import pytest + +from uploader.publications.misc import publications_differences + + +@pytest.mark.unit_test +@pytest.mark.parametrize( + "filedata,dbdata,pubmed2pubidmap,expected", + (((), (), {}, tuple()), # no data + + # Same Data + (({"phenotype_id": 1, "xref_id": 10001, "PubMed_ID": 9999999999999}, + {"phenotype_id": 1, "xref_id": 10002, "PubMed_ID": 9999999999999}, + {"phenotype_id": 1, "xref_id": 10003, "PubMed_ID": 9999999999999}, + {"phenotype_id": 1, "xref_id": 10005, "PubMed_ID": 9999999999999}), + ({"PhenotypeId": 1, "xref_id": 10001, "PublicationId": 15, + "PubMed_ID": 9999999999999}, + {"PhenotypeId": 1, "xref_id": 10002, "PublicationId": 15, + "PubMed_ID": 9999999999999}, + {"PhenotypeId": 1, "xref_id": 10003, "PublicationId": 15, + "PubMed_ID": 9999999999999}, + {"PhenotypeId": 1, "xref_id": 10004, "PublicationId": 15, + "PubMed_ID": 9999999999999}), + {9999999999999: 15}, + tuple()), + + # Differences: no new pubmeds (all pubmeds in db) + (({"phenotype_id": 1, "xref_id": 10001, "PubMed_ID": 9999999999999}, + {"phenotype_id": 1, "xref_id": 10002, "PubMed_ID": 9999999999998}, + {"phenotype_id": 1, "xref_id": 10003, "PubMed_ID": 9999999999999}, + {"phenotype_id": 1, "xref_id": 10004, "PubMed_ID": 9999999999997}), + ({"PhenotypeId": 1, "xref_id": 10001, "PublicationId": 15, + "PubMed_ID": 9999999999999}, + {"PhenotypeId": 1, "xref_id": 10002, "PublicationId": 15, + "PubMed_ID": 9999999999999}, + {"PhenotypeId": 1, "xref_id": 10003, "PublicationId": 15, + "PubMed_ID": 9999999999999}, + {"PhenotypeId": 1, "xref_id": 10004, "PublicationId": 15, + "PubMed_ID": None}), + {9999999999999: 15, 9999999999998: 18, 9999999999997: 12}, + ({"PhenotypeId": 1, "xref_id": 10002, "PublicationId": 18, + "PubMed_ID": 9999999999998}, + {"PhenotypeId": 1, "xref_id": 10004, "PublicationId": 12, + "PubMed_ID": 9999999999997})), + + # Differences: Deletions of pubmeds + (({"phenotype_id": 1, "xref_id": 10001, "PubMed_ID": 9999999999999}, + {"phenotype_id": 1, "xref_id": 10002, "PubMed_ID": None}, + {"phenotype_id": 1, "xref_id": 10003, "PubMed_ID": 9999999999999}, + {"phenotype_id": 1, "xref_id": 10004, "PubMed_ID": None}), + ({"PhenotypeId": 1, "xref_id": 10001, "PublicationId": 15, + "PubMed_ID": 9999999999999}, + {"PhenotypeId": 1, "xref_id": 10002, "PublicationId": 15, + "PubMed_ID": 9999999999999}, + {"PhenotypeId": 1, "xref_id": 10003, "PublicationId": 15, + "PubMed_ID": 9999999999999}, + {"PhenotypeId": 1, "xref_id": 10004, "PublicationId": 15, + "PubMed_ID": 9999999999999}), + {9999999999999: 15, 9999999999998: 18, 9999999999997: 12}, + ({"PhenotypeId": 1, "xref_id": 10002, "PublicationId": None, + "PubMed_ID": None}, + {"PhenotypeId": 1, "xref_id": 10004, "PublicationId": None, + "PubMed_ID": None})))) +def test_publications_differences(filedata, dbdata, pubmed2pubidmap, expected): + """Test publication differences — flesh out description…""" + assert publications_differences( + filedata, dbdata, pubmed2pubidmap) == expected diff --git a/tests/qc_app/test_entry.py b/tests/uploader/test_entry.py index 0c614a5..0c614a5 100644 --- a/tests/qc_app/test_entry.py +++ b/tests/uploader/test_entry.py diff --git a/tests/qc_app/test_expression_data_pages.py b/tests/uploader/test_expression_data_pages.py index c2f7de1..c2f7de1 100644 --- a/tests/qc_app/test_expression_data_pages.py +++ b/tests/uploader/test_expression_data_pages.py diff --git a/tests/uploader/test_files.py b/tests/uploader/test_files.py new file mode 100644 index 0000000..cb22fff --- /dev/null +++ b/tests/uploader/test_files.py @@ -0,0 +1,17 @@ +"""Tests functions in the `uploader.files` module.""" +from pathlib import Path + +import pytest + +from uploader.files import sha256_digest_over_file + +@pytest.mark.unit_test +@pytest.mark.parametrize( + "filepath,expectedhash", + ((Path("tests/test_data/average.tsv.zip"), + "a371c654c095c030edad468e1c3d6b176ea8adfbcd91a322afd37779044478d9"), + (Path("tests/test_data/standarderror.tsv"), + "a08332e0b06391d50eecb722f69d85fbdf374a2d77713ee879d3fd6c60419d55"))) +def test_sha256_digest_over_file(filepath: Path, expectedhash: str): + """Test the `sha256_digest_over_file` function.""" + assert sha256_digest_over_file(filepath) == expectedhash diff --git a/tests/qc_app/test_parse.py b/tests/uploader/test_parse.py index 076c47c..20c75b7 100644 --- a/tests/qc_app/test_parse.py +++ b/tests/uploader/test_parse.py @@ -8,7 +8,8 @@ from uploader.jobs import job, jobsnamespace from tests.conftest import uploadable_file_object -def test_parse_with_existing_uploaded_file(#pylint: disable=[too-many-arguments] +def test_parse_with_existing_uploaded_file( + #pylint: disable=[too-many-arguments,too-many-positional-arguments] client, db_url, redis_url, diff --git a/tests/qc_app/test_progress_indication.py b/tests/uploader/test_progress_indication.py index 14a1050..14a1050 100644 --- a/tests/qc_app/test_progress_indication.py +++ b/tests/uploader/test_progress_indication.py diff --git a/tests/qc_app/test_results_page.py b/tests/uploader/test_results_page.py index 8c8379f..8c8379f 100644 --- a/tests/qc_app/test_results_page.py +++ b/tests/uploader/test_results_page.py diff --git a/tests/qc_app/test_uploads_with_zip_files.py b/tests/uploader/test_uploads_with_zip_files.py index 1506cfa..1506cfa 100644 --- a/tests/qc_app/test_uploads_with_zip_files.py +++ b/tests/uploader/test_uploads_with_zip_files.py diff --git a/uploader/__init__.py b/uploader/__init__.py index 347f170..8b49ad5 100644 --- a/uploader/__init__.py +++ b/uploader/__init__.py @@ -3,18 +3,33 @@ import os import sys import logging from pathlib import Path +from typing import Optional from flask import Flask, request + +from cachelib import FileSystemCache + +from gn_libs import jobs as gnlibs_jobs + from flask_session import Session + from uploader.oauth2.client import user_logged_in, authserver_authorise_uri +from . import session from .base_routes import base +from .files.views import files from .species import speciesbp -from .dbinsert import dbinsertbp +from .publications import pubbp from .oauth2.views import oauth2 from .expression_data import exprdatabp from .errors import register_error_handlers +from .background_jobs import background_jobs_bp + +logging.basicConfig( + format=("%(asctime)s — %(filename)s:%(lineno)s — %(levelname)s " + "(%(thread)d:%(threadName)s): %(message)s") +) def override_settings_with_envvars( app: Flask, ignore: tuple[str, ...]=tuple()) -> None: @@ -49,10 +64,30 @@ def setup_logging(app: Flask) -> Flask: "SERVER_SOFTWARE", "").split('/') return __log_gunicorn__(app) if bool(software) else __log_dev__(app) +def setup_modules_logging(app_logger): + """Setup module-level loggers to the same log-level as the application.""" + loglevel = logging.getLevelName(app_logger.getEffectiveLevel()) + + def __setup__(logger_name): + _logger = logging.getLogger(logger_name) + _logger.setLevel(loglevel) + + __setup__("uploader.publications.models") + __setup__("uploader.publications.datatables") + + +def create_app(config: Optional[dict] = None): + """The application factory. + + config: dict + Useful to override settings in the settings files and environment + especially in environments such as testing.""" + if config is None: + config = {} -def create_app(): - """The application factory""" app = Flask(__name__) + + ### BEGIN: Application configuration app.config.from_pyfile( Path(__file__).parent.joinpath("default_settings.py")) if "UPLOADER_CONF" in os.environ: @@ -67,8 +102,16 @@ def create_app(): if secretsfile.exists(): # Silently ignore secrets if the file does not exist. app.config.from_pyfile(secretsfile) + app.config.update(config) # Override everything with passed in config + ### END: Application configuration + + app.config["SESSION_CACHELIB"] = FileSystemCache( + cache_dir=Path(app.config["SESSION_FILESYSTEM_CACHE_PATH"]).absolute(), + threshold=int(app.config["SESSION_FILESYSTEM_CACHE_THRESHOLD"]), + default_timeout=int(app.config["SESSION_FILESYSTEM_CACHE_TIMEOUT"])) setup_logging(app) + setup_modules_logging(app.logger) # setup jinja2 symbols app.add_template_global(lambda : request.url, name="request_url") @@ -76,15 +119,18 @@ def create_app(): app.add_template_global(lambda: app.config["GN2_SERVER_URL"], name="gn2server_uri") app.add_template_global(user_logged_in) + app.add_template_global(lambda : session.user_details()["email"], name="user_email") Session(app) # setup blueprints app.register_blueprint(base, url_prefix="/") + app.register_blueprint(files, url_prefix="/files") app.register_blueprint(oauth2, url_prefix="/oauth2") app.register_blueprint(speciesbp, url_prefix="/species") - app.register_blueprint(dbinsertbp, url_prefix="/dbinsert") - app.register_blueprint(exprdatabp, url_prefix="/expression-data") + app.register_blueprint(pubbp, url_prefix="/publications") + app.register_blueprint(background_jobs_bp, url_prefix="/background-jobs/") register_error_handlers(app) + gnlibs_jobs.init_app(app) return app diff --git a/uploader/authorisation.py b/uploader/authorisation.py index ee8fe97..3cf3585 100644 --- a/uploader/authorisation.py +++ b/uploader/authorisation.py @@ -16,13 +16,12 @@ def require_login(function): @wraps(function) def __is_session_valid__(*args, **kwargs): """Check that the user is logged in and their token is valid.""" - def __clear_session__(_no_token): - session.clear_session_info() - flash("You need to be logged in.", "alert-danger") + def __alert_needs_sign_in__(_no_token): + flash("You need to be signed in.", "alert alert-danger big-alert") return redirect("/") return session.user_token().either( - __clear_session__, + __alert_needs_sign_in__, lambda token: function(*args, **kwargs)) return __is_session_valid__ @@ -49,7 +48,7 @@ def require_token(func: Callable) -> Callable: """ def __invalid_token__(_whatever): logging.debug("==========> Failure log: %s", _whatever) - raise Exception( + raise Exception(# pylint: disable=[broad-exception-raised] "You attempted to access a feature of the system that requires " "authorisation. Unfortunately, we could not verify you have the " "appropriate authorisation to perform the action you requested. " diff --git a/uploader/background_jobs.py b/uploader/background_jobs.py new file mode 100644 index 0000000..dc9f837 --- /dev/null +++ b/uploader/background_jobs.py @@ -0,0 +1,119 @@ +"""Generic views and utilities to handle background jobs.""" +import uuid +import importlib +from typing import Callable +from functools import partial + +from flask import ( + url_for, + redirect, + Response, + Blueprint, + render_template, + current_app as app) + +from gn_libs import jobs +from gn_libs import sqlite3 +from gn_libs.jobs.jobs import JobNotFound + +from uploader.authorisation import require_login + +background_jobs_bp = Blueprint("background-jobs", __name__) +HandlerType = Callable[[dict], Response] + + +def __default_error_handler__(job: dict) -> Response: + return redirect(url_for("background-jobs.job_error", job_id=job["job_id"])) + +def register_handlers( + job_type: str, + success_handler: HandlerType, + # pylint: disable=[redefined-outer-name] + error_handler: HandlerType = __default_error_handler__ + # pylint: disable=[redefined-outer-name] +) -> str: + """Register success and error handlers for each job type.""" + if not bool(app.config.get("background-jobs")): + app.config["background-jobs"] = {} + + if not bool(app.config["background-jobs"].get(job_type)): + app.config["background-jobs"][job_type] = { + "success": success_handler, + "error": error_handler + } + + return job_type + + +def register_job_handlers(job: str): + """Related to register handlers above.""" + def __load_handler__(absolute_function_path): + _parts = absolute_function_path.split(".") + app.logger.debug("THE PARTS ARE: %s", _parts) + assert len(_parts) > 1, f"Invalid path: {absolute_function_path}" + module = importlib.import_module(f".{_parts[-2]}", + package=".".join(_parts[0:-2])) + return getattr(module, _parts[-1]) + + metadata = job["metadata"] + if metadata["success_handler"]: + _success_handler = __load_handler__(metadata["success_handler"]) + try: + _error_handler = __load_handler__(metadata["error_handler"]) + except Exception as _exc:# pylint: disable=[broad-exception-caught] + _error_handler = __default_error_handler__ + register_handlers( + metadata["job-type"], _success_handler, _error_handler) + + +def handler(job: dict, handler_type: str) -> HandlerType: + """Fetch a handler for the job.""" + _job_type = job["metadata"]["job-type"] + _handler = app.config.get( + "background-jobs", {} + ).get( + _job_type, {} + ).get(handler_type) + if bool(_handler): + return _handler(job) + raise Exception(# pylint: disable=[broad-exception-raised] + f"No '{handler_type}' handler registered for job type: {_job_type}") + + +error_handler = partial(handler, handler_type="error") +success_handler = partial(handler, handler_type="success") + + +@background_jobs_bp.route("/status/<uuid:job_id>") +@require_login +def job_status(job_id: uuid.UUID): + """View the job status.""" + with sqlite3.connection(app.config["ASYNCHRONOUS_JOBS_SQLITE_DB"]) as conn: + try: + job = jobs.job(conn, job_id, fulldetails=True) + status = job["metadata"]["status"] + + register_job_handlers(job) + if status == "error": + return error_handler(job) + + if status == "completed": + return success_handler(job) + + return render_template("jobs/job-status.html", job=job) + except JobNotFound as _jnf: + return render_template( + "jobs/job-not-found.html", + job_id=job_id) + + +@background_jobs_bp.route("/error/<uuid:job_id>") +@require_login +def job_error(job_id: uuid.UUID): + """Handle job errors in a generic manner.""" + with sqlite3.connection(app.config["ASYNCHRONOUS_JOBS_SQLITE_DB"]) as conn: + try: + job = jobs.job(conn, job_id, fulldetails=True) + return render_template("jobs/job-error.html", job=job) + except JobNotFound as _jnf: + return render_template("jobs/job-not-found.html", job_id=job_id) diff --git a/uploader/base_routes.py b/uploader/base_routes.py index 88247b2..74a3b90 100644 --- a/uploader/base_routes.py +++ b/uploader/base_routes.py @@ -1,15 +1,16 @@ """Basic routes required for all pages""" import os +from urllib.parse import urljoin -from flask import ( - Blueprint, - render_template, - current_app as app, - send_from_directory) +from flask import (Blueprint, + current_app as app, + send_from_directory) +from uploader.ui import make_template_renderer from uploader.oauth2.client import user_logged_in base = Blueprint("base", __name__) +render_template = make_template_renderer("home") @base.route("/favicon.ico", methods=["GET"]) @@ -23,7 +24,9 @@ def favicon(): @base.route("/", methods=["GET"]) def index(): """Load the landing page""" - return render_template("index.html" if user_logged_in() else "login.html") + return render_template("index.html" if user_logged_in() else "login.html", + gn2server_intro=urljoin(app.config["GN2_SERVER_URL"], + "/intro")) def appenv(): """Get app's guix environment path.""" @@ -32,8 +35,8 @@ def appenv(): @base.route("/bootstrap/<path:filename>") def bootstrap(filename): """Fetch bootstrap files.""" - return send_from_directory( - appenv(), f"share/genenetwork2/javascript/bootstrap/{filename}") + return send_from_directory(appenv(), f"share/web/bootstrap/{filename}") + @base.route("/jquery/<path:filename>") @@ -43,6 +46,19 @@ def jquery(filename): appenv(), f"share/genenetwork2/javascript/jquery/{filename}") +@base.route("/datatables/<path:filename>") +def datatables(filename): + """Fetch DataTables files.""" + return send_from_directory( + appenv(), f"share/genenetwork2/javascript/DataTables/{filename}") + +@base.route("/datatables-extensions/<path:filename>") +def datatables_extensions(filename): + """Fetch DataTables files.""" + return send_from_directory( + appenv(), f"share/genenetwork2/javascript/DataTablesExtensions/{filename}") + + @base.route("/node-modules/<path:filename>") def node_modules(filename): """Fetch node-js modules.""" diff --git a/uploader/check_connections.py b/uploader/check_connections.py index 2561e55..c9b9aa3 100644 --- a/uploader/check_connections.py +++ b/uploader/check_connections.py @@ -4,8 +4,7 @@ import traceback import redis import MySQLdb - -from uploader.db_utils import database_connection +from gn_libs.mysqldb import database_connection def check_redis(uri: str): "Check the redis connection" diff --git a/uploader/datautils.py b/uploader/datautils.py index b95a9e0..46a55c4 100644 --- a/uploader/datautils.py +++ b/uploader/datautils.py @@ -1,6 +1,13 @@ """Generic data utilities: Rename module.""" import math from functools import reduce +from typing import Union, Sequence + +def enumerate_sequence(seq: Sequence[dict], start:int = 1) -> Sequence[dict]: + """Enumerate sequence beginning at 1""" + return tuple({**item, "sequence_number": seqno} + for seqno, item in enumerate(seq, start=start)) + def order_by_family(items: tuple[dict, ...], family_key: str = "Family", @@ -19,3 +26,13 @@ def order_by_family(items: tuple[dict, ...], return sorted(tuple(reduce(__order__, items, {}).items()), key=lambda item: item[0][0]) + + +def safe_int(val: Union[str, int, float]) -> int: + """ + Convert val into an integer: if val cannot be converted, return a zero. + """ + try: + return int(val) + except ValueError: + return 0 diff --git a/uploader/db/platforms.py b/uploader/db/platforms.py deleted file mode 100644 index cb527a7..0000000 --- a/uploader/db/platforms.py +++ /dev/null @@ -1,25 +0,0 @@ -"""Handle db interactions for platforms.""" -from typing import Optional - -import MySQLdb as mdb -from MySQLdb.cursors import DictCursor - -def platforms_by_species( - conn: mdb.Connection, speciesid: int) -> tuple[dict, ...]: - """Retrieve platforms by the species""" - with conn.cursor(cursorclass=DictCursor) as cursor: - cursor.execute("SELECT * FROM GeneChip WHERE SpeciesId=%s " - "ORDER BY GeneChipName ASC", - (speciesid,)) - return tuple(dict(row) for row in cursor.fetchall()) - -def platform_by_id(conn: mdb.Connection, platformid: int) -> Optional[dict]: - """Retrieve a platform by its ID""" - with conn.cursor(cursorclass=DictCursor) as cursor: - cursor.execute("SELECT * FROM GeneChip WHERE Id=%s", - (platformid,)) - result = cursor.fetchone() - if bool(result): - return dict(result) - - return None diff --git a/uploader/db_utils.py b/uploader/db_utils.py index d31e2c2..d9d521e 100644 --- a/uploader/db_utils.py +++ b/uploader/db_utils.py @@ -1,54 +1,20 @@ """module contains all db related stuff""" -import logging -import traceback -import contextlib -from urllib.parse import urlparse -from typing import Any, Tuple, Iterator, Callable +from typing import Any, Callable import MySQLdb as mdb from redis import Redis -from MySQLdb.cursors import Cursor from flask import current_app as app +from gn_libs.mysqldb import database_connection -def parse_db_url(db_url) -> Tuple: - """ - Parse SQL_URI configuration variable. - """ - parsed_db = urlparse(db_url) - return (parsed_db.hostname, parsed_db.username, - parsed_db.password, parsed_db.path[1:], parsed_db.port) - - -@contextlib.contextmanager -def database_connection(db_url: str) -> Iterator[mdb.Connection]: - """function to create db connector""" - host, user, passwd, db_name, db_port = parse_db_url(db_url) - connection = mdb.connect( - host, user, passwd, db_name, port=(db_port or 3306)) - try: - yield connection - connection.commit() - except mdb.Error as _mdb_err: - logging.error(traceback.format_exc()) - connection.rollback() - finally: - connection.close() def with_db_connection(func: Callable[[mdb.Connection], Any]) -> Any: """Call `func` with a MySQDdb database connection.""" with database_connection(app.config["SQL_URI"]) as conn: return func(conn) + def with_redis_connection(func: Callable[[Redis], Any]) -> Any: """Call `func` with a redis connection.""" redisuri = app.config["REDIS_URL"] with Redis.from_url(redisuri, decode_responses=True) as rconn: return func(rconn) - - -def debug_query(cursor: Cursor): - """Debug the actual query run with MySQLdb""" - for attr in ("_executed", "statement", "_last_executed"): - if hasattr(cursor, attr): - logging.debug("MySQLdb QUERY: %s", getattr(cursor, attr)) - break diff --git a/uploader/default_settings.py b/uploader/default_settings.py index 26fe665..1136ff8 100644 --- a/uploader/default_settings.py +++ b/uploader/default_settings.py @@ -2,22 +2,31 @@ The default configuration file. The values here should be overridden in the actual configuration file used for the production and staging systems. """ +import hashlib -import os - -LOG_LEVEL = os.getenv("LOG_LEVEL", "WARNING") +LOG_LEVEL = "WARNING" SECRET_KEY = b"<Please! Please! Please! Change This!>" UPLOAD_FOLDER = "/tmp/qc_app_files" +TEMPORARY_DIRECTORY = "/tmp/gn-uploader-tmpdir" REDIS_URL = "redis://" JOBS_TTL_SECONDS = 1209600 # 14 days -GNQC_REDIS_PREFIX="GNQC" +GNQC_REDIS_PREFIX="gn-uploader" SQL_URI = "" GN2_SERVER_URL = "https://genenetwork.org/" -SESSION_TYPE = "redis" SESSION_PERMANENT = True SESSION_USE_SIGNER = True +SESSION_TYPE = "cachelib" +## --- Settings for CacheLib session type --- ## +## --- These are on flask-session config variables --- ## +## --- https://cachelib.readthedocs.io/en/stable/file/ --- ## +SESSION_FILESYSTEM_CACHE_PATH = "./flask_session" +SESSION_FILESYSTEM_CACHE_THRESHOLD = 500 +SESSION_FILESYSTEM_CACHE_TIMEOUT = 300 +SESSION_FILESYSTEM_CACHE_MODE = 0o600 +SESSION_FILESYSTEM_CACHE_HASH_METHOD = hashlib.md5 +## --- END: Settings for CacheLib session type --- ## JWKS_ROTATION_AGE_DAYS = 7 # Days (from creation) to keep a JWK in use. JWKS_DELETION_AGE_DAYS = 14 # Days (from creation) to keep a JWK around before deleting it. diff --git a/uploader/expression_data/__init__.py b/uploader/expression_data/__init__.py index 206a764..fc8bd41 100644 --- a/uploader/expression_data/__init__.py +++ b/uploader/expression_data/__init__.py @@ -1,11 +1,2 @@ """Package handling upload of files.""" -from flask import Blueprint - -from .rqtl2 import rqtl2 -from .index import indexbp -from .parse import parsebp - -exprdatabp = Blueprint("expression-data", __name__) -exprdatabp.register_blueprint(indexbp, url_prefix="/") -exprdatabp.register_blueprint(rqtl2, url_prefix="/rqtl2") -exprdatabp.register_blueprint(parsebp, url_prefix="/parse") +from .views import exprdatabp diff --git a/uploader/dbinsert.py b/uploader/expression_data/dbinsert.py index 2116031..6d8ce80 100644 --- a/uploader/dbinsert.py +++ b/uploader/expression_data/dbinsert.py @@ -7,16 +7,17 @@ from datetime import datetime from redis import Redis from MySQLdb.cursors import DictCursor +from gn_libs.mysqldb import database_connection from flask import ( flash, request, url_for, Blueprint, redirect, render_template, current_app as app) +from uploader import jobs from uploader.authorisation import require_login +from uploader.db_utils import with_db_connection from uploader.population.models import populations_by_species -from uploader.db_utils import with_db_connection, database_connection -from uploader.species.models import species_by_id, all_species as species - -from . import jobs +from uploader.species.models import all_species, species_by_id +from uploader.platforms.models import platform_by_species_and_id dbinsertbp = Blueprint("dbinsert", __name__) @@ -49,14 +50,6 @@ def genechips(): return {} -def platform_by_id(genechipid:int) -> Union[dict, None]: - "Retrieve the gene platform by id" - with database_connection(app.config["SQL_URI"]) as conn: - with conn.cursor(cursorclass=DictCursor) as cursor: - cursor.execute( - "SELECT * FROM GeneChip WHERE GeneChipId=%s", - (genechipid,)) - return cursor.fetchone() def studies_by_species_and_platform(speciesid:int, genechipid:int) -> tuple: "Retrieve the studies by the related species and gene platform" @@ -108,7 +101,7 @@ def select_platform(): return render_template( "select_platform.html", filename=filename, filetype=job["filetype"], totallines=int(job["currentline"]), - default_species=default_species, species=species(conn), + default_species=default_species, species=all_species(conn), genechips=gchips[default_species], genechips_data=json.dumps(gchips)) return render_error(f"File '{filename}' no longer exists.") @@ -327,37 +320,38 @@ def selected_keys(original: dict, keys: tuple) -> dict: @require_login def final_confirmation(): "Preview the data before triggering entry into the database" - form = request.form - try: - assert form.get("filename"), "filename" - assert form.get("filetype"), "filetype" - assert form.get("species"), "species" - assert form.get("genechipid"), "platform" - assert form.get("studyid"), "study" - assert form.get("datasetid"), "dataset" - - speciesid = form["species"] - genechipid = form["genechipid"] - studyid = form["studyid"] - datasetid=form["datasetid"] - return render_template( - "final_confirmation.html", filename=form["filename"], - filetype=form["filetype"], totallines=form["totallines"], - species=speciesid, genechipid=genechipid, studyid=studyid, - datasetid=datasetid, the_species=selected_keys( - with_db_connection(lambda conn: species_by_id(conn, speciesid)), - ("SpeciesName", "Name", "MenuName")), - platform=selected_keys( - platform_by_id(genechipid), - ("GeneChipName", "Name", "GeoPlatform", "Title", "GO_tree_value")), - study=selected_keys( - study_by_id(studyid), ("Name", "FullName", "ShortName")), - dataset=selected_keys( - dataset_by_id(datasetid), - ("AvgMethodName", "Name", "Name2", "FullName", "ShortName", - "DataScale"))) - except AssertionError as aserr: - return render_error(f"Missing data: {aserr.args[0]}") + with database_connection(app.config["SQL_URI"]) as conn: + form = request.form + try: + assert form.get("filename"), "filename" + assert form.get("filetype"), "filetype" + assert form.get("species"), "species" + assert form.get("genechipid"), "platform" + assert form.get("studyid"), "study" + assert form.get("datasetid"), "dataset" + + speciesid = form["species"] + genechipid = form["genechipid"] + studyid = form["studyid"] + datasetid=form["datasetid"] + return render_template( + "final_confirmation.html", filename=form["filename"], + filetype=form["filetype"], totallines=form["totallines"], + species=speciesid, genechipid=genechipid, studyid=studyid, + datasetid=datasetid, the_species=selected_keys( + with_db_connection(lambda conn: species_by_id(conn, speciesid)), + ("SpeciesName", "Name", "MenuName")), + platform=selected_keys( + platform_by_species_and_id(conn, speciesid, genechipid), + ("GeneChipName", "Name", "GeoPlatform", "Title", "GO_tree_value")), + study=selected_keys( + study_by_id(studyid), ("Name", "FullName", "ShortName")), + dataset=selected_keys( + dataset_by_id(datasetid), + ("AvgMethodName", "Name", "Name2", "FullName", "ShortName", + "DataScale"))) + except AssertionError as aserr: + return render_error(f"Missing data: {aserr.args[0]}") @dbinsertbp.route("/insert-data", methods=["POST"]) @require_login diff --git a/uploader/expression_data/index.py b/uploader/expression_data/index.py deleted file mode 100644 index db23136..0000000 --- a/uploader/expression_data/index.py +++ /dev/null @@ -1,125 +0,0 @@ -"""Entry-point module""" -import os -import mimetypes -from typing import Tuple -from zipfile import ZipFile, is_zipfile - -from werkzeug.utils import secure_filename -from flask import ( - flash, - request, - url_for, - redirect, - Blueprint, - render_template, - current_app as app) - -from uploader.species.models import all_species as species -from uploader.authorisation import require_login -from uploader.db_utils import with_db_connection - -indexbp = Blueprint("index", __name__) - - -def errors(rqst) -> Tuple[str, ...]: - """Return a tuple of the errors found in the request `rqst`. If no error is - found, then an empty tuple is returned.""" - def __filetype_error__(): - return ( - ("Invalid file type provided.",) - if rqst.form.get("filetype") not in ("average", "standard-error") - else tuple()) - - def __file_missing_error__(): - return ( - ("No file was uploaded.",) - if ("qc_text_file" not in rqst.files or - rqst.files["qc_text_file"].filename == "") - else tuple()) - - def __file_mimetype_error__(): - text_file = rqst.files["qc_text_file"] - return ( - ( - ("Invalid file! Expected a tab-separated-values file, or a zip " - "file of the a tab-separated-values file."),) - if text_file.mimetype not in ( - "text/plain", "text/tab-separated-values", - "application/zip") - else tuple()) - - return ( - __filetype_error__() + - (__file_missing_error__() or __file_mimetype_error__())) - -def zip_file_errors(filepath, upload_dir) -> Tuple[str, ...]: - """Check the uploaded zip file for errors.""" - zfile_errors: Tuple[str, ...] = tuple() - if is_zipfile(filepath): - with ZipFile(filepath, "r") as zfile: - infolist = zfile.infolist() - if len(infolist) != 1: - zfile_errors = zfile_errors + ( - ("Expected exactly one (1) member file within the uploaded zip " - f"file. Got {len(infolist)} member files."),) - if len(infolist) == 1 and infolist[0].is_dir(): - zfile_errors = zfile_errors + ( - ("Expected a member text file in the uploaded zip file. Got a " - "directory/folder."),) - - if len(infolist) == 1 and not infolist[0].is_dir(): - zfile.extract(infolist[0], path=upload_dir) - mime = mimetypes.guess_type(f"{upload_dir}/{infolist[0].filename}") - if mime[0] != "text/tab-separated-values": - zfile_errors = zfile_errors + ( - ("Expected the member text file in the uploaded zip file to" - " be a tab-separated file."),) - - return zfile_errors - - -@indexbp.route("/", methods=["GET"]) -@require_login -def index(): - """Display the expression data index page.""" - return render_template("expression-data/index.html") - - -@indexbp.route("/upload", methods=["GET", "POST"]) -@require_login -def upload_file(): - """Enables uploading the files""" - if request.method == "GET": - return render_template( - "select_species.html", species=with_db_connection(species)) - - upload_dir = app.config["UPLOAD_FOLDER"] - request_errors = errors(request) - if request_errors: - for error in request_errors: - flash(error, "alert-danger error-expr-data") - return redirect(url_for("expression-data.index.upload_file")) - - filename = secure_filename(request.files["qc_text_file"].filename) - if not os.path.exists(upload_dir): - os.mkdir(upload_dir) - - filepath = os.path.join(upload_dir, filename) - request.files["qc_text_file"].save(os.path.join(upload_dir, filename)) - - zip_errors = zip_file_errors(filepath, upload_dir) - if zip_errors: - for error in zip_errors: - flash(error, "alert-danger error-expr-data") - return redirect(url_for("expression-data.index.upload_file")) - - return redirect(url_for("expression-data.parse.parse", - speciesid=request.form["speciesid"], - filename=filename, - filetype=request.form["filetype"])) - -@indexbp.route("/data-review", methods=["GET"]) -@require_login -def data_review(): - """Provide some help on data expectations to the user.""" - return render_template("data_review.html") diff --git a/uploader/expression_data/parse.py b/uploader/expression_data/parse.py deleted file mode 100644 index fc1c3f0..0000000 --- a/uploader/expression_data/parse.py +++ /dev/null @@ -1,178 +0,0 @@ -"""File parsing module""" -import os - -import jsonpickle -from redis import Redis -from flask import flash, request, url_for, redirect, Blueprint, render_template -from flask import current_app as app - -from quality_control.errors import InvalidValue, DuplicateHeading - -from uploader import jobs -from uploader.dbinsert import species_by_id -from uploader.db_utils import with_db_connection -from uploader.authorisation import require_login - -parsebp = Blueprint("parse", __name__) - -def isinvalidvalue(item): - """Check whether item is of type InvalidValue""" - return isinstance(item, InvalidValue) - -def isduplicateheading(item): - """Check whether item is of type DuplicateHeading""" - return isinstance(item, DuplicateHeading) - -@parsebp.route("/parse", methods=["GET"]) -@require_login -def parse(): - """Trigger file parsing""" - errors = False - speciesid = request.args.get("speciesid") - filename = request.args.get("filename") - filetype = request.args.get("filetype") - if speciesid is None: - flash("No species selected", "alert-error error-expr-data") - errors = True - else: - try: - speciesid = int(speciesid) - species = with_db_connection( - lambda con: species_by_id(con, speciesid)) - if not bool(species): - flash("No such species.", "alert-error error-expr-data") - errors = True - except ValueError: - flash("Invalid speciesid provided. Expected an integer.", - "alert-error error-expr-data") - errors = True - - if filename is None: - flash("No file provided", "alert-error error-expr-data") - errors = True - - if filetype is None: - flash("No filetype provided", "alert-error error-expr-data") - errors = True - - if filetype not in ("average", "standard-error"): - flash("Invalid filetype provided", "alert-error error-expr-data") - errors = True - - if filename: - filepath = os.path.join(app.config["UPLOAD_FOLDER"], filename) - if not os.path.exists(filepath): - flash("Selected file does not exist (any longer)", - "alert-error error-expr-data") - errors = True - - if errors: - return redirect(url_for("expression-data.index.upload_file")) - - redisurl = app.config["REDIS_URL"] - with Redis.from_url(redisurl, decode_responses=True) as rconn: - job = jobs.launch_job( - jobs.build_file_verification_job( - rconn, app.config["SQL_URI"], redisurl, - speciesid, filepath, filetype, - app.config["JOBS_TTL_SECONDS"]), - redisurl, - f"{app.config['UPLOAD_FOLDER']}/job_errors") - - return redirect(url_for("expression-data.parse.parse_status", job_id=job["jobid"])) - -@parsebp.route("/status/<job_id>", methods=["GET"]) -def parse_status(job_id: str): - "Retrieve the status of the job" - with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: - try: - job = jobs.job(rconn, jobs.jobsnamespace(), job_id) - except jobs.JobNotFound as _exc: - return render_template("no_such_job.html", job_id=job_id), 400 - - error_filename = jobs.error_filename( - job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors") - if os.path.exists(error_filename): - stat = os.stat(error_filename) - if stat.st_size > 0: - return redirect(url_for("parse.fail", job_id=job_id)) - - job_id = job["jobid"] - progress = float(job["percent"]) - status = job["status"] - filename = job.get("filename", "uploaded file") - errors = jsonpickle.decode( - job.get("errors", jsonpickle.encode(tuple()))) - if status in ("success", "aborted"): - return redirect(url_for("expression-data.parse.results", job_id=job_id)) - - if status == "parse-error": - return redirect(url_for("parse.fail", job_id=job_id)) - - app.jinja_env.globals.update( - isinvalidvalue=isinvalidvalue, - isduplicateheading=isduplicateheading) - return render_template( - "job_progress.html", - job_id = job_id, - job_status = status, - progress = progress, - message = job.get("message", ""), - job_name = f"Parsing '{filename}'", - errors=errors) - -@parsebp.route("/results/<job_id>", methods=["GET"]) -def results(job_id: str): - """Show results of parsing...""" - with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: - job = jobs.job(rconn, jobs.jobsnamespace(), job_id) - - if job: - filename = job["filename"] - errors = jsonpickle.decode(job.get("errors", jsonpickle.encode(tuple()))) - app.jinja_env.globals.update( - isinvalidvalue=isinvalidvalue, - isduplicateheading=isduplicateheading) - return render_template( - "parse_results.html", - errors=errors, - job_name = f"Parsing '{filename}'", - user_aborted = job.get("user_aborted"), - job_id=job["jobid"]) - - return render_template("no_such_job.html", job_id=job_id) - -@parsebp.route("/fail/<job_id>", methods=["GET"]) -def fail(job_id: str): - """Handle parsing failure""" - with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: - job = jobs.job(rconn, jobs.jobsnamespace(), job_id) - - if job: - error_filename = jobs.error_filename( - job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors") - if os.path.exists(error_filename): - stat = os.stat(error_filename) - if stat.st_size > 0: - return render_template( - "worker_failure.html", job_id=job_id) - - return render_template("parse_failure.html", job=job) - - return render_template("no_such_job.html", job_id=job_id) - -@parsebp.route("/abort", methods=["POST"]) -@require_login -def abort(): - """Handle user request to abort file processing""" - job_id = request.form["job_id"] - - with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: - job = jobs.job(rconn, jobs.jobsnamespace(), job_id) - - if job: - rconn.hset(name=jobs.job_key(jobs.jobsnamespace(), job_id), - key="user_aborted", - value=int(True)) - - return redirect(url_for("expression-data.parse.parse_status", job_id=job_id)) diff --git a/uploader/expression_data/views.py b/uploader/expression_data/views.py new file mode 100644 index 0000000..7629f3e --- /dev/null +++ b/uploader/expression_data/views.py @@ -0,0 +1,385 @@ +"""Views for expression data""" +import os +import uuid +import mimetypes +from typing import Tuple +from zipfile import ZipFile, is_zipfile + +import jsonpickle +from redis import Redis +from werkzeug.utils import secure_filename +from gn_libs.mysqldb import database_connection +from flask import (flash, + request, + url_for, + redirect, + Blueprint, + current_app as app) + +from quality_control.errors import InvalidValue, DuplicateHeading + +from uploader import jobs +from uploader.datautils import order_by_family +from uploader.ui import make_template_renderer +from uploader.authorisation import require_login +from uploader.db_utils import with_db_connection +from uploader.species.models import all_species, species_by_id +from uploader.population.models import (populations_by_species, + population_by_species_and_id) + +exprdatabp = Blueprint("expression-data", __name__) +render_template = make_template_renderer("expression-data") + +def isinvalidvalue(item): + """Check whether item is of type InvalidValue""" + return isinstance(item, InvalidValue) + + +def isduplicateheading(item): + """Check whether item is of type DuplicateHeading""" + return isinstance(item, DuplicateHeading) + + +def errors(rqst) -> Tuple[str, ...]: + """Return a tuple of the errors found in the request `rqst`. If no error is + found, then an empty tuple is returned.""" + def __filetype_error__(): + return ( + ("Invalid file type provided.",) + if rqst.form.get("filetype") not in ("average", "standard-error") + else tuple()) + + def __file_missing_error__(): + return ( + ("No file was uploaded.",) + if ("qc_text_file" not in rqst.files or + rqst.files["qc_text_file"].filename == "") + else tuple()) + + def __file_mimetype_error__(): + text_file = rqst.files["qc_text_file"] + return ( + ( + ("Invalid file! Expected a tab-separated-values file, or a zip " + "file of the a tab-separated-values file."),) + if text_file.mimetype not in ( + "text/plain", "text/tab-separated-values", + "application/zip") + else tuple()) + + return ( + __filetype_error__() + + (__file_missing_error__() or __file_mimetype_error__())) + + +def zip_file_errors(filepath, upload_dir) -> Tuple[str, ...]: + """Check the uploaded zip file for errors.""" + zfile_errors: Tuple[str, ...] = tuple() + if is_zipfile(filepath): + with ZipFile(filepath, "r") as zfile: + infolist = zfile.infolist() + if len(infolist) != 1: + zfile_errors = zfile_errors + ( + ("Expected exactly one (1) member file within the uploaded zip " + f"file. Got {len(infolist)} member files."),) + if len(infolist) == 1 and infolist[0].is_dir(): + zfile_errors = zfile_errors + ( + ("Expected a member text file in the uploaded zip file. Got a " + "directory/folder."),) + + if len(infolist) == 1 and not infolist[0].is_dir(): + zfile.extract(infolist[0], path=upload_dir) + mime = mimetypes.guess_type(f"{upload_dir}/{infolist[0].filename}") + if mime[0] != "text/tab-separated-values": + zfile_errors = zfile_errors + ( + ("Expected the member text file in the uploaded zip file to" + " be a tab-separated file."),) + + return zfile_errors + + +@exprdatabp.route("populations/expression-data", methods=["GET"]) +@require_login +def index(): + """Display the expression data index page.""" + with database_connection(app.config["SQL_URI"]) as conn: + if not bool(request.args.get("species_id")): + return render_template("expression-data/index.html", + species=order_by_family(all_species(conn)), + activelink="expression-data") + species = species_by_id(conn, request.args.get("species_id")) + if not bool(species): + flash("Could not find species selected!", "alert-danger") + return redirect(url_for("species.populations.expression-data.index")) + return redirect(url_for( + "species.populations.expression-data.select_population", + species_id=species["SpeciesId"])) + + +@exprdatabp.route("<int:species_id>/populations/expression-data/select-population", + methods=["GET"]) +@require_login +def select_population(species_id: int): + """Select the expression data's population.""" + with database_connection(app.config["SQL_URI"]) as conn: + species = species_by_id(conn, species_id) + if not bool(species): + flash("No such species!", "alert-danger") + return redirect(url_for("species.populations.expression-data.index")) + + if not bool(request.args.get("population_id")): + return render_template("expression-data/select-population.html", + species=species, + populations=order_by_family( + populations_by_species(conn, species_id), + order_key="FamilyOrder"), + activelink="expression-data") + + population = population_by_species_and_id( + conn, species_id, request.args.get("population_id")) + if not bool(population): + flash("No such population!", "alert-danger") + return redirect(url_for( + "species.populations.expression-data.select_population", + species_id=species_id)) + + return redirect(url_for("species.populations.expression-data.upload_file", + species_id=species_id, + population_id=population["Id"])) + + +@exprdatabp.route("<int:species_id>/populations/<int:population_id>/" + "expression-data/upload", + methods=["GET", "POST"]) +@require_login +def upload_file(species_id: int, population_id: int): + """Enables uploading the files""" + with database_connection(app.config["SQL_URI"]) as conn: + species = species_by_id(conn, species_id) + population = population_by_species_and_id(conn, species_id, population_id) + if request.method == "GET": + return render_template("expression-data/select-file.html", + species=species, + population=population) + + upload_dir = app.config["UPLOAD_FOLDER"] + request_errors = errors(request) + if request_errors: + for error in request_errors: + flash(error, "alert-danger error-expr-data") + return redirect(url_for("species.populations.expression-data.upload_file")) + + filename = secure_filename( + request.files["qc_text_file"].filename)# type: ignore[arg-type] + if not os.path.exists(upload_dir): + os.mkdir(upload_dir) + + filepath = os.path.join(upload_dir, filename) + request.files["qc_text_file"].save(os.path.join(upload_dir, filename)) + + zip_errors = zip_file_errors(filepath, upload_dir) + if zip_errors: + for error in zip_errors: + flash(error, "alert-danger error-expr-data") + return redirect(url_for("species.populations.expression-data.index.upload_file")) + + return redirect(url_for("species.populations.expression-data.parse_file", + species_id=species_id, + population_id=population_id, + filename=filename, + filetype=request.form["filetype"])) + + +@exprdatabp.route("/data-review", methods=["GET"]) +@require_login +def data_review(): + """Provide some help on data expectations to the user.""" + return render_template("expression-data/data-review.html") + + +@exprdatabp.route( + "<int:species_id>/populations/<int:population_id>/expression-data/parse", + methods=["GET"]) +@require_login +def parse_file(species_id: int, population_id: int): + """Trigger file parsing""" + _errors = False + filename = request.args.get("filename") + filetype = request.args.get("filetype") + + species = with_db_connection(lambda con: species_by_id(con, species_id)) + if not bool(species): + flash("No such species.", "alert-danger") + _errors = True + + if filename is None: + flash("No file provided", "alert-danger") + _errors = True + + if filetype is None: + flash("No filetype provided", "alert-danger") + _errors = True + + if filetype not in ("average", "standard-error"): + flash("Invalid filetype provided", "alert-danger") + _errors = True + + if filename: + filepath = os.path.join(app.config["UPLOAD_FOLDER"], filename) + if not os.path.exists(filepath): + flash("Selected file does not exist (any longer)", "alert-danger") + _errors = True + + if _errors: + return redirect(url_for("species.populations.expression-data.upload_file")) + + redisurl = app.config["REDIS_URL"] + with Redis.from_url(redisurl, decode_responses=True) as rconn: + job = jobs.launch_job( + jobs.build_file_verification_job( + rconn, app.config["SQL_URI"], redisurl, + species_id, filepath, filetype,# type: ignore[arg-type] + app.config["JOBS_TTL_SECONDS"]), + redisurl, + f"{app.config['UPLOAD_FOLDER']}/job_errors") + + return redirect(url_for("species.populations.expression-data.parse_status", + species_id=species_id, + population_id=population_id, + job_id=job["jobid"])) + + +@exprdatabp.route( + "<int:species_id>/populations/<int:population_id>/expression-data/parse/" + "status/<uuid:job_id>", + methods=["GET"]) +@require_login +def parse_status(species_id: int, population_id: int, job_id: str): + "Retrieve the status of the job" + with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: + try: + job = jobs.job(rconn, jobs.jobsnamespace(), job_id) + except jobs.JobNotFound as _exc: + return render_template("no_such_job.html", job_id=job_id), 400 + + error_filename = jobs.error_filename( + job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors") + if os.path.exists(error_filename): + stat = os.stat(error_filename) + if stat.st_size > 0: + return redirect(url_for("parse.fail", job_id=job_id)) + + job_id = job["jobid"] + progress = float(job["percent"]) + status = job["status"] + filename = job.get("filename", "uploaded file") + _errors = jsonpickle.decode( + job.get("errors", jsonpickle.encode(tuple()))) + if status in ("success", "aborted"): + return redirect(url_for("species.populations.expression-data.results", + species_id=species_id, + population_id=population_id, + job_id=job_id)) + + if status == "parse-error": + return redirect(url_for("species.populations.expression-data.fail", job_id=job_id)) + + app.jinja_env.globals.update( + isinvalidvalue=isinvalidvalue, + isduplicateheading=isduplicateheading) + return render_template( + "expression-data/job-progress.html", + job_id = job_id, + job_status = status, + progress = progress, + message = job.get("message", ""), + job_name = f"Parsing '{filename}'", + errors=_errors, + species=with_db_connection( + lambda conn: species_by_id(conn, species_id)), + population=with_db_connection( + lambda conn: population_by_species_and_id( + conn, species_id, population_id))) + + +@exprdatabp.route( + "<int:species_id>/populations/<int:population_id>/expression-data/parse/" + "<uuid:job_id>/results", + methods=["GET"]) +@require_login +def results(species_id: int, population_id: int, job_id: uuid.UUID): + """Show results of parsing...""" + with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: + job = jobs.job(rconn, jobs.jobsnamespace(), job_id) + + if job: + filename = job["filename"] + _errors = jsonpickle.decode(job.get("errors", jsonpickle.encode(tuple()))) + app.jinja_env.globals.update( + isinvalidvalue=isinvalidvalue, + isduplicateheading=isduplicateheading) + return render_template( + "expression-data/parse-results.html", + errors=_errors, + job_name = f"Parsing '{filename}'", + user_aborted = job.get("user_aborted"), + job_id=job["jobid"], + species=with_db_connection( + lambda conn: species_by_id(conn, species_id)), + population=with_db_connection( + lambda conn: population_by_species_and_id( + conn, species_id, population_id))) + + return render_template("expression-data/no-such-job.html", job_id=job_id) + + +@exprdatabp.route( + "<int:species_id>/populations/<int:population_id>/expression-data/parse/" + "<uuid:job_id>/fail", + methods=["GET"]) +@require_login +def fail(species_id: int, population_id: int, job_id: str): + """Handle parsing failure""" + with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: + job = jobs.job(rconn, jobs.jobsnamespace(), job_id) + + if job: + error_filename = jobs.error_filename( + job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors") + if os.path.exists(error_filename): + stat = os.stat(error_filename) + if stat.st_size > 0: + return render_template( + "worker_failure.html", job_id=job_id) + + return render_template("parse_failure.html", job=job) + + return render_template("expression-data/no-such-job.html", + **with_db_connection(lambda conn: { + "species_id": species_by_id(conn, species_id), + "population_id": population_by_species_and_id( + conn, species_id, population_id)}), + job_id=job_id) + + +@exprdatabp.route( + "<int:species_id>/populations/<int:population_id>/expression-data/parse/" + "abort", + methods=["POST"]) +@require_login +def abort(species_id: int, population_id: int): + """Handle user request to abort file processing""" + job_id = request.form["job_id"] + + with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: + job = jobs.job(rconn, jobs.jobsnamespace(), job_id) + + if job: + rconn.hset(name=jobs.job_key(jobs.jobsnamespace(), job_id), + key="user_aborted", + value=int(True)) + + return redirect(url_for("species.populations.expression-data.parse_status", + species_id=species_id, + population_id=population_id, + job_id=job_id)) diff --git a/uploader/files/__init__.py b/uploader/files/__init__.py new file mode 100644 index 0000000..53c3176 --- /dev/null +++ b/uploader/files/__init__.py @@ -0,0 +1,5 @@ +"""General files and chunks utilities.""" +from .chunks import chunked_binary_read +from .functions import (fullpath, + save_file, + sha256_digest_over_file) diff --git a/uploader/files/chunks.py b/uploader/files/chunks.py new file mode 100644 index 0000000..c4360b5 --- /dev/null +++ b/uploader/files/chunks.py @@ -0,0 +1,32 @@ +"""Functions dealing with chunking of files.""" +from pathlib import Path +from typing import Iterator + +from flask import current_app as app +from werkzeug.utils import secure_filename + + +def chunked_binary_read(filepath: Path, chunksize: int = 2048) -> Iterator: + """Read a file in binary mode in chunks.""" + with open(filepath, "rb") as inputfile: + while True: + data = inputfile.read(chunksize) + if data != b"": + yield data + continue + break + +def chunk_name(uploadfilename: str, chunkno: int) -> str: + """Generate chunk name from original filename and chunk number""" + if uploadfilename == "": + raise ValueError("Name cannot be empty!") + if chunkno < 1: + raise ValueError("Chunk number must be greater than zero") + return f"{secure_filename(uploadfilename)}_part_{chunkno:05d}" + + +def chunks_directory(uniqueidentifier: str) -> Path: + """Compute the directory where chunks are temporarily stored.""" + if uniqueidentifier == "": + raise ValueError("Unique identifier cannot be empty!") + return Path(app.config["UPLOAD_FOLDER"], f"tempdir_{uniqueidentifier}") diff --git a/uploader/files.py b/uploader/files/functions.py index b163612..7b9f06b 100644 --- a/uploader/files.py +++ b/uploader/files/functions.py @@ -2,17 +2,23 @@ import hashlib from pathlib import Path from datetime import datetime + from flask import current_app from werkzeug.utils import secure_filename from werkzeug.datastructures import FileStorage -def save_file(fileobj: FileStorage, upload_dir: Path) -> Path: +from .chunks import chunked_binary_read + +def save_file(fileobj: FileStorage, upload_dir: Path, hashed: bool = True) -> Path: """Save the uploaded file and return the path.""" assert bool(fileobj), "Invalid file object!" - hashed_name = hashlib.sha512( - f"{fileobj.filename}::{datetime.now().isoformat()}".encode("utf8") - ).hexdigest() + hashed_name = ( + hashlib.sha512( + f"{fileobj.filename}::{datetime.now().isoformat()}".encode("utf8") + ).hexdigest() + if hashed else + fileobj.filename) filename = Path(secure_filename(hashed_name)) # type: ignore[arg-type] if not upload_dir.exists(): upload_dir.mkdir() @@ -21,6 +27,16 @@ def save_file(fileobj: FileStorage, upload_dir: Path) -> Path: fileobj.save(filepath) return filepath + def fullpath(filename: str): """Get a file's full path. This makes use of `flask.current_app`.""" return Path(current_app.config["UPLOAD_FOLDER"], filename).absolute() + + +def sha256_digest_over_file(filepath: Path) -> str: + """Compute the sha256 digest over a file's contents.""" + filehash = hashlib.sha256() + for chunk in chunked_binary_read(filepath): + filehash.update(chunk) + + return filehash.hexdigest() diff --git a/uploader/files/views.py b/uploader/files/views.py new file mode 100644 index 0000000..29059c7 --- /dev/null +++ b/uploader/files/views.py @@ -0,0 +1,157 @@ +"""Module for generic files endpoints.""" +import time +import random +import traceback +from pathlib import Path + +from flask import request, jsonify, Blueprint, current_app as app + +from .chunks import chunk_name, chunks_directory + +files = Blueprint("files", __name__) + +def target_file(fileid: str) -> Path: + """Compute the full path for the target file.""" + return Path(app.config["UPLOAD_FOLDER"], fileid) + + +@files.route("/upload/resumable", methods=["GET"]) +def resumable_upload_get(): + """Used for checking whether **ALL** chunks have been uploaded.""" + fileid = request.args.get("resumableIdentifier", type=str) or "" + filename = request.args.get("resumableFilename", type=str) or "" + chunk = request.args.get("resumableChunkNumber", type=int) or 0 + if not(fileid or filename or chunk): + return jsonify({ + "message": "At least one required query parameter is missing.", + "error": "BadRequest", + "statuscode": 400 + }), 400 + + # If the complete target file exists, return 200 for all chunks. + _targetfile = target_file(fileid) + if _targetfile.exists(): + return jsonify({ + "uploaded-file": _targetfile.name, + "original-name": filename, + "chunk": chunk, + "message": "The complete file already exists.", + "statuscode": 200 + }), 200 + + if Path(chunks_directory(fileid), + chunk_name(filename, chunk)).exists(): + return jsonify({ + "chunk": chunk, + "message": f"Chunk {chunk} exists.", + "statuscode": 200 + }), 200 + + return jsonify({ + "message": f"Chunk {chunk} was not found.", + "error": "NotFound", + "statuscode": 404 + }), 404 + + +def __merge_chunks__(targetfile: Path, chunkpaths: tuple[Path, ...]) -> Path: + """Merge the chunks into a single file.""" + with open(targetfile, "ab") as _target: + for chunkfile in chunkpaths: + app.logger.error("Merging chunk: %s", chunkfile) + with open(chunkfile, "rb") as _chunkdata: + _target.write(_chunkdata.read()) + + chunkfile.unlink() # Don't use `missing_ok=True` — chunk MUST exist + # If chunk does't exist, it might indicate a race condition. Handle + # that instead. + return targetfile + + +@files.route("/upload/resumable", methods=["POST"]) +def resumable_upload_post(): + """Do the actual chunks upload here.""" + _totalchunks = request.form.get("resumableTotalChunks", type=int) or 0 + _chunk = request.form.get("resumableChunkNumber", default=1, type=int) + _uploadfilename = request.form.get( + "resumableFilename", default="", type=str) or "" + _fileid = request.form.get( + "resumableIdentifier", default="", type=str) or "" + _targetfile = target_file(_fileid) + + if _targetfile.exists(): + return jsonify({ + "uploaded-file": _targetfile.name, + "original-name": _uploadfilename, + "message": "File was uploaded successfully!", + "statuscode": 200 + }), 200 + + try: + chunks_directory(_fileid).mkdir(exist_ok=True, parents=True) + request.files["file"].save(Path(chunks_directory(_fileid), + chunk_name(_uploadfilename, _chunk))) + + # Check whether upload is complete + chunkpaths = tuple( + Path(chunks_directory(_fileid), chunk_name(_uploadfilename, _achunk)) + for _achunk in range(1, _totalchunks+1)) + if all(_file.exists() for _file in chunkpaths): + ### HACK: Break possible race condition ### + # Looks like sometimes, there are multiple threads/requests trying + # to merge one file, leading to race conditions and in some rare + # instances, actual data corruption. This hack is meant to break + # that race condition. + _delays = ( + 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151, 157, 163, + 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, + 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 293) + _lockfile = Path(chunks_directory(_fileid), "merge.lock") + while True: + time.sleep(random.choice(_delays) / 1000) + if (chunks_directory(_fileid).exists() + and not (_lockfile.exists() and _targetfile.exists())): + # merge_files and clean up chunks + _lockfile.touch() + __merge_chunks__(_targetfile, chunkpaths) + _lockfile.unlink() + chunks_directory(_fileid).rmdir() + continue + + if (_targetfile.exists() + and not ( + chunks_directory(_fileid).exists() + and _lockfile.exists())): + # merge complete + break + + # There is still a thread that's merging this file + continue + ### END: HACK: Break possible race condition ### + + if _targetfile.exists(): + return jsonify({ + "uploaded-file": _targetfile.name, + "original-name": _uploadfilename, + "message": "File was uploaded successfully!", + "statuscode": 200 + }), 200 + return jsonify({ + "uploaded-file": _targetfile.name, + "original-name": _uploadfilename, + "message": "Uploaded file is missing!", + "statuscode": 404 + }), 404 + return jsonify({ + "message": f"Chunk {int(_chunk)} uploaded successfully.", + "statuscode": 201 + }), 201 + except Exception as exc:# pylint: disable=[broad-except] + msg = "Error processing uploaded file chunks." + app.logger.error(msg, exc_info=True, stack_info=True) + return jsonify({ + "message": msg, + "error": type(exc).__name__, + "error-description": " ".join(str(arg) for arg in exc.args), + "error-trace": traceback.format_exception(exc) + }), 500 diff --git a/uploader/genotypes/__init__.py b/uploader/genotypes/__init__.py new file mode 100644 index 0000000..d0025d6 --- /dev/null +++ b/uploader/genotypes/__init__.py @@ -0,0 +1 @@ +"""The Genotypes module.""" diff --git a/uploader/genotypes/models.py b/uploader/genotypes/models.py new file mode 100644 index 0000000..4c3e634 --- /dev/null +++ b/uploader/genotypes/models.py @@ -0,0 +1,102 @@ +"""Functions for handling genotypes.""" +from typing import Optional +from datetime import datetime + +import MySQLdb as mdb +from MySQLdb.cursors import Cursor, DictCursor +from flask import current_app as app + +from gn_libs.mysqldb import debug_query + +def genocode_by_population( + conn: mdb.Connection, population_id: int) -> tuple[dict, ...]: + """Get the allele/genotype codes.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute("SELECT * FROM GenoCode WHERE InbredSetId=%s", + (population_id,)) + return tuple(dict(item) for item in cursor.fetchall()) + + +def genotype_markers_count(conn: mdb.Connection, species_id: int) -> int: + """Find the total count of the genotype markers for a species.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT COUNT(Name) AS markers_count FROM Geno WHERE SpeciesId=%s", + (species_id,)) + return int(cursor.fetchone()["markers_count"]) + + +def genotype_markers( + conn: mdb.Connection, + species_id: int, + offset: int = 0, + limit: Optional[int] = None +) -> tuple[dict, ...]: + """Retrieve markers from the database.""" + _query = "SELECT * FROM Geno WHERE SpeciesId=%s" + if bool(limit) and limit > 0:# type: ignore[operator] + _query = _query + f" LIMIT {limit} OFFSET {offset}" + + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_query, (species_id,)) + debug_query(cursor, app.logger) + return tuple(dict(row) for row in cursor.fetchall()) + + +def genotype_dataset( + conn: mdb.Connection, + species_id: int, + population_id: int, + dataset_id: Optional[int] = None +) -> Optional[dict]: + """Retrieve genotype datasets from the database. + + Apparently, you should only ever have one genotype dataset for a population. + """ + _query = ( + "SELECT gf.* FROM Species AS s INNER JOIN InbredSet AS iset " + "ON s.Id=iset.SpeciesId INNER JOIN GenoFreeze AS gf " + "ON iset.Id=gf.InbredSetId " + "WHERE s.Id=%s AND iset.Id=%s") + _params = (species_id, population_id) + if bool(dataset_id): + _query = _query + " AND gf.Id=%s" + _params = _params + (dataset_id,)# type: ignore[assignment] + + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_query, _params) + debug_query(cursor, app.logger) + result = cursor.fetchone() + if bool(result): + return dict(result) + return None + + +def save_new_dataset( + cursor: Cursor, + population_id: int, + name: str, + fullname: str, + shortname: str +) -> dict: + """Save a new genotype dataset into the database.""" + params = { + "InbredSetId": population_id, + "Name": name, + "FullName": fullname, + "ShortName": shortname, + "CreateTime": datetime.now().date().isoformat(), + "public": 2, + "confidentiality": 0, + "AuthorisedUsers": None + } + cursor.execute( + "INSERT INTO GenoFreeze(" + "Name, FullName, ShortName, CreateTime, public, InbredSetId, " + "confidentiality, AuthorisedUsers" + ") VALUES (" + "%(Name)s, %(FullName)s, %(ShortName)s, %(CreateTime)s, %(public)s, " + "%(InbredSetId)s, %(confidentiality)s, %(AuthorisedUsers)s" + ")", + params) + return {**params, "Id": cursor.lastrowid} diff --git a/uploader/genotypes/views.py b/uploader/genotypes/views.py new file mode 100644 index 0000000..54c2444 --- /dev/null +++ b/uploader/genotypes/views.py @@ -0,0 +1,207 @@ +"""Views for the genotypes.""" +from MySQLdb.cursors import DictCursor +from gn_libs.mysqldb import database_connection +from flask import (flash, + request, + url_for, + redirect, + Blueprint, + render_template, + current_app as app) + +from uploader.ui import make_template_renderer +from uploader.oauth2.client import oauth2_post +from uploader.authorisation import require_login +from uploader.route_utils import generic_select_population +from uploader.datautils import safe_int, enumerate_sequence +from uploader.species.models import all_species, species_by_id +from uploader.monadic_requests import make_either_error_handler +from uploader.request_checks import with_species, with_population +from uploader.population.models import population_by_species_and_id + +from .models import (genotype_markers, + genotype_dataset, + save_new_dataset, + genotype_markers_count, + genocode_by_population) + +genotypesbp = Blueprint("genotypes", __name__) +render_template = make_template_renderer("genotypes") + +@genotypesbp.route("populations/genotypes", methods=["GET"]) +@require_login +def index(): + """Direct entry-point for genotypes.""" + with database_connection(app.config["SQL_URI"]) as conn: + if not bool(request.args.get("species_id")): + return render_template("genotypes/index.html", + species=all_species(conn), + activelink="genotypes") + + species_id = request.args.get("species_id") + if species_id == "CREATE-SPECIES": + return redirect(url_for( + "species.create_species", + return_to="species.populations.genotypes.select_population")) + + species = species_by_id(conn, request.args.get("species_id")) + if not bool(species): + flash(f"Could not find species with ID '{request.args.get('species_id')}'!", + "alert-danger") + return redirect(url_for("species.populations.genotypes.index")) + return redirect(url_for("species.populations.genotypes.select_population", + species_id=species["SpeciesId"])) + + +@genotypesbp.route("/<int:species_id>/populations/genotypes/select-population", + methods=["GET"]) +@require_login +@with_species(redirect_uri="species.populations.genotypes.index") +def select_population(species: dict, species_id: int):# pylint: disable=[unused-argument] + """Select the population under which the genotypes go.""" + return generic_select_population( + species, + "genotypes/select-population.html", + request.args.get("population_id") or "", + "species.populations.genotypes.select_population", + "species.populations.genotypes.list_genotypes", + "genotypes", + "Invalid population selected!") + + +@genotypesbp.route( + "/<int:species_id>/populations/<int:population_id>/genotypes", + methods=["GET"]) +@require_login +@with_population(species_redirect_uri="species.populations.genotypes.index", + redirect_uri="species.populations.genotypes.select_population") +def list_genotypes(species: dict, population: dict, **kwargs):# pylint: disable=[unused-argument] + """List genotype details for species and population.""" + with database_connection(app.config["SQL_URI"]) as conn: + return render_template("genotypes/list-genotypes.html", + species=species, + population=population, + genocode=genocode_by_population( + conn, population["Id"]), + total_markers=genotype_markers_count( + conn, species["SpeciesId"]), + dataset=genotype_dataset(conn, + species["SpeciesId"], + population["Id"]), + activelink="list-genotypes") + + +@genotypesbp.route( + "/<int:species_id>/populations/<int:population_id>/genotypes/list-markers", + methods=["GET"]) +@require_login +@with_population(species_redirect_uri="species.populations.genotypes.index", + redirect_uri="species.populations.genotypes.select_population") +def list_markers( + species: dict, + population: dict, + **kwargs +):# pylint: disable=[unused-argument] + """List a species' genetic markers.""" + with database_connection(app.config["SQL_URI"]) as conn: + start_from = max(safe_int(request.args.get("start_from") or 0), 0) + count = safe_int(request.args.get("count") or 20) + return render_template("genotypes/list-markers.html", + species=species, + population=population, + total_markers=genotype_markers_count( + conn, species["SpeciesId"]), + start_from=start_from, + count=count, + markers=enumerate_sequence( + genotype_markers(conn, + species["SpeciesId"], + offset=start_from, + limit=count), + start=start_from+1), + activelink="list-markers") + +@genotypesbp.route( + "/<int:species_id>/populations/<int:population_id>/genotypes/datasets/" + "<int:dataset_id>/view", + methods=["GET"]) +@require_login +def view_dataset(species_id: int, population_id: int, dataset_id: int): + """View details regarding a specific dataset.""" + with database_connection(app.config["SQL_URI"]) as conn: + species = species_by_id(conn, species_id) + if not bool(species): + flash("Invalid species provided!", "alert-danger") + return redirect(url_for("species.populations.genotypes.index")) + + population = population_by_species_and_id( + conn, species_id, population_id) + if not bool(population): + flash("Invalid population selected!", "alert-danger") + return redirect(url_for( + "species.populations.genotypes.select_population", + species_id=species_id)) + + dataset = genotype_dataset(conn, species_id, population_id, dataset_id) + if not bool(dataset): + flash("Could not find such a dataset!", "alert-danger") + return redirect(url_for( + "species.populations.genotypes.list_genotypes", + species_id=species_id, + population_id=population_id)) + + return render_template("genotypes/view-dataset.html", + species=species, + population=population, + dataset=dataset, + activelink="view-dataset") + + +@genotypesbp.route( + "/<int:species_id>/populations/<int:population_id>/genotypes/datasets/" + "create", + methods=["GET", "POST"]) +@require_login +@with_population(species_redirect_uri="species.populations.genotypes.index", + redirect_uri="species.populations.genotypes.select_population") +def create_dataset(species: dict, population: dict, **kwargs):# pylint: disable=[unused-argument] + """Create a genotype dataset.""" + with (database_connection(app.config["SQL_URI"]) as conn, + conn.cursor(cursorclass=DictCursor) as cursor): + if request.method == "GET": + return render_template("genotypes/create-dataset.html", + species=species, + population=population, + activelink="create-dataset") + + form = request.form + new_dataset = save_new_dataset( + cursor, + population["Id"], + form["geno-dataset-name"], + form["geno-dataset-fullname"], + form["geno-dataset-shortname"]) + + def __success__(_success): + flash("Successfully created genotype dataset.", "alert-success") + return redirect(url_for( + "species.populations.genotypes.list_genotypes", + species_id=species["SpeciesId"], + population_id=population["Id"])) + + return oauth2_post( + "auth/resource/genotypes/create", + json={ + **dict(request.form), + "species_id": species["SpeciesId"], + "population_id": population["Id"], + "dataset_id": new_dataset["Id"], + "dataset_name": form["geno-dataset-name"], + "dataset_fullname": form["geno-dataset-fullname"], + "dataset_shortname": form["geno-dataset-shortname"], + "public": "on" + } + ).either( + make_either_error_handler( + "There was an error creating the genotype dataset."), + __success__) diff --git a/uploader/input_validation.py b/uploader/input_validation.py index 9abe742..627c69e 100644 --- a/uploader/input_validation.py +++ b/uploader/input_validation.py @@ -1,14 +1,19 @@ """Input validation utilities""" +import re +import json +import base64 from typing import Any def is_empty_string(value: str) -> bool: """Check whether as string is empty""" return (isinstance(value, str) and value.strip() == "") + def is_empty_input(value: Any) -> bool: """Check whether user provided an empty value.""" return (value is None or is_empty_string(value)) + def is_integer_input(value: Any) -> bool: """ Check whether user provided a value that can be parsed into an integer. @@ -25,3 +30,42 @@ def is_integer_input(value: Any) -> bool: __is_int__(value, 10) or __is_int__(value, 8) or __is_int__(value, 16)))) + + +def is_valid_representative_name(repr_name: str) -> bool: + """ + Check whether the given representative name is a valid according to our rules. + + Parameters + ---------- + repr_name: a string of characters. + + Checks For + ---------- + * The name MUST start with an alphabet [a-zA-Z] + * The name MUST end with an alphabet [a-zA-Z] or number [0-9] + * The name MUST be composed of alphabets [a-zA-Z], numbers [0-9], + underscores (_) and/or hyphens (-). + + Returns + ------- + Boolean indicating whether or not the name is valid. + """ + pattern = re.compile(r"^[a-zA-Z]+[a-zA-Z0-9_-]*[a-zA-Z0-9]$") + return bool(pattern.match(repr_name)) + + +def encode_errors(errors: tuple[tuple[str, str], ...], form) -> bytes: + """Encode form errors into base64 string.""" + return base64.b64encode( + json.dumps({ + "errors": dict(errors), + "original_formdata": dict(form) + }).encode("utf8")) + + +def decode_errors(errorstr) -> dict[str, dict]: + """Decode errors from base64 string""" + if not bool(errorstr): + return {"errors": {}, "original_formdata": {}} + return json.loads(base64.b64decode(errorstr.encode("utf8")).decode("utf8")) diff --git a/uploader/jobs.py b/uploader/jobs.py index 21889da..5968c03 100644 --- a/uploader/jobs.py +++ b/uploader/jobs.py @@ -1,6 +1,8 @@ """Handle jobs""" import os import sys +import uuid +import json import shlex import subprocess from uuid import UUID, uuid4 @@ -10,7 +12,9 @@ from typing import Union, Optional from redis import Redis from flask import current_app as app -JOBS_PREFIX = "JOBS" +from functional_tools import take + +JOBS_PREFIX = "jobs" class JobNotFound(Exception): """Raised if we try to retrieve a non-existent job.""" @@ -37,7 +41,8 @@ def error_filename(jobid, error_dir): "Compute the path of the file where errors will be dumped." return f"{error_dir}/job_{jobid}.error" -def initialise_job(# pylint: disable=[too-many-arguments] +def initialise_job( + # pylint: disable=[too-many-arguments, too-many-positional-arguments] rconn: Redis, rprefix: str, jobid: str, command: list, job_type: str, ttl_seconds: int = 86400, extra_meta: Optional[dict] = None) -> dict: "Initialise a job 'object' and put in on redis" @@ -50,7 +55,8 @@ def initialise_job(# pylint: disable=[too-many-arguments] name=job_key(rprefix, jobid), time=timedelta(seconds=ttl_seconds)) return the_job -def build_file_verification_job(#pylint: disable=[too-many-arguments] +def build_file_verification_job( + #pylint: disable=[too-many-arguments, too-many-positional-arguments] redis_conn: Redis, dburi: str, redisuri: str, @@ -73,7 +79,8 @@ def build_file_verification_job(#pylint: disable=[too-many-arguments] "filename": os.path.basename(filepath), "percent": 0 }) -def data_insertion_job(# pylint: disable=[too-many-arguments] +def data_insertion_job( + # pylint: disable=[too-many-arguments, too-many-positional-arguments] redis_conn: Redis, filepath: str, filetype: str, totallines: int, speciesid: int, platformid: int, datasetid: int, databaseuri: str, redisuri: str, ttl_seconds: int) -> dict: @@ -128,3 +135,33 @@ def update_stdout_stderr(rconn: Redis, contents = thejob.get(stream, '') new_contents = contents + bytes_read.decode("utf-8") rconn.hset(name=job_key(rprefix, jobid), key=stream, value=new_contents) + + +def job_errors( + rconn: Redis, + prefix: str, + job_id: Union[str, uuid.UUID], + count: int = 100 +) -> list: + """Fetch job errors""" + return take( + ( + json.loads(error) + for key in rconn.keys(f"{prefix}:{str(job_id)}:*:errors:*") + for error in rconn.lrange(key, 0, -1)), + count) + + +def job_files_metadata( + rconn: Redis, + prefix: str, + job_id: Union[str, uuid.UUID] +) -> dict: + """Get the metadata for specific job file.""" + return { + key.split(":")[-1]: { + **rconn.hgetall(key), + "filetype": key.split(":")[-3] + } + for key in rconn.keys(f"{prefix}:{str(job_id)}:*:metadata*") + } diff --git a/uploader/monadic_requests.py b/uploader/monadic_requests.py index aa34951..eda42d0 100644 --- a/uploader/monadic_requests.py +++ b/uploader/monadic_requests.py @@ -5,13 +5,12 @@ from typing import Union, Optional, Callable import requests from requests.models import Response from pymonad.either import Left, Right, Either -from flask import ( - flash, - request, - redirect, - render_template, - current_app as app, - escape as flask_escape) +from markupsafe import escape as markupsafe_escape +from flask import (flash, + request, + redirect, + render_template, + current_app as app) # HTML Status codes indicating a successful request. SUCCESS_CODES = (200, 201, 202, 203, 204, 205, 206, 207, 208, 226) @@ -40,9 +39,9 @@ def make_error_handler( trace=traceback.format_exception(resp_or_exc)) if isinstance(resp_or_exc, Response): flash("The authorisation server responded with " - f"({flask_escape(resp_or_exc.status_code)}, " - f"{flask_escape(resp_or_exc.reason)}) for the request to " - f"'{flask_escape(resp_or_exc.request.url)}'", + f"({markupsafe_escape(resp_or_exc.status_code)}, " + f"{markupsafe_escape(resp_or_exc.reason)}) for the request to " + f"'{markupsafe_escape(resp_or_exc.request.url)}'", "alert-danger") return redirect_to @@ -60,6 +59,11 @@ def get(url, params=None, **kwargs) -> Either: :rtype: pymonad.either.Either """ + timeout = kwargs.get("timeout") + kwargs = {key: val for key,val in kwargs.items() if key != "timeout"} + if timeout is None: + timeout = (9.13, 20) + try: resp = requests.get(url, params=params, **kwargs) if resp.status_code in SUCCESS_CODES: @@ -77,6 +81,11 @@ def post(url, data=None, json=None, **kwargs) -> Either: :rtype: pymonad.either.Either """ + timeout = kwargs.get("timeout") + kwargs = {key: val for key,val in kwargs.items() if key != "timeout"} + if timeout is None: + timeout = (9.13, 20) + try: resp = requests.post(url, data=data, json=json, **kwargs) if resp.status_code in SUCCESS_CODES: @@ -84,3 +93,22 @@ def post(url, data=None, json=None, **kwargs) -> Either: return Left(resp) except requests.exceptions.RequestException as exc: return Left(exc) + + +def make_either_error_handler(msg): + """Make generic error handler for pymonads Either objects.""" + def __fail__(error): + if issubclass(type(error), Exception): + app.logger.debug("\n\n%s (Exception)\n\n", msg, exc_info=True) + raise error + if issubclass(type(error), Response): + try: + _data = error.json() + except Exception as _exc: + raise Exception(error.content) from _exc# pylint: disable=[broad-exception-raised] + raise Exception(_data)# pylint: disable=[broad-exception-raised] + + app.logger.debug("\n\n%s\n\n", msg) + raise Exception(error)# pylint: disable=[broad-exception-raised] + + return __fail__ diff --git a/uploader/oauth2/client.py b/uploader/oauth2/client.py index e119cc3..12fbf80 100644 --- a/uploader/oauth2/client.py +++ b/uploader/oauth2/client.py @@ -1,6 +1,7 @@ """OAuth2 client utilities.""" import json import time +import uuid import random from datetime import datetime, timedelta from urllib.parse import urljoin, urlparse @@ -61,7 +62,7 @@ def __update_auth_server_jwks__(jwks) -> KeySet: def auth_server_jwks() -> KeySet: """Fetch the auth-server JSON Web Keys information.""" - _jwks = session.session_info().get("auth_server_jwks") + _jwks = session.session_info().get("auth_server_jwks") or {} if bool(_jwks): return __update_auth_server_jwks__({ "last-updated": _jwks["last-updated"], @@ -112,7 +113,8 @@ def oauth2_client(): try: jwt = JsonWebToken(["RS256"]).decode( token["access_token"], key=jwk) - return datetime.now().timestamp() > jwt["exp"] + if bool(jwt.get("exp")): + return datetime.now().timestamp() > jwt["exp"] except BadSignatureError as _bse: pass @@ -145,9 +147,24 @@ def oauth2_client(): __client__) +def fetch_user_details() -> Either: + """Retrieve user details from the auth server""" + suser = session.session_info()["user"] + if suser["email"] == "anon@ymous.user": + udets = oauth2_get("auth/user/").then( + lambda usrdets: session.set_user_details({ + "user_id": uuid.UUID(usrdets["user_id"]), + "name": usrdets["name"], + "email": usrdets["email"], + "token": session.user_token()})) + return udets + return Right(suser) + + def user_logged_in(): """Check whether the user has logged in.""" suser = session.session_info()["user"] + fetch_user_details() return suser["logged_in"] and suser["token"].is_right() @@ -191,7 +208,7 @@ def oauth2_get(url, **kwargs) -> Either: return Right(resp.json()) return Left(resp) except Exception as exc:#pylint: disable=[broad-except] - app.logger.error("Error retriving data from auth server: (GET %s)", + app.logger.error("Error retrieving data from auth server: (GET %s)", _uri, exc_info=True) return Left(exc) @@ -223,7 +240,7 @@ def oauth2_post(url, data=None, json=None, **kwargs):#pylint: disable=[redefined return Right(resp.json()) return Left(resp) except Exception as exc:#pylint: disable=[broad-except] - app.logger.error("Error retriving data from auth server: (POST %s)", + app.logger.error("Error retrieving data from auth server: (POST %s)", _uri, exc_info=True) return Left(exc) diff --git a/uploader/oauth2/views.py b/uploader/oauth2/views.py index 61037f3..db4ef61 100644 --- a/uploader/oauth2/views.py +++ b/uploader/oauth2/views.py @@ -24,22 +24,24 @@ from .client import ( user_logged_in, authserver_uri, oauth2_clientid, + fetch_user_details, oauth2_clientsecret) oauth2 = Blueprint("oauth2", __name__) + @oauth2.route("/code") def authorisation_code(): """Receive authorisation code from auth server and use it to get token.""" def __process_error__(resp_or_exception): app.logger.debug("ERROR: (%s)", resp_or_exception) flash("There was an error retrieving the authorisation token.", - "alert-danger") + "alert alert-danger") return redirect("/") def __fail_set_user_details__(_failure): app.logger.debug("Fetching user details fails: %s", _failure) - flash("Could not retrieve the user details", "alert-danger") + flash("Could not retrieve the user details", "alert alert-danger") return redirect("/") def __success_set_user_details__(_success): @@ -48,19 +50,13 @@ def authorisation_code(): def __success__(token): session.set_user_token(token) - return oauth2_get("auth/user/").then( - lambda usrdets: session.set_user_details({ - "user_id": uuid.UUID(usrdets["user_id"]), - "name": usrdets["name"], - "email": usrdets["email"], - "token": session.user_token(), - "logged_in": True})).either( + return fetch_user_details().either( __fail_set_user_details__, __success_set_user_details__) code = request.args.get("code", "").strip() if not bool(code): - flash("AuthorisationError: No code was provided.", "alert-danger") + flash("AuthorisationError: No code was provided.", "alert alert-danger") return redirect("/") baseurl = urlparse(request.base_url, scheme=request.scheme) @@ -116,7 +112,7 @@ def logout(): _user = session_info["user"] _user_str = f"{_user['name']} ({_user['email']})" session.clear_session_info() - flash("Successfully logged out.", "alert-success") + flash("Successfully signed out.", "alert alert-success") return redirect("/") if user_logged_in(): @@ -134,5 +130,5 @@ def logout(): cleanup_thunk=lambda: __unset_session__( session.session_info())), lambda res: __unset_session__(session.session_info())) - flash("There is no user that is currently logged in.", "alert-info") + flash("There is no user that is currently logged in.", "alert alert-info") return redirect("/") diff --git a/uploader/phenotypes/__init__.py b/uploader/phenotypes/__init__.py new file mode 100644 index 0000000..c17d32c --- /dev/null +++ b/uploader/phenotypes/__init__.py @@ -0,0 +1,2 @@ +"""Package for handling ('classical') phenotype data""" +from .views import phenotypesbp diff --git a/uploader/phenotypes/misc.py b/uploader/phenotypes/misc.py new file mode 100644 index 0000000..cbe3b7f --- /dev/null +++ b/uploader/phenotypes/misc.py @@ -0,0 +1,26 @@ +"""Miscellaneous functions handling phenotypes and phenotypes data.""" +import logging + +logger = logging.getLogger(__name__) + + +def phenotypes_data_differences( + filedata: tuple[dict, ...], dbdata: tuple[dict, ...] +) -> tuple[dict, ...]: + """Compute differences between file data and db data""" + diff = tuple() + for filerow, dbrow in zip( + sorted(filedata, key=lambda item: (item["phenotype_id"], item["xref_id"])), + sorted(dbdata, key=lambda item: (item["PhenotypeId"], item["xref_id"]))): + for samplename, value in filerow["data"].items(): + if value != dbrow["data"].get(samplename, {}).get("value"): + diff = diff + ({ + "PhenotypeId": filerow["phenotype_id"], + "xref_id": filerow["xref_id"], + "DataId": dbrow["DataId"], + "StrainId": dbrow["data"].get(samplename, {}).get("StrainId"), + "StrainName": samplename, + "value": value + },) + + return diff diff --git a/uploader/phenotypes/models.py b/uploader/phenotypes/models.py new file mode 100644 index 0000000..c2aeebf --- /dev/null +++ b/uploader/phenotypes/models.py @@ -0,0 +1,396 @@ +"""Database and utility functions for phenotypes.""" +import logging +import tempfile +from pathlib import Path +from functools import reduce +from datetime import datetime +from typing import Optional, Iterable + +import MySQLdb as mdb +from MySQLdb.cursors import Cursor, DictCursor + +from functional_tools import take +from gn_libs.mysqldb import debug_query + +logger = logging.getLogger(__name__) + + +__PHENO_DATA_TABLES__ = { + "PublishData": { + "table": "PublishData", "valueCol": "value", "DataIdCol": "Id"}, + "PublishSE": { + "table": "PublishSE", "valueCol": "error", "DataIdCol": "DataId"}, + "NStrain": { + "table": "NStrain", "valueCol": "count", "DataIdCol": "DataId"} +} + + +def datasets_by_population( + conn: mdb.Connection, + species_id: int, + population_id: int +) -> tuple[dict, ...]: + """Retrieve all of a population's phenotype studies.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT s.SpeciesId, pf.* FROM Species AS s " + "INNER JOIN InbredSet AS iset ON s.Id=iset.SpeciesId " + "INNER JOIN PublishFreeze AS pf ON iset.Id=pf.InbredSetId " + "WHERE s.Id=%s AND iset.Id=%s;", + (species_id, population_id)) + return tuple(dict(row) for row in cursor.fetchall()) + + +def dataset_by_id(conn: mdb.Connection, + species_id: int, + population_id: int, + dataset_id: int) -> dict: + """Fetch dataset details by identifier""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT Species.SpeciesId, PublishFreeze.* FROM Species " + "INNER JOIN InbredSet ON Species.Id=InbredSet.SpeciesId " + "INNER JOIN PublishFreeze ON InbredSet.Id=PublishFreeze.InbredSetId " + "WHERE Species.Id=%s AND InbredSet.Id=%s AND PublishFreeze.Id=%s", + (species_id, population_id, dataset_id)) + return dict(cursor.fetchone()) + + +def phenotypes_count(conn: mdb.Connection, + population_id: int, + dataset_id: int) -> int: + """Count the number of phenotypes in the dataset.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT COUNT(*) AS total_phenos FROM Phenotype AS pheno " + "INNER JOIN PublishXRef AS pxr ON pheno.Id=pxr.PhenotypeId " + "INNER JOIN PublishFreeze AS pf ON pxr.InbredSetId=pf.InbredSetId " + "WHERE pxr.InbredSetId=%s AND pf.Id=%s", + (population_id, dataset_id)) + return int(cursor.fetchone()["total_phenos"]) + + +def phenotype_publication_data(conn, phenotype_id) -> Optional[dict]: + """Retrieve the publication data for a phenotype if it exists.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT DISTINCT pxr.PhenotypeId, pub.* FROM PublishXRef AS pxr " + "INNER JOIN Publication as pub ON pxr.PublicationId=pub.Id " + "WHERE pxr.PhenotypeId=%s", + (phenotype_id,)) + res = cursor.fetchone() + if res is None: + return res + return dict(res) + + +def dataset_phenotypes(conn: mdb.Connection, + population_id: int, + dataset_id: int, + offset: int = 0, + limit: Optional[int] = None) -> tuple[dict, ...]: + """Fetch the actual phenotypes.""" + _query = ( + "SELECT pheno.*, pxr.Id AS xref_id, pxr.InbredSetId, ist.InbredSetCode FROM Phenotype AS pheno " + "INNER JOIN PublishXRef AS pxr ON pheno.Id=pxr.PhenotypeId " + "INNER JOIN PublishFreeze AS pf ON pxr.InbredSetId=pf.InbredSetId " + "INNER JOIN InbredSet AS ist ON pf.InbredSetId=ist.Id " + "WHERE pxr.InbredSetId=%s AND pf.Id=%s") + ( + f" LIMIT {limit} OFFSET {offset}" if bool(limit) else "") + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_query, (population_id, dataset_id)) + debug_query(cursor, logger) + return tuple(dict(row) for row in cursor.fetchall()) + + +def __phenotype_se__(cursor: Cursor, xref_id, dataids_and_strainids): + """Fetch standard-error values (if they exist) for a phenotype.""" + paramstr = ", ".join(["(%s, %s)"] * len(dataids_and_strainids)) + flat = tuple(item for sublist in dataids_and_strainids for item in sublist) + cursor.execute("SELECT * FROM PublishSE WHERE (DataId, StrainId) IN " + f"({paramstr})", + flat) + debug_query(cursor, logger) + _se = { + (row["DataId"], row["StrainId"]): { + "DataId": row["DataId"], + "StrainId": row["StrainId"], + "error": row["error"] + } + for row in cursor.fetchall() + } + + cursor.execute("SELECT * FROM NStrain WHERE (DataId, StrainId) IN " + f"({paramstr})", + flat) + debug_query(cursor, logger) + _n = { + (row["DataId"], row["StrainId"]): { + "DataId": row["DataId"], + "StrainId": row["StrainId"], + "count": row["count"] + } + for row in cursor.fetchall() + } + + keys = set(tuple(_se.keys()) + tuple(_n.keys())) + return { + key: {"xref_id": xref_id, **_se.get(key,{}), **_n.get(key,{})} + for key in keys + } + +def __organise_by_phenotype__(pheno, row): + """Organise disparate data rows into phenotype 'objects'.""" + _pheno = pheno.get(row["Id"]) + return { + **pheno, + row["Id"]: { + "Id": row["Id"], + "Pre_publication_description": row["Pre_publication_description"], + "Post_publication_description": row["Post_publication_description"], + "Original_description": row["Original_description"], + "Units": row["Units"], + "Pre_publication_abbreviation": row["Pre_publication_abbreviation"], + "Post_publication_abbreviation": row["Post_publication_abbreviation"], + "xref_id": row["pxr.Id"], + "DataId": row["DataId"], + "data": { + **(_pheno["data"] if bool(_pheno) else {}), + (row["DataId"], row["StrainId"]): { + "DataId": row["DataId"], + "StrainId": row["StrainId"], + "mean": row["mean"], + "Locus": row["Locus"], + "LRS": row["LRS"], + "additive": row["additive"], + "Sequence": row["Sequence"], + "comments": row["comments"], + "value": row["value"], + "StrainName": row["Name"], + "StrainName2": row["Name2"], + "StrainSymbol": row["Symbol"], + "StrainAlias": row["Alias"] + } + } + } + } + + +def __merge_pheno_data_and_se__(data, sedata) -> dict: + """Merge phenotype data with the standard errors.""" + return { + key: {**value, **sedata.get(key, {})} + for key, value in data.items() + } + + +def phenotype_by_id( + conn: mdb.Connection, + species_id: int, + population_id: int, + dataset_id: int, + xref_id +) -> Optional[dict]: + """Fetch a specific phenotype.""" + _dataquery = ("SELECT pheno.*, pxr.*, pd.*, str.*, iset.InbredSetCode " + "FROM Phenotype AS pheno " + "INNER JOIN PublishXRef AS pxr ON pheno.Id=pxr.PhenotypeId " + "INNER JOIN PublishData AS pd ON pxr.DataId=pd.Id " + "INNER JOIN Strain AS str ON pd.StrainId=str.Id " + "INNER JOIN StrainXRef AS sxr ON str.Id=sxr.StrainId " + "INNER JOIN PublishFreeze AS pf ON sxr.InbredSetId=pf.InbredSetId " + "INNER JOIN InbredSet AS iset ON pf.InbredSetId=iset.InbredSetId " + "WHERE " + "(str.SpeciesId, pxr.InbredSetId, pf.Id, pxr.Id)=(%s, %s, %s, %s)") + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_dataquery, + (species_id, population_id, dataset_id, xref_id)) + _pheno: dict = reduce(__organise_by_phenotype__, cursor.fetchall(), {}) + if bool(_pheno) and len(_pheno.keys()) == 1: + _pheno = tuple(_pheno.values())[0] + return { + **_pheno, + "data": tuple(__merge_pheno_data_and_se__( + _pheno["data"], + __phenotype_se__( + cursor, xref_id, tuple(_pheno["data"].keys())) + ).values()) + } + if bool(_pheno) and len(_pheno.keys()) > 1: + raise Exception( + "We found more than one phenotype with the same identifier!") + + return None + + +def phenotypes_data(conn: mdb.Connection, + population_id: int, + dataset_id: int, + offset: int = 0, + limit: Optional[int] = None) -> tuple[dict, ...]: + """Fetch the data for the phenotypes.""" + # — Phenotype -> PublishXRef -> PublishData -> Strain -> StrainXRef -> PublishFreeze + _query = ("SELECT pheno.*, pxr.*, pd.*, str.*, iset.InbredSetCode " + "FROM Phenotype AS pheno " + "INNER JOIN PublishXRef AS pxr ON pheno.Id=pxr.PhenotypeId " + "INNER JOIN PublishData AS pd ON pxr.DataId=pd.Id " + "INNER JOIN Strain AS str ON pd.StrainId=str.Id " + "INNER JOIN StrainXRef AS sxr ON str.Id=sxr.StrainId " + "INNER JOIN PublishFreeze AS pf ON sxr.InbredSetId=pf.InbredSetId " + "INNER JOIN InbredSet AS iset ON pf.InbredSetId=iset.InbredSetId " + "WHERE pxr.InbredSetId=%s AND pf.Id=%s") + ( + f" LIMIT {limit} OFFSET {offset}" if bool(limit) else "") + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_query, (population_id, dataset_id)) + debug_query(cursor, logger) + return tuple(dict(row) for row in cursor.fetchall()) + + +def save_new_dataset(cursor: Cursor, + population_id: int, + dataset_name: str, + dataset_fullname: str, + dataset_shortname: str) -> dict: + """Create a new phenotype dataset.""" + params = { + "population_id": population_id, + "dataset_name": dataset_name, + "dataset_fullname": dataset_fullname, + "dataset_shortname": dataset_shortname, + "created": datetime.now().date().isoformat(), + "public": 2, + "confidentiality": 0, + "users": None + } + cursor.execute( + "INSERT INTO PublishFreeze(Name, FullName, ShortName, CreateTime, " + "public, InbredSetId, confidentiality, AuthorisedUsers) " + "VALUES(%(dataset_name)s, %(dataset_fullname)s, %(dataset_shortname)s, " + "%(created)s, %(public)s, %(population_id)s, %(confidentiality)s, " + "%(users)s)", + params) + debug_query(cursor, logger) + return {**params, "Id": cursor.lastrowid} + + +def phenotypes_data_by_ids( + conn: mdb.Connection, + inbred_pheno_xref: dict[str, int] +) -> tuple[dict, ...]: + """Fetch all phenotype data, filtered by the `inbred_pheno_xref` mapping.""" + _paramstr = ",".join(["(%s, %s, %s)"] * len(inbred_pheno_xref)) + _query = ("SELECT " + "pub.PubMed_ID, pheno.*, pxr.*, pd.*, str.*, iset.InbredSetCode " + "FROM Publication AS pub " + "RIGHT JOIN PublishXRef AS pxr0 ON pub.Id=pxr0.PublicationId " + "INNER JOIN Phenotype AS pheno ON pxr0.PhenotypeId=pheno.id " + "INNER JOIN PublishXRef AS pxr ON pheno.Id=pxr.PhenotypeId " + "INNER JOIN PublishData AS pd ON pxr.DataId=pd.Id " + "INNER JOIN Strain AS str ON pd.StrainId=str.Id " + "INNER JOIN StrainXRef AS sxr ON str.Id=sxr.StrainId " + "INNER JOIN PublishFreeze AS pf ON sxr.InbredSetId=pf.InbredSetId " + "INNER JOIN InbredSet AS iset ON pf.InbredSetId=iset.InbredSetId " + f"WHERE (pxr.InbredSetId, pheno.Id, pxr.Id) IN ({_paramstr}) " + "ORDER BY pheno.Id") + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_query, tuple(item for row in inbred_pheno_xref + for item in (row["population_id"], + row["phenoid"], + row["xref_id"]))) + debug_query(cursor, logger) + return tuple( + reduce(__organise_by_phenotype__, cursor.fetchall(), {}).values()) + + +def create_new_phenotypes(conn: mdb.Connection, + phenotypes: Iterable[dict]) -> tuple[dict, ...]: + """Add entirely new phenotypes to the database.""" + _phenos = tuple() + with conn.cursor(cursorclass=DictCursor) as cursor: + while True: + batch = take(phenotypes, 1000) + if len(batch) == 0: + break + + cursor.executemany( + ("INSERT INTO " + "Phenotype(Pre_publication_description, Original_description, Units, Authorized_Users) " + "VALUES (%s, %s, %s, 'robwilliams')"), + tuple((row["id"], row["description"], row["units"]) + for row in batch)) + paramstr = ", ".join(["%s"] * len(batch)) + cursor.execute( + "SELECT * FROM Phenotype WHERE Pre_publication_description IN " + f"({paramstr})", + tuple(item["id"] for item in batch)) + _phenos = _phenos + tuple({ + "phenotype_id": row["Id"], + "id": row["Pre_publication_description"], + "description": row["Original_description"], + "units": row["Units"] + } for row in cursor.fetchall()) + + return _phenos + + +def save_phenotypes_data( + conn: mdb.Connection, + table: str, + data: Iterable[dict] +) -> int: + """Save new phenotypes data into the database.""" + _table_details = __PHENO_DATA_TABLES__[table] + with conn.cursor(cursorclass=DictCursor) as cursor: + _count = 0 + while True: + batch = take(data, 100000) + if len(batch) == 0: + logger.warning("Got an empty batch. This needs investigation.") + break + + logger.debug("Saving batch of %s items.", len(batch)) + cursor.executemany( + (f"INSERT INTO {_table_details['table']}" + f"({_table_details['DataIdCol']}, StrainId, {_table_details['valueCol']}) " + "VALUES " + f"(%(data_id)s, %(sample_id)s, %(value)s) "), + tuple(batch)) + debug_query(cursor, logger) + _count = _count + len(batch) + + + logger.debug("Saved a total of %s data rows", _count) + return _count + + +def quick_save_phenotypes_data( + conn: mdb.Connection, + table: str, + dataitems: Iterable[dict], + tmpdir: Path +) -> int: + """Save data items to the database, but using """ + _table_details = __PHENO_DATA_TABLES__[table] + with (tempfile.NamedTemporaryFile( + prefix=f"{table}_data", mode="wt", dir=tmpdir) as tmpfile, + conn.cursor(cursorclass=DictCursor) as cursor): + _count = 0 + logger.debug("Write data rows to text file.") + for row in dataitems: + tmpfile.write( + f'{row["data_id"]}\t{row["sample_id"]}\t{row["value"]}\n') + _count = _count + 1 + tmpfile.flush() + + logger.debug("Load text file into database (table: %s)", + _table_details["table"]) + cursor.execute( + f"LOAD DATA LOCAL INFILE '{tmpfile.name}' " + f"INTO TABLE {_table_details['table']} " + "(" + f"{_table_details['DataIdCol']}, " + "StrainId, " + f"{_table_details['valueCol']}" + ")") + debug_query(cursor, logger) + return _count diff --git a/uploader/phenotypes/views.py b/uploader/phenotypes/views.py new file mode 100644 index 0000000..bc15f2d --- /dev/null +++ b/uploader/phenotypes/views.py @@ -0,0 +1,1200 @@ +"""Views handling ('classical') phenotypes.""" +import sys +import csv +import uuid +import json +import logging +import tempfile +from typing import Any +from pathlib import Path +from zipfile import ZipFile +from functools import wraps, reduce +from logging import INFO, ERROR, DEBUG, FATAL, CRITICAL, WARNING +from urllib.parse import urljoin, urlparse, ParseResult, urlunparse, urlencode + +import datetime +from datetime import timedelta + +from redis import Redis +from pymonad.either import Left +from requests.models import Response +from MySQLdb.cursors import DictCursor +from werkzeug.utils import secure_filename + +from gn_libs import sqlite3 +from gn_libs import jobs as gnlibs_jobs +from gn_libs.jobs.jobs import JobNotFound +from gn_libs.mysqldb import database_connection +from gn_libs import monadic_requests as mrequests + +from authlib.jose import jwt +from flask import (flash, + request, + url_for, + jsonify, + redirect, + Blueprint, + send_file, + current_app as app) + +# from r_qtl import r_qtl2 as rqtl2 +from r_qtl import r_qtl2_qc as rqc +from r_qtl import exceptions as rqe + + +from uploader import jobs +from uploader import session +from uploader.files import save_file#, fullpath +from uploader.ui import make_template_renderer +from uploader.oauth2.client import oauth2_post +from uploader.authorisation import require_login +from uploader.oauth2 import jwks, client as oauth2client +from uploader.route_utils import generic_select_population +from uploader.datautils import safe_int, enumerate_sequence +from uploader.species.models import all_species, species_by_id +from uploader.monadic_requests import make_either_error_handler +from uploader.publications.models import fetch_publication_by_id +from uploader.request_checks import with_species, with_population +from uploader.samples.models import samples_by_species_and_population +from uploader.input_validation import (encode_errors, + decode_errors, + is_valid_representative_name) + +from .models import (dataset_by_id, + phenotype_by_id, + phenotypes_count, + save_new_dataset, + dataset_phenotypes, + datasets_by_population, + phenotypes_data_by_ids, + phenotype_publication_data) + +phenotypesbp = Blueprint("phenotypes", __name__) +render_template = make_template_renderer("phenotypes") + +_FAMILIES_WITH_SE_AND_N_ = ( + "Reference Populations (replicate average, SE, N)",) + +@phenotypesbp.route("/phenotypes", methods=["GET"]) +@require_login +def index(): + """Direct entry-point for phenotypes data handling.""" + with database_connection(app.config["SQL_URI"]) as conn: + if not bool(request.args.get("species_id")): + return render_template("phenotypes/index.html", + species=all_species(conn), + activelink="phenotypes") + + species_id = request.args.get("species_id") + if species_id == "CREATE-SPECIES": + return redirect(url_for( + "species.create_species", + return_to="species.populations.phenotypes.select_population")) + + species = species_by_id(conn, species_id) + if not bool(species): + flash("No such species!", "alert-danger") + return redirect(url_for("species.populations.phenotypes.index")) + return redirect(url_for("species.populations.phenotypes.select_population", + species_id=species["SpeciesId"])) + + +@phenotypesbp.route("<int:species_id>/phenotypes/select-population", + methods=["GET"]) +@require_login +@with_species(redirect_uri="species.populations.phenotypes.index") +def select_population(species: dict, **kwargs):# pylint: disable=[unused-argument] + """Select the population for your phenotypes.""" + return generic_select_population( + species, + "phenotypes/select-population.html", + request.args.get("population_id") or "", + "species.populations.phenotypes.select_population", + "species.populations.phenotypes.list_datasets", + "phenotypes", + "No such population found!") + + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets", + methods=["GET"]) +@require_login +@with_population(species_redirect_uri="species.populations.phenotypes.index", + redirect_uri="species.populations.phenotypes.select_population") +def list_datasets(species: dict, population: dict, **kwargs):# pylint: disable=[unused-argument] + """List available phenotype datasets.""" + with database_connection(app.config["SQL_URI"]) as conn: + datasets = datasets_by_population( + conn, species["SpeciesId"], population["Id"]) + if len(datasets) == 1: + return redirect(url_for( + "species.populations.phenotypes.view_dataset", + species_id=species["SpeciesId"], + population_id=population["Id"], + dataset_id=datasets[0]["Id"])) + return render_template("phenotypes/list-datasets.html", + species=species, + population=population, + datasets=datasets, + activelink="list-datasets") + + +def with_dataset( + species_redirect_uri: str, + population_redirect_uri: str, + redirect_uri: str +): + """Ensure the dataset actually exists.""" + def __decorator__(func): + @wraps(func) + @with_population(species_redirect_uri, population_redirect_uri) + def __with_dataset__(**kwargs): + try: + _spcid = int(kwargs["species_id"]) + _popid = int(kwargs["population_id"]) + _dsetid = int(kwargs.get("dataset_id")) + select_dataset_uri = redirect(url_for( + redirect_uri, species_id=_spcid, population_id=_popid)) + if not bool(_dsetid): + flash("You need to select a valid 'dataset_id' value.", + "alert-danger") + return select_dataset_uri + with database_connection(app.config["SQL_URI"]) as conn: + dataset = dataset_by_id(conn, _spcid, _popid, _dsetid) + if not bool(dataset): + flash("You must select a valid dataset.", + "alert-danger") + return select_dataset_uri + except ValueError as _verr: + app.logger.debug( + "Exception converting 'dataset_id' to integer: %s", + kwargs.get("dataset_id"), + exc_info=True) + flash("Expected 'dataset_id' value to be an integer." + "alert-danger") + return select_dataset_uri + return func(dataset=dataset, **kwargs) + return __with_dataset__ + return __decorator__ + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/view", + methods=["GET"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def view_dataset(# pylint: disable=[unused-argument] + species: dict, population: dict, dataset: dict, **kwargs): + """View a specific dataset""" + with database_connection(app.config["SQL_URI"]) as conn: + dataset = dataset_by_id( + conn, species["SpeciesId"], population["Id"], dataset["Id"]) + if not bool(dataset): + flash("Could not find such a phenotype dataset!", "alert-danger") + return redirect(url_for( + "species.populations.phenotypes.list_datasets", + species_id=species["SpeciesId"], + population_id=population["Id"])) + + start_at = max(safe_int(request.args.get("start_at") or 0), 0) + count = int(request.args.get("count") or 20) + return render_template("phenotypes/view-dataset.html", + species=species, + population=population, + dataset=dataset, + phenotype_count=phenotypes_count( + conn, population["Id"], dataset["Id"]), + phenotypes=enumerate_sequence( + dataset_phenotypes( + conn, + population["Id"], + dataset["Id"])), + start_from=start_at, + count=count, + activelink="view-dataset") + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/phenotype/<xref_id>", + methods=["GET"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def view_phenotype(# pylint: disable=[unused-argument] + species: dict, + population: dict, + dataset: dict, + xref_id: int, + **kwargs +): + """View an individual phenotype from the dataset.""" + def __render__(privileges): + phenotype = phenotype_by_id(conn, + species["SpeciesId"], + population["Id"], + dataset["Id"], + xref_id) + def __non_empty__(value) -> bool: + if isinstance(value, str): + return value.strip() != "" + return bool(value) + + return render_template( + "phenotypes/view-phenotype.html", + species=species, + population=population, + dataset=dataset, + xref_id=xref_id, + phenotype=phenotype, + has_se=any(bool(item.get("error")) for item in phenotype["data"]), + publish_data={ + key.replace("_", " "): val + for key,val in + (phenotype_publication_data(conn, phenotype["Id"]) or {}).items() + if (key in ("PubMed_ID", "Authors", "Title", "Journal") + and __non_empty__(val)) + }, + privileges=(privileges + ### For demo! Do not commit this part + + ("group:resource:edit-resource", + "group:resource:delete-resource",) + ### END: For demo! Do not commit this part + ), + activelink="view-phenotype") + + def __fail__(error): + if isinstance(error, Response) and error.json() == "No linked resource!": + return __render__(tuple()) + return make_either_error_handler( + "There was an error fetching the roles and privileges.")(error) + + with database_connection(app.config["SQL_URI"]) as conn: + return oauth2_post( + "/auth/resource/phenotypes/individual/linked-resource", + json={ + "species_id": species["SpeciesId"], + "population_id": population["Id"], + "dataset_id": dataset["Id"], + "xref_id": xref_id + } + ).then( + lambda resource: tuple( + privilege["privilege_id"] for role in resource["roles"] + for privilege in role["privileges"]) + ).then(__render__).either(__fail__, lambda resp: resp) + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets/create", + methods=["GET", "POST"]) +@require_login +@with_population( + species_redirect_uri="species.populations.phenotypes.index", + redirect_uri="species.populations.phenotypes.select_population") +def create_dataset(species: dict, population: dict, **kwargs):# pylint: disable=[unused-argument] + """Create a new phenotype dataset.""" + with (database_connection(app.config["SQL_URI"]) as conn, + conn.cursor(cursorclass=DictCursor) as cursor): + if request.method == "GET": + return render_template("phenotypes/create-dataset.html", + activelink="create-dataset", + species=species, + population=population, + **decode_errors( + request.args.get("error_values", ""))) + + form = request.form + _errors: tuple[tuple[str, str], ...] = tuple() + if not is_valid_representative_name( + (form.get("dataset-name") or "").strip()): + _errors = _errors + (("dataset-name", "Invalid dataset name."),) + + if not bool((form.get("dataset-fullname") or "").strip()): + _errors = _errors + (("dataset-fullname", + "You must provide a value for 'Full Name'."),) + + if bool(_errors) > 0: + return redirect(url_for( + "species.populations.phenotypes.create_dataset", + species_id=species["SpeciesId"], + population_id=population["Id"], + error_values=encode_errors(_errors, form))) + + dataset_shortname = ( + form["dataset-shortname"] or form["dataset-name"]).strip() + _pheno_dataset = save_new_dataset( + cursor, + population["Id"], + form["dataset-name"].strip(), + form["dataset-fullname"].strip(), + dataset_shortname) + return redirect(url_for("species.populations.phenotypes.list_datasets", + species_id=species["SpeciesId"], + population_id=population["Id"])) + + +def process_phenotypes_rqtl2_bundle(error_uri): + """Process phenotypes from the uploaded R/qtl2 bundle.""" + _redisuri = app.config["REDIS_URL"] + _sqluri = app.config["SQL_URI"] + try: + ## Handle huge files here... + phenobundle = save_file(request.files["phenotypes-bundle"], + Path(app.config["UPLOAD_FOLDER"])) + rqc.validate_bundle(phenobundle) + return phenobundle + except AssertionError as _aerr: + app.logger.debug("File upload error!", exc_info=True) + flash("Expected a zipped bundle of files with phenotypes' " + "information.", + "alert-danger") + return error_uri + except rqe.RQTLError as rqtlerr: + app.logger.debug("Bundle validation error!", exc_info=True) + flash("R/qtl2 Error: " + " ".join(rqtlerr.args), "alert-danger") + return error_uri + + +def process_phenotypes_individual_files(error_uri): + """Process the uploaded individual files.""" + form = request.form + cdata = { + "sep": form["file-separator"], + "comment.char": form["file-comment-character"], + "na.strings": form["file-na"].split(" "), + } + bundlepath = Path(app.config["UPLOAD_FOLDER"], + f"{str(uuid.uuid4()).replace('-', '')}.zip") + with ZipFile(bundlepath,mode="w") as zfile: + for rqtlkey, formkey in (("phenocovar", "phenotype-descriptions"), + ("pheno", "phenotype-data"), + ("phenose", "phenotype-se"), + ("phenonum", "phenotype-n")): + cdata[f"{rqtlkey}_transposed"] = ( + (form.get(f"{formkey}-transposed") or "off") == "on") + + if form.get("resumable-upload", False): + # Chunked upload of large files was used + filedata = json.loads(form[formkey]) + zfile.write( + Path(app.config["UPLOAD_FOLDER"], filedata["uploaded-file"]), + arcname=filedata["original-name"]) + cdata[rqtlkey] = cdata.get(rqtlkey, []) + [filedata["original-name"]] + else: + # TODO: Check this path: fix any bugs. + _sentfile = request.files[formkey] + if not bool(_sentfile): + flash(f"Expected file ('{formkey}') was not provided.", + "alert-danger") + return error_uri + + filepath = save_file( + _sentfile, Path(app.config["UPLOAD_FOLDER"]), hashed=False) + zfile.write( + Path(app.config["UPLOAD_FOLDER"], filepath), + arcname=filepath.name) + cdata[rqtlkey] = cdata.get(rqtlkey, []) + [filepath.name] + + + zfile.writestr("control_data.json", data=json.dumps(cdata, indent=2)) + + return bundlepath + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/add-phenotypes", + methods=["GET", "POST"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def add_phenotypes(species: dict, population: dict, dataset: dict, **kwargs):# pylint: disable=[unused-argument, too-many-locals] + """Add one or more phenotypes to the dataset.""" + use_bundle = request.args.get("use_bundle", "").lower() == "true" + add_phenos_uri = redirect(url_for( + "species.populations.phenotypes.add_phenotypes", + species_id=species["SpeciesId"], + population_id=population["Id"], + dataset_id=dataset["Id"])) + _redisuri = app.config["REDIS_URL"] + _sqluri = app.config["SQL_URI"] + with (Redis.from_url(_redisuri, decode_responses=True) as rconn, + # database_connection(_sqluri) as conn, + # conn.cursor(cursorclass=DictCursor) as cursor + ): + if request.method == "GET": + today = datetime.date.today() + return render_template( + ("phenotypes/add-phenotypes-with-rqtl2-bundle.html" + if use_bundle else "phenotypes/add-phenotypes-raw-files.html"), + species=species, + population=population, + dataset=dataset, + monthnames=( + "January", "February", "March", "April", + "May", "June", "July", "August", + "September", "October", "November", + "December"), + current_month=today.strftime("%B"), + current_year=int(today.strftime("%Y")), + families_with_se_and_n=_FAMILIES_WITH_SE_AND_N_, + use_bundle=use_bundle, + activelink="add-phenotypes") + + phenobundle = (process_phenotypes_rqtl2_bundle(add_phenos_uri) + if use_bundle else + process_phenotypes_individual_files(add_phenos_uri)) + + _jobid = uuid.uuid4() + _namespace = jobs.jobsnamespace() + _ttl_seconds = app.config["JOBS_TTL_SECONDS"] + _job = jobs.launch_job( + jobs.initialise_job( + rconn, + _namespace, + str(_jobid), + [sys.executable, "-m", "scripts.rqtl2.phenotypes_qc", _sqluri, + _redisuri, _namespace, str(_jobid), str(species["SpeciesId"]), + str(population["Id"]), + # str(dataset["Id"]), + str(phenobundle), + "--loglevel", + logging.getLevelName( + app.logger.getEffectiveLevel() + ).lower(), + "--redisexpiry", + str(_ttl_seconds)], "phenotype_qc", _ttl_seconds, + {"job-metadata": json.dumps({ + "speciesid": species["SpeciesId"], + "populationid": population["Id"], + "datasetid": dataset["Id"], + "bundle": str(phenobundle.absolute()), + **({"publicationid": request.form["publication-id"]} + if request.form.get("publication-id") else {})})}), + _redisuri, + f"{app.config['UPLOAD_FOLDER']}/job_errors") + + app.logger.debug("JOB DETAILS: %s", _job) + jobstatusuri = url_for("species.populations.phenotypes.job_status", + species_id=species["SpeciesId"], + population_id=population["Id"], + dataset_id=dataset["Id"], + job_id=str(_job["jobid"])) + return ((jsonify({ + "redirect-to": jobstatusuri, + "statuscode": 200, + "message": ("Follow the 'redirect-to' URI to see the state " + "of the quality-control job started for your " + "uploaded files.") + }), 200) + if request.form.get("resumable-upload", False) else + redirect(jobstatusuri)) + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/job/<uuid:job_id>", + methods=["GET"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def job_status( + species: dict, + population: dict, + dataset: dict, + job_id: uuid.UUID, + **kwargs +):# pylint: disable=[unused-argument] + """Retrieve current status of a particular phenotype QC job.""" + with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: + try: + job = jobs.job(rconn, jobs.jobsnamespace(), str(job_id)) + except jobs.JobNotFound as _jnf: + job = None + return render_template("phenotypes/job-status.html", + species=species, + population=population, + dataset=dataset, + job_id=job_id, + job=job, + errors=jobs.job_errors( + rconn, jobs.jobsnamespace(), job['jobid']), + metadata=jobs.job_files_metadata( + rconn, jobs.jobsnamespace(), job['jobid']), + activelink="add-phenotypes") + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/job/<uuid:job_id>/review", + methods=["GET"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def review_job_data( + species: dict, + population: dict, + dataset: dict, + job_id: uuid.UUID, + **kwargs +):# pylint: disable=[unused-argument] + """Review data one more time before entering it into the database.""" + with (Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn, + database_connection(app.config["SQL_URI"]) as conn): + try: + job = jobs.job(rconn, jobs.jobsnamespace(), str(job_id)) + except jobs.JobNotFound as _jnf: + job = None + + def __metadata_by_type__(by_type, item): + filetype = item[1]["filetype"] + return { + **by_type, + filetype: (by_type.get(filetype, tuple()) + + ({"filename": item[0], **item[1]},)) + } + metadata: dict[str, Any] = reduce( + __metadata_by_type__, + (jobs.job_files_metadata( + rconn, jobs.jobsnamespace(), job['jobid']) + if job else {}).items(), + {}) + + def __desc__(filetype): + match filetype: + case "phenocovar": + desc = "phenotypes" + case "pheno": + desc = "phenotypes data" + case "phenose": + desc = "phenotypes standard-errors" + case "phenonum": + desc = "phenotypes samples" + case _: + desc = f"unknown file type '{filetype}'." + + return desc + + def __summarise__(filetype, files): + return { + "filetype": filetype, + "number-of-files": len(files), + "total-data-rows": sum( + int(afile["linecount"]) - 1 for afile in files), + "description": __desc__(filetype) + } + + summary = { + filetype: __summarise__(filetype, meta) + for filetype,meta in metadata.items() + } + _job_metadata = json.loads(job["job-metadata"]) + return render_template("phenotypes/review-job-data.html", + species=species, + population=population, + dataset=dataset, + job_id=job_id, + job=job, + summary=summary, + publication=( + fetch_publication_by_id( + conn, int(_job_metadata["publicationid"])) + if _job_metadata.get("publicationid") + else None), + activelink="add-phenotypes") + + +def load_phenotypes_success_handler(job): + """Handle loading new phenotypes into the database successfully.""" + return redirect(url_for( + "species.populations.phenotypes.load_data_success", + species_id=job["metadata"]["species_id"], + population_id=job["metadata"]["population_id"], + dataset_id=job["metadata"]["dataset_id"], + job_id=job["job_id"])) + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/load-data-to-database", + methods=["POST"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def load_data_to_database( + species: dict, + population: dict, + dataset: dict, + **kwargs +):# pylint: disable=[unused-argument] + """Load the data from the given QC job into the database.""" + jobs_db = app.config["ASYNCHRONOUS_JOBS_SQLITE_DB"] + with (Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn, + sqlite3.connection(jobs_db) as conn): + qc_job = jobs.job(rconn, jobs.jobsnamespace(), request.form["data-qc-job-id"]) + _meta = json.loads(qc_job["job-metadata"]) + load_job_id = uuid.uuid4() + _loglevel = logging.getLevelName(app.logger.getEffectiveLevel()).lower() + command = [ + sys.executable, + "-u", + "-m", + "scripts.load_phenotypes_to_db", + app.config["SQL_URI"], + jobs_db, + str(load_job_id), + "--log-level", + _loglevel + ] + + def __handle_error__(resp): + return render_template("http-error.html", *resp.json()) + + def __handle_success__(load_job): + app.logger.debug("The phenotypes loading job: %s", load_job) + return redirect(url_for( + "background-jobs.job_status", job_id=load_job["job_id"])) + + issued = datetime.datetime.now() + jwtkey = jwks.newest_jwk_with_rotation( + jwks.jwks_directory(app, "UPLOADER_SECRETS"), + int(app.config["JWKS_ROTATION_AGE_DAYS"])) + + return mrequests.post( + urljoin(oauth2client.authserver_uri(), "auth/token"), + json={ + "grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer", + "scope": oauth2client.SCOPE, + "assertion": jwt.encode( + header={ + "alg": "RS256", + "typ": "JWT", + "kid": jwtkey.as_dict()["kid"] + }, + payload={ + "iss": str(oauth2client.oauth2_clientid()), + "sub": str(session.user_details()["user_id"]), + "aud": urljoin(oauth2client.authserver_uri(), + "auth/token"), + # TODO: Update expiry time once fix is implemented in + # auth server. + "exp": (issued + timedelta(minutes=5)).timestamp(), + "nbf": int(issued.timestamp()), + "iat": int(issued.timestamp()), + "jti": str(uuid.uuid4()) + }, + key=jwtkey).decode("utf8"), + "client_id": oauth2client.oauth2_clientid() + } + ).then( + lambda token: gnlibs_jobs.initialise_job( + conn, + load_job_id, + command, + "load-new-phenotypes-data", + extra_meta={ + "species_id": species["SpeciesId"], + "population_id": population["Id"], + "dataset_id": dataset["Id"], + "bundle_file": _meta["bundle"], + "publication_id": _meta["publicationid"], + "authserver": oauth2client.authserver_uri(), + "token": token["access_token"], + "success_handler": ( + "uploader.phenotypes.views" + ".load_phenotypes_success_handler") + }) + ).then( + lambda job: gnlibs_jobs.launch_job( + job, + jobs_db, + Path(f"{app.config['UPLOAD_FOLDER']}/job_errors"), + worker_manager="gn_libs.jobs.launcher", + loglevel=_loglevel) + ).either(__handle_error__, __handle_success__) + + +def update_phenotype_metadata(conn, metadata: dict): + """Update a phenotype's basic metadata values.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute("SELECT * FROM Phenotype WHERE Id=%(phenotype-id)s", + metadata) + res = { + **{ + _key: _val for _key,_val in { + key.lower().replace("_", "-"): value + for key, value in (cursor.fetchone() or {}).items() + }.items() + if _key in metadata.keys() + }, + "phenotype-id": metadata.get("phenotype-id") + } + if res == metadata: + return False + + cursor.execute( + "UPDATE Phenotype SET " + "Pre_publication_description=%(pre-publication-description)s, " + "Post_publication_description=%(post-publication-description)s, " + "Original_description=%(original-description)s, " + "Units=%(units)s, " + "Pre_publication_abbreviation=%(pre-publication-abbreviation)s, " + "Post_publication_abbreviation=%(post-publication-abbreviation)s " + "WHERE Id=%(phenotype-id)s", + metadata) + return cursor.rowcount + + +def update_phenotype_values(conn, values): + """Update a phenotype's data values.""" + with conn.cursor() as cursor: + cursor.executemany( + "UPDATE PublishData SET value=%(new)s " + "WHERE Id=%(data_id)s AND StrainId=%(strain_id)s", + tuple(item for item in values if item["new"] is not None)) + cursor.executemany( + "DELETE FROM PublishData " + "WHERE Id=%(data_id)s AND StrainId=%(strain_id)s", + tuple(item for item in values if item["new"] is None)) + return len(values) + return 0 + + +def update_phenotype_se(conn, serrs): + """Update a phenotype's standard-error values.""" + with conn.cursor() as cursor: + cursor.executemany( + "INSERT INTO PublishSE(DataId, StrainId, error) " + "VALUES(%(data_id)s, %(strain_id)s, %(new)s) " + "ON DUPLICATE KEY UPDATE error=VALUES(error)", + tuple(item for item in serrs if item["new"] is not None)) + cursor.executemany( + "DELETE FROM PublishSE " + "WHERE DataId=%(data_id)s AND StrainId=%(strain_id)s", + tuple(item for item in serrs if item["new"] is None)) + return len(serrs) + return 0 + + +def update_phenotype_n(conn, counts): + """Update a phenotype's strain counts.""" + with conn.cursor() as cursor: + cursor.executemany( + "INSERT INTO NStrain(DataId, StrainId, count) " + "VALUES(%(data_id)s, %(strain_id)s, %(new)s) " + "ON DUPLICATE KEY UPDATE count=VALUES(count)", + tuple(item for item in counts if item["new"] is not None)) + cursor.executemany( + "DELETE FROM NStrain " + "WHERE DataId=%(data_id)s AND StrainId=%(strain_id)s", + tuple(item for item in counts if item["new"] is None)) + return len(counts) + + return 0 + + +def update_phenotype_data(conn, data: dict): + """Update the numeric data for a phenotype.""" + def __organise_by_dataid_and_strainid__(acc, current): + _key, dataid, strainid = current[0].split("::") + _keysrc, _keytype = _key.split("-") + newkey = f"{dataid}::{strainid}" + newitem = acc.get(newkey, {}) + newitem[_keysrc] = newitem.get(_keysrc, {}) + newitem[_keysrc][_keytype] = current[1] + return {**acc, newkey: newitem} + + def __separate_items__(acc, row): + key, val = row + return ({ + **acc[0], + key: { + **val["value"], + "changed?": (not val["value"]["new"] == val["value"]["original"]) + } + }, { + **acc[1], + key: { + **val["se"], + "changed?": (not val["se"]["new"] == val["se"]["original"]) + } + },{ + **acc[2], + key: { + **val["n"], + "changed?": (not val["n"]["new"] == val["n"]["original"]) + } + }) + + values, serrs, counts = tuple( + tuple({ + "data_id": row[0].split("::")[0], + "strain_id": row[0].split("::")[1], + "new": row[1]["new"] + } for row in item) + for item in ( + filter(lambda val: val[1]["changed?"], item.items())# type: ignore[arg-type] + for item in reduce(# type: ignore[var-annotated] + __separate_items__, + reduce(__organise_by_dataid_and_strainid__, + data.items(), + {}).items(), + ({}, {}, {})))) + + return (update_phenotype_values(conn, values), + update_phenotype_se(conn, serrs), + update_phenotype_n(conn, counts)) + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/phenotype/<int:xref_id>/edit", + methods=["GET", "POST"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def edit_phenotype_data(# pylint: disable=[unused-argument] + species: dict, + population: dict, + dataset: dict, + xref_id: int, + **kwargs +): + """Edit the data for a particular phenotype.""" + def __render__(**kwargs): + processed_kwargs = { + **kwargs, + "privileges": (kwargs.get("privileges", tuple()) + ### For demo! Do not commit this part + + ("group:resource:edit-resource", + "group:resource:delete-resource",) + ### END: For demo! Do not commit this part + ) + } + return render_template( + "phenotypes/edit-phenotype.html", + species=species, + population=population, + dataset=dataset, + xref_id=xref_id, + families_with_se_and_n=_FAMILIES_WITH_SE_AND_N_, + **processed_kwargs, + activelink="edit-phenotype") + + with database_connection(app.config["SQL_URI"]) as conn: + if request.method == "GET": + def __fetch_phenotype__(privileges): + phenotype = phenotype_by_id(conn, + species["SpeciesId"], + population["Id"], + dataset["Id"], + xref_id) + if phenotype is None: + msg = ("Could not find the phenotype with cross-reference ID" + f" '{xref_id}' from dataset '{dataset['FullName']}' " + f" from the '{population['FullName']}' population of " + f" species '{species['FullName']}'.") + return Left({"privileges": privileges, "phenotype-error": msg}) + return {"privileges": privileges, "phenotype": phenotype} + + def __fetch_publication_data__(**kwargs): + pheno = kwargs["phenotype"] + return { + **kwargs, + "publication_data": phenotype_publication_data( + conn, pheno["Id"]) + } + + def __fail__(failure_object): + # process the object + return __render__(failure_object=failure_object) + + return oauth2_post( + "/auth/resource/phenotypes/individual/linked-resource", + json={ + "species_id": species["SpeciesId"], + "population_id": population["Id"], + "dataset_id": dataset["Id"], + "xref_id": xref_id + } + ).then( + lambda resource: tuple( + privilege["privilege_id"] for role in resource["roles"] + for privilege in role["privileges"]) + ).then( + __fetch_phenotype__ + ).then( + lambda args: __fetch_publication_data__(**args) + ).either(__fail__, lambda args: __render__(**args)) + + ## POST + _change = False + match request.form.get("submit", "invalid-action"): + case "update basic metadata": + _change = update_phenotype_metadata(conn, { + key: value.strip() if bool(value.strip()) else None + for key, value in request.form.items() + if key not in ("submit",) + }) + msg = "Basic metadata was updated successfully." + case "update data": + _update = update_phenotype_data(conn, { + key: value.strip() if bool(value.strip()) else None + for key, value in request.form.items() + if key not in ("submit",) + }) + msg = (f"{_update[0]} value rows, {_update[1]} standard-error " + f"rows and {_update[2]} 'N' rows were updated.") + _change = any(item != 0 for item in _update) + case "update publication": + flash("NOT IMPLEMENTED: Would update publication data.", "alert-success") + case _: + flash("Invalid phenotype editing action.", "alert-danger") + + if _change: + flash(msg, "alert-success") + return redirect(url_for( + "species.populations.phenotypes.view_phenotype", + species_id=species["SpeciesId"], + population_id=population["Id"], + dataset_id=dataset["Id"], + xref_id=xref_id)) + + flash("No change was made by the user.", "alert-info") + return redirect(url_for( + "species.populations.phenotypes.edit_phenotype_data", + species_id=species["SpeciesId"], + population_id=population["Id"], + dataset_id=dataset["Id"], + xref_id=xref_id)) + + +def process_phenotype_data_for_download(pheno: dict) -> dict: + """Sanitise data for download.""" + return { + "UniqueIdentifier": f"phId:{pheno['Id']}::xrId:{pheno['xref_id']}", + **{ + key: val for key, val in pheno.items() + if key not in ("Id", "xref_id", "data", "Units") + }, + **{ + data_item["StrainName"]: data_item["value"] + for data_item in pheno.get("data", {}).values() + } + } + + +BULK_EDIT_COMMON_FIELDNAMES = [ + "UniqueIdentifier", + "Post_publication_description", + "Pre_publication_abbreviation", + "Pre_publication_description", + "Original_description", + "Post_publication_abbreviation", + "PubMed_ID" +] + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/edit-download", + methods=["POST"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def edit_download_phenotype_data(# pylint: disable=[unused-argument] + species: dict, + population: dict, + dataset: dict, + **kwargs +): + formdata = request.json + with database_connection(app.config["SQL_URI"]) as conn: + samples_list = [ + sample["Name"] for sample in samples_by_species_and_population( + conn, species["SpeciesId"], population["Id"])] + data = ( + process_phenotype_data_for_download(pheno) + for pheno in phenotypes_data_by_ids(conn, tuple({ + "population_id": population["Id"], + "phenoid": row["phenotype_id"], + "xref_id": row["xref_id"] + } for row in formdata))) + + with (tempfile.TemporaryDirectory( + prefix=app.config["TEMPORARY_DIRECTORY"]) as tmpdir): + filename = Path(tmpdir).joinpath("tempfile.tsv") + with open(filename, mode="w") as outfile: + outfile.write( + "# **DO NOT** delete the 'UniqueIdentifier' row. It is used " + "by the system to identify and edit the correct rows and " + "columns in the database.\n") + outfile.write( + "# The '…_description' fields are useful for you to figure out " + "what row you are working on. Changing any of this fields will " + "also update the database, so do be careful.\n") + outfile.write( + "# Leave a field empty to delete the value in the database.\n") + outfile.write( + "# Any line beginning with a '#' character is considered a " + "comment line. This line, and all the lines above it, are " + "all comment lines. Comment lines will be ignored.\n") + writer = csv.DictWriter(outfile, + fieldnames= ( + BULK_EDIT_COMMON_FIELDNAMES + + samples_list), + dialect="excel-tab") + writer.writeheader() + writer.writerows(data) + outfile.flush() + + return send_file( + filename, + mimetype="text/csv", + as_attachment=True, + download_name=secure_filename(f"{dataset['Name']}_data")) + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/edit-upload", + methods=["GET", "POST"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def edit_upload_phenotype_data(# pylint: disable=[unused-argument] + species: dict, + population: dict, + dataset: dict, + **kwargs +): + if request.method == "GET": + return render_template( + "phenotypes/bulk-edit-upload.html", + species=species, + population=population, + dataset=dataset, + activelink="edit-phenotype") + + edit_file = save_file(request.files["file-upload-bulk-edit-upload"], + Path(app.config["UPLOAD_FOLDER"])) + + jobs_db = app.config["ASYNCHRONOUS_JOBS_SQLITE_DB"] + with sqlite3.connection(jobs_db) as conn: + job_id = uuid.uuid4() + job_cmd = [ + sys.executable, "-u", + "-m", "scripts.phenotypes_bulk_edit", + app.config["SQL_URI"], + jobs_db, + str(job_id), + "--log-level", + logging.getLevelName( + app.logger.getEffectiveLevel() + ).lower() + ] + app.logger.debug("Phenotype-edit, bulk-upload command: %s", job_cmd) + _job = gnlibs_jobs.launch_job( + gnlibs_jobs.initialise_job(conn, + job_id, + job_cmd, + "phenotype-bulk-edit", + extra_meta = { + "edit-file": str(edit_file), + "species-id": species["SpeciesId"], + "population-id": population["Id"], + "dataset-id": dataset["Id"] + }), + jobs_db, + f"{app.config['UPLOAD_FOLDER']}/job_errors", + worker_manager="gn_libs.jobs.launcher") + + + return redirect(url_for("background-jobs.job_status", + job_id=job_id, + job_type="phenotype-bulk-edit")) + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/load-data-success/<uuid:job_id>", + methods=["GET"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def load_data_success( + species: dict, + population: dict, + dataset: dict, + job_id: uuid.UUID, + **kwargs +):# pylint: disable=[unused-argument] + with (database_connection(app.config["SQL_URI"]) as conn, + sqlite3.connection(app.config["ASYNCHRONOUS_JOBS_SQLITE_DB"]) + as jobsconn): + try: + gn2_uri = urlparse(app.config["GN2_SERVER_URL"]) + job = gnlibs_jobs.job(jobsconn, job_id, fulldetails=True) + app.logger.debug("THE JOB: %s", job) + _xref_ids = (str(item) for item + in json.loads(job["metadata"].get("xref_ids", "[]"))) + _publication = fetch_publication_by_id( + conn, int(job["metadata"].get("publication_id", "0"))) + _search_terms = (item for item in + (str(_publication["PubMed_ID"] or ""), + _publication["Authors"], + (_publication["Title"] or "")) + if item != "") + return render_template("phenotypes/load-phenotypes-success.html", + species=species, + population=population, + dataset=dataset, + job=job, + search_page_uri=urlunparse(ParseResult( + scheme=gn2_uri.scheme, + netloc=gn2_uri.netloc, + path="/search", + params="", + query=urlencode({ + "species": species["Name"], + "group": population["Name"], + "type": "Phenotypes", + "dataset": dataset["Name"], + "search_terms_or": ( + # Very long URLs will cause + # errors. + " ".join(_xref_ids) + if len(_xref_ids) <= 100 + else ""), + "search_terms_and": " ".join( + _search_terms).strip(), + "accession_id": "None", + "FormID": "searchResult" + }), + fragment=""))) + except JobNotFound as jnf: + return render_template("jobs/job-not-found.html", job_id=job_id) diff --git a/uploader/platforms/__init__.py b/uploader/platforms/__init__.py new file mode 100644 index 0000000..8cb89c9 --- /dev/null +++ b/uploader/platforms/__init__.py @@ -0,0 +1,2 @@ +"""Module to handle management of genetic platforms.""" +from .views import platformsbp diff --git a/uploader/platforms/models.py b/uploader/platforms/models.py new file mode 100644 index 0000000..0dd9368 --- /dev/null +++ b/uploader/platforms/models.py @@ -0,0 +1,96 @@ +"""Handle db interactions for platforms.""" +from typing import Optional + +import MySQLdb as mdb +from MySQLdb.cursors import Cursor, DictCursor + +def platforms_by_species( + conn: mdb.Connection, + speciesid: int, + offset: int = 0, + limit: Optional[int] = None +) -> tuple[dict, ...]: + """Retrieve platforms by the species""" + _query = ("SELECT * FROM GeneChip WHERE SpeciesId=%s " + "ORDER BY GeneChipName ASC") + if bool(limit) and limit > 0:# type: ignore[operator] + _query = f"{_query} LIMIT {limit} OFFSET {offset}" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_query, (speciesid,)) + return tuple(dict(row) for row in cursor.fetchall()) + + +def species_platforms_count(conn: mdb.Connection, species_id: int) -> int: + """Get the number of platforms in the database for a particular species.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT COUNT(GeneChipName) AS count FROM GeneChip " + "WHERE SpeciesId=%s", + (species_id,)) + return int(cursor.fetchone()["count"]) + + +def platform_by_id(conn: mdb.Connection, platformid: int) -> Optional[dict]: + """Retrieve a platform by its ID""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute("SELECT * FROM GeneChip WHERE Id=%s", + (platformid,)) + result = cursor.fetchone() + if bool(result): + return dict(result) + + return None + + +def platform_by_species_and_id( + conn: mdb.Connection, species_id: int, platformid: int +) -> Optional[dict]: + """Retrieve a platform by its species and ID""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute("SELECT * FROM GeneChip WHERE SpeciesId=%s AND Id=%s", + (species_id, platformid)) + result = cursor.fetchone()#pylint: disable=[duplicate-code] + if bool(result): + return dict(result) + + return None + + +def save_new_platform( + # pylint: disable=[too-many-arguments, too-many-positional-arguments] + cursor: Cursor, + species_id: int, + geo_platform: str, + platform_name: str, + platform_shortname: str, + platform_title: str, + go_tree_value: Optional[str] +) -> dict: + """Save a new platform to the database.""" + params = { + "species_id": species_id, + "GeoPlatform": geo_platform, + "GeneChipName": platform_name, + "Name": platform_shortname, + "Title": platform_title, + "GO_tree_value": go_tree_value + } + cursor.execute("SELECT SpeciesId, GeoPlatform FROM GeneChip") + assert (species_id, geo_platform) not in ( + (row["SpeciesId"], row["GeoPlatform"]) for row in cursor.fetchall()) + cursor.execute( + "INSERT INTO " + "GeneChip(SpeciesId, GeneChipName, Name, GeoPlatform, Title, GO_tree_value) " + "VALUES(" + "%(species_id)s, %(GeneChipName)s, %(Name)s, %(GeoPlatform)s, " + "%(Title)s, %(GO_tree_value)s" + ")", + params) + new_id = cursor.lastrowid + cursor.execute("UPDATE GeneChip SET GeneChipId=%s WHERE Id=%s", + (new_id, new_id)) + return { + **params, + "Id": new_id, + "GeneChipId": new_id + } diff --git a/uploader/platforms/views.py b/uploader/platforms/views.py new file mode 100644 index 0000000..d12a9ef --- /dev/null +++ b/uploader/platforms/views.py @@ -0,0 +1,118 @@ +"""The endpoints for the platforms""" +from MySQLdb.cursors import DictCursor +from gn_libs.mysqldb import database_connection +from flask import ( + flash, + request, + url_for, + redirect, + Blueprint, + current_app as app) + +from uploader.ui import make_template_renderer +from uploader.authorisation import require_login +from uploader.species.models import all_species, species_by_id +from uploader.datautils import safe_int, enumerate_sequence + +from .models import (save_new_platform, + platforms_by_species, + species_platforms_count) + +platformsbp = Blueprint("platforms", __name__) +render_template = make_template_renderer("platforms") + +@platformsbp.route("platforms", methods=["GET"]) +@require_login +def index(): + """Entry-point to the platforms feature.""" + with database_connection(app.config["SQL_URI"]) as conn: + if not bool(request.args.get("species_id")): + return render_template( + "platforms/index.html", + species=all_species(conn), + activelink="platforms") + + species_id = request.args.get("species_id") + if species_id == "CREATE-SPECIES": + return redirect(url_for( + "species.create_species", + return_to="species.platforms.list_platforms")) + + species = species_by_id(conn, request.args["species_id"]) + if not bool(species): + flash("No species selected.", "alert-danger") + return redirect(url_for("species.platforms.index")) + + return redirect(url_for("species.platforms.list_platforms", + species_id=species["SpeciesId"])) + + +@platformsbp.route("<int:species_id>/platforms", methods=["GET"]) +@require_login +def list_platforms(species_id: int): + """List all the available genetic sequencing platforms.""" + with database_connection(app.config["SQL_URI"]) as conn: + species = species_by_id(conn, species_id) + if not bool(species): + flash("No species provided.", "alert-danger") + return redirect(url_for("species.platforms.index")) + + start_from = max(safe_int(request.args.get("start_from") or 0), 0) + count = safe_int(request.args.get("count") or 20) + return render_template( + "platforms/list-platforms.html", + species=species, + platforms=enumerate_sequence( + platforms_by_species(conn, + species_id, + offset=start_from, + limit=count), + start=start_from+1), + start_from=start_from, + count=count, + total_platforms=species_platforms_count(conn, species_id), + activelink="list-platforms") + + +@platformsbp.route("<int:species_id>/platforms/create", methods=["GET", "POST"]) +@require_login +def create_platform(species_id: int): + """Create a new genetic sequencing platform.""" + with (database_connection(app.config["SQL_URI"]) as conn, + conn.cursor(cursorclass=DictCursor) as cursor): + species = species_by_id(conn, species_id) + if not bool(species): + flash("No species provided.", "alert-danger") + return redirect(url_for("species.platforms.index")) + + if request.method == "GET": + return render_template( + "platforms/create-platform.html", + species=species, + activelink="create-platform") + + try: + form = request.form + _new_platform = save_new_platform( + cursor, + species_id, + form["geo-platform"], + form["platform-name"], + form["platform-shortname"], + form["platform-title"], + form.get("go-tree-value") or None) + except KeyError as _kerr: + flash(f"Required value for field {_kerr.args[0]} was not provided.", + "alert-danger") + return redirect(url_for("species.platforms.create_platform", + species_id=species_id)) + except AssertionError as _aerr: + flash(f"Platform with GeoPlatform value of '{form['geo-platform']}'" + f" already exists for species '{species['FullName']}'.", + "alert-danger") + return redirect(url_for("species.platforms.create_platform", + species_id=species_id)) + + flash("Platform created successfully", "alert-success") + return redirect(url_for("species.platforms.list_platforms", + species_id=species_id)) diff --git a/uploader/population/models.py b/uploader/population/models.py index 782bc9f..d78a821 100644 --- a/uploader/population/models.py +++ b/uploader/population/models.py @@ -44,33 +44,44 @@ def population_genetic_types(conn) -> tuple: return tuple(row["GeneticType"] for row in cursor.fetchall()) -def save_population(conn: mdb.Connection, population_details: dict) -> dict: +def save_population(cursor: mdb.cursors.Cursor, population_details: dict) -> dict: """Save the population details to the db.""" - with conn.cursor(cursorclass=DictCursor) as cursor: - #TODO: Handle FamilyOrder here - cursor.execute( - "INSERT INTO InbredSet(" - "InbredSetId, InbredSetName, Name, SpeciesId, FullName, " - "public, MappingMethodId, GeneticType, Family, MenuOrderId, " - "InbredSetCode, Description" - ") " - "VALUES (" - "%(InbredSetId)s, %(InbredSetName)s, %(Name)s, %(SpeciesId)s, " - "%(FullName)s, %(public)s, %(MappingMethodId)s, %(GeneticType)s, " - "%(Family)s, %(MenuOrderId)s, %(InbredSetCode)s, %(Description)s" - ")", - { - "MenuOrderId": 0, - "InbredSetId": 0, - "public": 2, - **population_details - }) - new_id = cursor.lastrowid - cursor.execute("UPDATE InbredSet SET InbredSetId=%s WHERE Id=%s", - (new_id, new_id)) - return { - **population_details, - "Id": new_id, - "InbredSetId": new_id, - "population_id": new_id - } + cursor.execute("SELECT DISTINCT(Family), FamilyOrder FROM InbredSet " + "WHERE Family IS NOT NULL AND Family != '' " + "AND FamilyOrder IS NOT NULL " + "ORDER BY FamilyOrder ASC") + _families = { + row["Family"]: int(row["FamilyOrder"]) + for row in cursor.fetchall() + } + params = { + "MenuOrderId": 0, + "InbredSetId": 0, + "public": 2, + **population_details, + "FamilyOrder": _families.get( + population_details["Family"], + max((0,) + tuple(_families.values()))+1) + } + cursor.execute( + "INSERT INTO InbredSet(" + "InbredSetId, InbredSetName, Name, SpeciesId, FullName, " + "public, MappingMethodId, GeneticType, Family, FamilyOrder," + " MenuOrderId, InbredSetCode, Description" + ") " + "VALUES (" + "%(InbredSetId)s, %(InbredSetName)s, %(Name)s, %(SpeciesId)s, " + "%(FullName)s, %(public)s, %(MappingMethodId)s, %(GeneticType)s, " + "%(Family)s, %(FamilyOrder)s, %(MenuOrderId)s, %(InbredSetCode)s, " + "%(Description)s" + ")", + params) + new_id = cursor.lastrowid + cursor.execute("UPDATE InbredSet SET InbredSetId=%s WHERE Id=%s", + (new_id, new_id)) + return { + **params, + "Id": new_id, + "InbredSetId": new_id, + "population_id": new_id + } diff --git a/uploader/expression_data/rqtl2.py b/uploader/population/rqtl2.py index a855699..044cdd4 100644 --- a/uploader/expression_data/rqtl2.py +++ b/uploader/population/rqtl2.py @@ -3,7 +3,6 @@ import sys import json import traceback from pathlib import Path -from datetime import date from uuid import UUID, uuid4 from functools import partial from zipfile import ZipFile, is_zipfile @@ -12,12 +11,11 @@ from typing import Union, Callable, Optional import MySQLdb as mdb from redis import Redis from MySQLdb.cursors import DictCursor -from werkzeug.utils import secure_filename +from gn_libs.mysqldb import database_connection from flask import ( flash, escape, request, - jsonify, url_for, redirect, Response, @@ -29,15 +27,14 @@ from r_qtl import r_qtl2 from uploader import jobs from uploader.files import save_file, fullpath -from uploader.dbinsert import species as all_species -from uploader.db_utils import with_db_connection, database_connection +from uploader.species.models import all_species +from uploader.db_utils import with_db_connection from uploader.authorisation import require_login -from uploader.db.platforms import platform_by_id, platforms_by_species +from uploader.platforms.models import platform_by_id, platforms_by_species from uploader.db.averaging import averaging_methods, averaging_method_by_id from uploader.db.tissues import all_tissues, tissue_by_id, create_new_tissue -from uploader.population.models import (save_population, - populations_by_species, +from uploader.population.models import (populations_by_species, population_by_species_and_id) from uploader.species.models import species_by_id from uploader.db.datasets import ( @@ -60,19 +57,21 @@ rqtl2 = Blueprint("rqtl2", __name__) def select_species(): """Select the species.""" if request.method == "GET": - return render_template("rqtl2/index.html", species=with_db_connection(all_species)) + return render_template("expression-data/rqtl2/index.html", + species=with_db_connection(all_species)) species_id = request.form.get("species_id") species = with_db_connection( lambda conn: species_by_id(conn, species_id)) if bool(species): return redirect(url_for( - "expression-data.rqtl2.select_population", species_id=species_id)) + "species.populations.expression-data.rqtl2.select_population", + species_id=species_id)) flash("Invalid species or no species selected!", "alert-error error-rqtl2") return redirect(url_for("expression-data.rqtl2.select_species")) -@rqtl2.route("/upload/species/<int:species_id>/select-population", +@rqtl2.route("<int:species_id>/expression-data/rqtl2/select-population", methods=["GET", "POST"]) @require_login def select_population(species_id: int): @@ -85,7 +84,7 @@ def select_population(species_id: int): if request.method == "GET": return render_template( - "rqtl2/select-population.html", + "expression-data/rqtl2/select-population.html", species=species, populations=populations_by_species(conn, species_id)) @@ -102,44 +101,6 @@ def select_population(species_id: int): population_id=population["InbredSetId"])) -@rqtl2.route("/upload/species/<int:species_id>/create-population", - methods=["POST"]) -@require_login -def create_population(species_id: int): - """Create a new population for the given species.""" - population_page = redirect(url_for("expression-data.rqtl2.select_population", - species_id=species_id)) - with database_connection(app.config["SQL_URI"]) as conn: - species = species_by_id(conn, species_id) - population_name = request.form.get("inbredset_name", "").strip() - population_fullname = request.form.get("inbredset_fullname", "").strip() - if not bool(species): - flash("Invalid species!", "alert-error error-rqtl2") - return redirect(url_for("expression-data.rqtl2.select_species")) - if not bool(population_name): - flash("Invalid Population Name!", "alert-error error-rqtl2") - return population_page - if not bool(population_fullname): - flash("Invalid Population Full Name!", "alert-error error-rqtl2") - return population_page - new_population = save_population(conn, { - "SpeciesId": species["SpeciesId"], - "Name": population_name, - "InbredSetName": population_fullname, - "FullName": population_fullname, - "Family": request.form.get("inbredset_family") or None, - "Description": request.form.get("description") or None - }) - - flash("Population created successfully.", "alert-success") - return redirect( - url_for("expression-data.rqtl2.upload_rqtl2_bundle", - species_id=species_id, - population_id=new_population["population_id"], - pgsrc="create-population"), - code=307) - - class __RequestError__(Exception): #pylint: disable=[invalid-name] """Internal class to avoid pylint's `too-many-return-statements` error.""" @@ -165,9 +126,10 @@ def upload_rqtl2_bundle(species_id: int, population_id: int): if request.method == "GET" or ( request.method == "POST" and bool(request.args.get("pgsrc"))): - return render_template("rqtl2/upload-rqtl2-bundle-step-01.html", - species=species, - population=population) + return render_template( + "expression-data/rqtl2/upload-rqtl2-bundle-step-01.html", + species=species, + population=population) try: app.logger.debug("Files in the form: %s", request.files) @@ -227,127 +189,6 @@ def trigger_rqtl2_bundle_qc( return jobid -def chunk_name(uploadfilename: str, chunkno: int) -> str: - """Generate chunk name from original filename and chunk number""" - if uploadfilename == "": - raise ValueError("Name cannot be empty!") - if chunkno < 1: - raise ValueError("Chunk number must be greater than zero") - return f"{secure_filename(uploadfilename)}_part_{chunkno:05d}" - - -def chunks_directory(uniqueidentifier: str) -> Path: - """Compute the directory where chunks are temporarily stored.""" - if uniqueidentifier == "": - raise ValueError("Unique identifier cannot be empty!") - return Path(app.config["UPLOAD_FOLDER"], f"tempdir_{uniqueidentifier}") - - -@rqtl2.route(("/upload/species/<int:species_id>/population/<int:population_id>" - "/rqtl2-bundle-chunked"), - methods=["GET"]) -@require_login -def upload_rqtl2_bundle_chunked_get(# pylint: disable=["unused-argument"] - species_id: int, - population_id: int -): - """ - Extension to the `upload_rqtl2_bundle` endpoint above that provides a way - for testing whether all the chunks have been uploaded and to assist with - resuming a failed expression-data. - """ - fileid = request.args.get("resumableIdentifier", type=str) or "" - filename = request.args.get("resumableFilename", type=str) or "" - chunk = request.args.get("resumableChunkNumber", type=int) or 0 - if not(fileid or filename or chunk): - return jsonify({ - "message": "At least one required query parameter is missing.", - "error": "BadRequest", - "statuscode": 400 - }), 400 - - if Path(chunks_directory(fileid), - chunk_name(filename, chunk)).exists(): - return "OK" - - return jsonify({ - "message": f"Chunk {chunk} was not found.", - "error": "NotFound", - "statuscode": 404 - }), 404 - - -def __merge_chunks__(targetfile: Path, chunkpaths: tuple[Path, ...]) -> Path: - """Merge the chunks into a single file.""" - with open(targetfile, "ab") as _target: - for chunkfile in chunkpaths: - with open(chunkfile, "rb") as _chunkdata: - _target.write(_chunkdata.read()) - - chunkfile.unlink() - return targetfile - - -@rqtl2.route(("/upload/species/<int:species_id>/population/<int:population_id>" - "/rqtl2-bundle-chunked"), - methods=["POST"]) -@require_login -def upload_rqtl2_bundle_chunked_post(species_id: int, population_id: int): - """ - Extension to the `upload_rqtl2_bundle` endpoint above that allows large - files to be uploaded in chunks. - - This should hopefully speed up uploads, and if done right, even enable - resumable uploads - """ - _totalchunks = request.form.get("resumableTotalChunks", type=int) or 0 - _chunk = request.form.get("resumableChunkNumber", default=1, type=int) - _uploadfilename = request.form.get( - "resumableFilename", default="", type=str) or "" - _fileid = request.form.get( - "resumableIdentifier", default="", type=str) or "" - _targetfile = Path(app.config["UPLOAD_FOLDER"], _fileid) - - if _targetfile.exists(): - return jsonify({ - "message": ( - "A file with a similar unique identifier has previously been " - "uploaded and possibly is/has being/been processed."), - "error": "BadRequest", - "statuscode": 400 - }), 400 - - try: - # save chunk data - chunks_directory(_fileid).mkdir(exist_ok=True, parents=True) - request.files["file"].save(Path(chunks_directory(_fileid), - chunk_name(_uploadfilename, _chunk))) - - # Check whether upload is complete - chunkpaths = tuple( - Path(chunks_directory(_fileid), chunk_name(_uploadfilename, _achunk)) - for _achunk in range(1, _totalchunks+1)) - if all(_file.exists() for _file in chunkpaths): - # merge_files and clean up chunks - __merge_chunks__(_targetfile, chunkpaths) - chunks_directory(_fileid).rmdir() - jobid = trigger_rqtl2_bundle_qc( - species_id, population_id, _targetfile, _uploadfilename) - return url_for( - "expression-data.rqtl2.rqtl2_bundle_qc_status", jobid=jobid) - except Exception as exc:# pylint: disable=[broad-except] - msg = "Error processing uploaded file chunks." - app.logger.error(msg, exc_info=True, stack_info=True) - return jsonify({ - "message": msg, - "error": type(exc).__name__, - "error-description": " ".join(str(arg) for arg in exc.args), - "error-trace": traceback.format_exception(exc) - }), 500 - - return "OK" - - @rqtl2.route("/upload/species/rqtl2-bundle/qc-status/<uuid:jobid>", methods=["GET", "POST"]) @require_login @@ -362,24 +203,25 @@ def rqtl2_bundle_qc_status(jobid: UUID): if bool(messagelistname) else []) jobstatus = thejob["status"] if jobstatus == "error": - return render_template("rqtl2/rqtl2-qc-job-error.html", - job=thejob, - errorsgeneric=json.loads( - thejob.get("errors-generic", "[]")), - errorsgeno=json.loads( - thejob.get("errors-geno", "[]")), - errorspheno=json.loads( - thejob.get("errors-pheno", "[]")), - errorsphenose=json.loads( - thejob.get("errors-phenose", "[]")), - errorsphenocovar=json.loads( - thejob.get("errors-phenocovar", "[]")), - messages=logmessages) + return render_template( + "expression-data/rqtl2/rqtl2-qc-job-error.html", + job=thejob, + errorsgeneric=json.loads( + thejob.get("errors-generic", "[]")), + errorsgeno=json.loads( + thejob.get("errors-geno", "[]")), + errorspheno=json.loads( + thejob.get("errors-pheno", "[]")), + errorsphenose=json.loads( + thejob.get("errors-phenose", "[]")), + errorsphenocovar=json.loads( + thejob.get("errors-phenocovar", "[]")), + messages=logmessages) if jobstatus == "success": jobmeta = json.loads(thejob["job-metadata"]) species = species_by_id(dbconn, jobmeta["speciesid"]) return render_template( - "rqtl2/rqtl2-qc-job-results.html", + "expression-data/rqtl2/rqtl2-qc-job-results.html", species=species, population=population_by_species_and_id( dbconn, species["SpeciesId"], jobmeta["populationid"]), @@ -398,14 +240,14 @@ def rqtl2_bundle_qc_status(jobid: UUID): return None return render_template( - "rqtl2/rqtl2-qc-job-status.html", + "expression-data/rqtl2/rqtl2-qc-job-status.html", job=thejob, geno_percent=compute_percentage(thejob, "geno"), pheno_percent=compute_percentage(thejob, "pheno"), phenose_percent=compute_percentage(thejob, "phenose"), messages=logmessages) except jobs.JobNotFound: - return render_template("rqtl2/no-such-job.html", jobid=jobid) + return render_template("expression-data/rqtl2/no-such-job.html", jobid=jobid) def redirect_on_error(flaskroute, **kwargs): @@ -609,76 +451,6 @@ def select_geno_dataset(species_id: int, population_id: int): @rqtl2.route(("/upload/species/<int:species_id>/population/<int:population_id>" - "/rqtl2-bundle/create-geno-dataset"), - methods=["POST"]) -@require_login -def create_geno_dataset(species_id: int, population_id: int): - """Create a new geno dataset.""" - with database_connection(app.config["SQL_URI"]) as conn: - def __thunk__(): - sgeno_page = redirect(url_for("expression-data.rqtl2.select_dataset_info", - species_id=species_id, - population_id=population_id, - pgsrc="error"), - code=307) - errorclasses = "alert-error error-rqtl2 error-rqtl2-create-geno-dataset" - if not bool(request.form.get("dataset-name")): - flash("You must provide the dataset name", errorclasses) - return sgeno_page - if not bool(request.form.get("dataset-fullname")): - flash("You must provide the dataset full name", errorclasses) - return sgeno_page - public = 2 if request.form.get("dataset-public") == "on" else 0 - - with conn.cursor(cursorclass=DictCursor) as cursor: - datasetname = request.form["dataset-name"] - new_dataset = { - "name": datasetname, - "fname": request.form.get("dataset-fullname"), - "sname": request.form.get("dataset-shortname") or datasetname, - "today": date.today().isoformat(), - "pub": public, - "isetid": population_id - } - cursor.execute("SELECT * FROM GenoFreeze WHERE Name=%s", - (datasetname,)) - results = cursor.fetchall() - if bool(results): - flash( - f"A genotype dataset with name '{escape(datasetname)}' " - "already exists.", - errorclasses) - return redirect(url_for("expression-data.rqtl2.select_dataset_info", - species_id=species_id, - population_id=population_id, - pgsrc="error"), - code=307) - cursor.execute( - "INSERT INTO GenoFreeze(" - "Name, FullName, ShortName, CreateTime, public, InbredSetId" - ") " - "VALUES(" - "%(name)s, %(fname)s, %(sname)s, %(today)s, %(pub)s, %(isetid)s" - ")", - new_dataset) - flash("Created dataset successfully.", "alert-success") - return render_template( - "rqtl2/create-geno-dataset-success.html", - species=species_by_id(conn, species_id), - population=population_by_species_and_id( - conn, species_id, population_id), - rqtl2_bundle_file=request.form["rqtl2_bundle_file"], - geno_dataset={**new_dataset, "id": cursor.lastrowid}) - - return with_errors(__thunk__, - partial(check_species, conn=conn), - partial(check_population, conn=conn, species_id=species_id), - partial(check_r_qtl2_bundle, - species_id=species_id, - population_id=population_id)) - - -@rqtl2.route(("/upload/species/<int:species_id>/population/<int:population_id>" "/rqtl2-bundle/select-tissue"), methods=["POST"]) @require_login @@ -739,7 +511,7 @@ def create_tissue(species_id: int, population_id: int): tissue = create_new_tissue(conn, tissuename, tissueshortname) flash("Tissue created successfully!", "alert-success") return render_template( - "rqtl2/create-tissue-success.html", + "expression-data/rqtl2/create-tissue-success.html", species=species_by_id(conn, species_id), population=population_by_species_and_id( conn, species_id, population_id), @@ -869,7 +641,7 @@ def create_probeset_study(species_id: int, population_id: int): errorclasses) return dataset_info_page return render_template( - "rqtl2/create-probe-study-success.html", + "expression-data/rqtl2/create-probe-study-success.html", species=species_by_id(conn, species_id), population=population_by_species_and_id( conn, species_id, population_id), @@ -954,7 +726,7 @@ def create_probeset_dataset(species_id: int, population_id: int):#pylint: disabl errorclasses) return summary_page return render_template( - "rqtl2/create-probe-dataset-success.html", + "expression-data/rqtl2/create-probe-dataset-success.html", species=species_by_id(conn, species_id), population=population_by_species_and_id( conn, species_id, population_id), @@ -1009,7 +781,7 @@ def select_dataset_info(species_id: int, population_id: int): conn,form.get("geno-dataset-id", "").strip()) if "geno" in cdata and not bool(form.get("geno-dataset-id")): return render_template( - "rqtl2/select-geno-dataset.html", + "expression-data/rqtl2/select-geno-dataset.html", species=species, population=population, rqtl2_bundle_file=thefile.name, @@ -1019,7 +791,7 @@ def select_dataset_info(species_id: int, population_id: int): tissue = tissue_by_id(conn, form.get("tissueid", "").strip()) if "pheno" in cdata and not bool(tissue): return render_template( - "rqtl2/select-tissue.html", + "expression-data/rqtl2/select-tissue.html", species=species, population=population, rqtl2_bundle_file=thefile.name, @@ -1033,7 +805,7 @@ def select_dataset_info(species_id: int, population_id: int): conn, form.get("probe-study-id", "").strip()) if "pheno" in cdata and not bool(probeset_study): return render_template( - "rqtl2/select-probeset-study-id.html", + "expression-data/rqtl2/select-probeset-study-id.html", species=species, population=population, rqtl2_bundle_file=thefile.name, @@ -1049,7 +821,7 @@ def select_dataset_info(species_id: int, population_id: int): conn, form.get("probe-dataset-id", "").strip()) if "pheno" in cdata and not bool(probeset_dataset): return render_template( - "rqtl2/select-probeset-dataset.html", + "expression-data/rqtl2/select-probeset-dataset.html", species=species, population=population, rqtl2_bundle_file=thefile.name, @@ -1060,7 +832,7 @@ def select_dataset_info(species_id: int, population_id: int): conn, int(form["probe-study-id"])), avgmethods=averaging_methods(conn)) - return render_template("rqtl2/summary-info.html", + return render_template("expression-data/rqtl2/summary-info.html", species=species, population=population, rqtl2_bundle_file=thefile.name, @@ -1163,13 +935,19 @@ def rqtl2_processing_status(jobid: UUID): if thejob["status"] == "error": return render_template( - "rqtl2/rqtl2-job-error.html", job=thejob, messages=logmessages) + "expression-data/rqtl2/rqtl2-job-error.html", + job=thejob, + messages=logmessages) if thejob["status"] == "success": - return render_template("rqtl2/rqtl2-job-results.html", - job=thejob, - messages=logmessages) + return render_template( + "expression-data/rqtl2/rqtl2-job-results.html", + job=thejob, + messages=logmessages) return render_template( - "rqtl2/rqtl2-job-status.html", job=thejob, messages=logmessages) + "expression-data/rqtl2/rqtl2-job-status.html", + job=thejob, + messages=logmessages) except jobs.JobNotFound as _exc: - return render_template("rqtl2/no-such-job.html", jobid=jobid) + return render_template("expression-data/rqtl2/no-such-job.html", + jobid=jobid) diff --git a/uploader/population/views.py b/uploader/population/views.py index 003787a..270dd5f 100644 --- a/uploader/population/views.py +++ b/uploader/population/views.py @@ -1,8 +1,10 @@ """Views dealing with populations/inbredsets""" -import re import json import base64 +from markupsafe import escape +from MySQLdb.cursors import DictCursor +from gn_libs.mysqldb import database_connection from flask import (flash, request, url_for, @@ -10,13 +12,17 @@ from flask import (flash, Blueprint, current_app as app) +from uploader.samples.views import samplesbp +from uploader.oauth2.client import oauth2_post from uploader.ui import make_template_renderer from uploader.authorisation import require_login -from uploader.db_utils import database_connection -from uploader.samples.views import samplesbp -from uploader.species.models import (all_species, - species_by_id, - order_species_by_family) +from uploader.genotypes.views import genotypesbp +from uploader.datautils import enumerate_sequence +from uploader.phenotypes.views import phenotypesbp +from uploader.expression_data.views import exprdatabp +from uploader.species.models import all_species, species_by_id +from uploader.monadic_requests import make_either_error_handler +from uploader.input_validation import is_valid_representative_name from .models import (save_population, population_families, @@ -27,6 +33,9 @@ from .models import (save_population, __active_link__ = "populations" popbp = Blueprint("populations", __name__) popbp.register_blueprint(samplesbp, url_prefix="/") +popbp.register_blueprint(genotypesbp, url_prefix="/") +popbp.register_blueprint(phenotypesbp, url_prefix="/") +popbp.register_blueprint(exprdatabp, url_prefix="/") render_template = make_template_renderer("populations") @@ -38,7 +47,15 @@ def index(): if not bool(request.args.get("species_id")): return render_template( "populations/index.html", - species=order_species_by_family(all_species(conn))) + species=all_species(conn), + activelink="populations") + + species_id = request.args.get("species_id") + if species_id == "CREATE-SPECIES": + return redirect(url_for( + "species.create_species", + return_to="species.populations.list_species_populations")) + species = species_by_id(conn, request.args.get("species_id")) if not bool(species): flash("Invalid species identifier provided!", "alert-danger") @@ -58,38 +75,17 @@ def list_species_populations(species_id: int): return render_template( "populations/list-populations.html", species=species, - populations=populations_by_species(conn, species_id), + populations=enumerate_sequence(populations_by_species( + conn, species_id)), activelink="list-populations") -def valid_population_name(population_name: str) -> bool: - """ - Check whether the given name is a valid population name. - - Parameters - ---------- - population_name: a string of characters. - - Checks For - ---------- - * The name MUST start with an alphabet [a-zA-Z] - * The name MUST end with an alphabet [a-zA-Z] or number [0-9] - * The name MUST be composed of alphabets [a-zA-Z], numbers [0-9], - underscores (_) and/or hyphens (-). - - Returns - ------- - Boolean indicating whether or not the name is valid. - """ - pattern = re.compile(r"^[a-zA-Z]+[a-zA-Z0-9_-]*[a-zA-Z0-9]$") - return bool(pattern.match(population_name)) - - @popbp.route("/<int:species_id>/populations/create", methods=["GET", "POST"]) @require_login def create_population(species_id: int): """Create a new population.""" - with database_connection(app.config["SQL_URI"]) as conn: + with (database_connection(app.config["SQL_URI"]) as conn, + conn.cursor(cursorclass=DictCursor) as cursor): species = species_by_id(conn, species_id) if request.method == "GET": @@ -100,7 +96,7 @@ def create_population(species_id: int): ).decode("utf8") error_values = json.loads(base64.b64decode( - error_values.encode("utf8")).decode("utf8")) + error_values.encode("utf8")).decode("utf8"))# type: ignore[union-attr] return render_template( "populations/create-population.html", species=species, @@ -112,6 +108,7 @@ def create_population(species_id: int): {"id": "2", "value": "GEMMA"}, {"id": "3", "value": "R/qtl"}, {"id": "4", "value": "GEMMA, PLINK"}), + return_to=(request.args.get("return_to") or ""), activelink="create-population", **error_values) @@ -119,7 +116,7 @@ def create_population(species_id: int): flash("You must select a species.", "alert-danger") return redirect(url_for("species.populations.index")) - errors = tuple() + errors: tuple[tuple[str, str], ...] = tuple() population_name = (request.form.get( "population_name") or "").strip() @@ -127,7 +124,7 @@ def create_population(species_id: int): errors = errors + (("population_name", "You must provide a name for the population!"),) - if not valid_population_name(population_name): + if not is_valid_representative_name(population_name): errors = errors + (( "population_name", "The population name can only contain letters, numbers, " @@ -149,7 +146,7 @@ def create_population(species_id: int): species_id=species["SpeciesId"], error_values=values)) - new_population = save_population(conn, { + new_population = save_population(cursor, { "SpeciesId": species["SpeciesId"], "Name": population_name, "InbredSetName": population_fullname, @@ -161,9 +158,34 @@ def create_population(species_id: int): "GeneticType": request.form.get("population_genetic_type") or None }) - return redirect(url_for("species.populations.view_population", - species_id=species["SpeciesId"], - population_id=new_population["InbredSetId"])) + def __flash_success__(_success): + flash("Successfully created population " + f"{escape(new_population['FullName'])}.", + "alert-success") + return_to = request.form.get("return_to") or "" + if return_to: + return redirect(url_for( + return_to, + species_id=species["SpeciesId"], + population_id=new_population["InbredSetId"])) + return redirect(url_for( + "species.populations.view_population", + species_id=species["SpeciesId"], + population_id=new_population["InbredSetId"])) + + app.logger.debug("We begin setting up the privileges here…") + return oauth2_post( + "auth/resource/populations/create", + json={ + **dict(request.form), + "species_id": species_id, + "population_id": new_population["Id"], + "public": "on" + } + ).either( + make_either_error_handler( + "There was an error creating the population"), + __flash_success__) @popbp.route("/<int:species_id>/populations/<int:population_id>", diff --git a/uploader/publications/__init__.py b/uploader/publications/__init__.py new file mode 100644 index 0000000..7efcabb --- /dev/null +++ b/uploader/publications/__init__.py @@ -0,0 +1,2 @@ +"""Package for handling publications.""" +from .views import pubbp diff --git a/uploader/publications/datatables.py b/uploader/publications/datatables.py new file mode 100644 index 0000000..e07fafd --- /dev/null +++ b/uploader/publications/datatables.py @@ -0,0 +1,52 @@ +"""Fetch data for datatables.""" +import logging +from typing import Optional + +from MySQLdb.cursors import DictCursor + +from gn_libs.mysqldb import Connection, debug_query + +logger = logging.getLogger(__name__) + +def fetch_publications( + conn: Connection, + search: Optional[str] = None, + offset: int = 0, + limit: int = -1 +) -> tuple[dict, int, int, int]: + """Fetch publications from the database.""" + _query = "SELECT * FROM Publication" + _count_query = "SELECT COUNT(*) FROM Publication" + _params = None + _where_clause = "" + _limit_clause = "" + if search is not None and bool(search): + _where_clause = ("WHERE PubMed_ID LIKE %s " + "OR Authors LIKE %s " + "OR Title LIKE %s") + _params = (f"%{search}%",) * 3 + + if limit > 0: + _limit_clause = f"LIMIT {limit} OFFSET {offset}" + + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute("SELECT COUNT(*) FROM Publication") + _total_rows = int(cursor.fetchone()["COUNT(*)"]) + + cursor.execute(f"{_count_query} {_where_clause}", _params) + debug_query(cursor, logger) + _result = cursor.fetchone() + _total_filtered = int(_result["COUNT(*)"] if bool(_result) else 0) + + cursor.execute(f"{_query} {_where_clause} {_limit_clause}", _params) + debug_query(cursor, logger) + _current_filtered = tuple( + {**dict(row), "index": idx} + for idx, row + in enumerate(cursor.fetchall(), start=offset+1)) + + return ( + _current_filtered, + len(_current_filtered), + _total_filtered, + _total_rows) diff --git a/uploader/publications/misc.py b/uploader/publications/misc.py new file mode 100644 index 0000000..fca6f71 --- /dev/null +++ b/uploader/publications/misc.py @@ -0,0 +1,25 @@ +"""Miscellaneous functions dealing with publications.""" + + +def publications_differences( + filedata: tuple[dict, ...], + dbdata: tuple[dict, ...], + pubmedid2pubidmap: tuple[dict, ...] +) -> tuple[dict, ...]: + """Compute the differences between file data and db data""" + diff = tuple() + for filerow, dbrow in zip( + sorted(filedata, key=lambda item: ( + item["phenotype_id"], item["xref_id"])), + sorted(dbdata, key=lambda item: ( + item["PhenotypeId"], item["xref_id"]))): + if filerow["PubMed_ID"] == dbrow["PubMed_ID"]: + continue + + newpubmed = filerow["PubMed_ID"] + diff = diff + ({ + **dbrow, + "PubMed_ID": newpubmed, + "PublicationId": pubmedid2pubidmap.get(newpubmed)},) + + return diff diff --git a/uploader/publications/models.py b/uploader/publications/models.py new file mode 100644 index 0000000..b199991 --- /dev/null +++ b/uploader/publications/models.py @@ -0,0 +1,96 @@ +"""Module to handle persistence and retrieval of publication to/from MariaDB""" +import logging +from typing import Iterable, Optional + +from MySQLdb.cursors import DictCursor + +from gn_libs.mysqldb import Connection, debug_query + +logger = logging.getLogger(__name__) + + +def fetch_phenotype_publications( + conn: Connection, + ids: tuple[tuple[int, int], ...] +) -> tuple[dict, ...]: + """Fetch publication from database by ID.""" + paramstr = ",".join(["(%s, %s)"] * len(ids)) + query = ( + "SELECT " + "pxr.PhenotypeId, pxr.Id AS xref_id, pxr.PublicationId, pub.PubMed_ID " + "FROM PublishXRef AS pxr INNER JOIN Publication AS pub " + "ON pxr.PublicationId=pub.Id " + f"WHERE (pxr.PhenotypeId, pxr.Id) IN ({paramstr})") + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(query, tuple(item for row in ids for item in row)) + return tuple(dict(row) for row in cursor.fetchall()) + + +def create_new_publications( + conn: Connection, + publications: tuple[dict, ...] +) -> tuple[dict, ...]: + if len(publications) > 0: + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.executemany( + ("INSERT INTO " + "Publication( " + "PubMed_ID, Abstract, Authors, Title, Journal, Volume, Pages, " + "Month, Year" + ") " + "VALUES(" + "%(pubmed_id)s, %(abstract)s, %(authors)s, %(title)s, " + "%(journal)s, %(volume)s, %(pages)s, %(month)s, %(year)s" + ") " + "RETURNING *"), + publications) + return tuple({ + **row, "publication_id": row["Id"] + } for row in cursor.fetchall()) + return tuple() + + +def update_publications(conn: Connection , publications: tuple[dict, ...]) -> tuple[dict, ...]: + """Update details for multiple publications""" + if len(publications) > 0: + with conn.cursor(cursorclass=DictCursor) as cursor: + logger.debug("UPDATING PUBLICATIONS: %s", publications) + cursor.executemany( + ("UPDATE Publication SET " + "PubMed_ID=%(pubmed_id)s, Abstract=%(abstract)s, " + "Authors=%(authors)s, Title=%(title)s, Journal=%(journal)s, " + "Volume=%(volume)s, Pages=%(pages)s, Month=%(month)s, " + "Year=%(year)s " + "WHERE Id=%(publication_id)s"), + publications) + debug_query(cursor, logger) + return publications + return tuple() + return tuple() + + +def fetch_publication_by_id(conn: Connection, publication_id: int) -> dict: + """Fetch a specific publication from the database.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute("SELECT * FROM Publication WHERE Id=%s", + (publication_id,)) + _res = cursor.fetchone() + return dict(_res) if _res else {} + + +def fetch_publication_phenotypes( + conn: Connection, publication_id: int) -> Iterable[dict]: + """Fetch all phenotypes linked to this publication.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT pxr.Id AS xref_id, pxr.PublicationId, phe.* " + "FROM PublishXRef AS pxr INNER JOIN Phenotype AS phe " + "ON pxr.PhenotypeId=phe.Id " + "WHERE pxr.PublicationId=%s", + (publication_id,)) + while True: + row = cursor.fetchone() + if row: + yield row + else: + break diff --git a/uploader/publications/pubmed.py b/uploader/publications/pubmed.py new file mode 100644 index 0000000..ed9b652 --- /dev/null +++ b/uploader/publications/pubmed.py @@ -0,0 +1,103 @@ +"""Module to interact with NCBI's PubMed""" +import logging + +import requests +from lxml import etree + +logger = logging.getLogger(__name__) + + +def __pub_date__(pubdate: etree.Element): + pubyear = pubdate.find("Year") + pubmonth = pubdate.find("Month") + pubday = pubdate.find("Day") + return { + "year": pubyear.text if pubyear is not None else None, + "month": pubmonth.text if pubmonth is not None else None, + "day": pubday.text if pubday is not None else None + } + + +def __journal__(journal: etree.Element) -> dict: + volume = journal.find("JournalIssue/Volume") + issue = journal.find("JournalIssue/Issue") + return { + "volume": volume.text if volume is not None else None, + "issue": issue.text if issue is not None else None, + **__pub_date__(journal.find("JournalIssue/PubDate")), + "journal": journal.find("Title").text + } + +def __author__(author: etree.Element) -> str: + return "%s %s" % ( + author.find("LastName").text, + author.find("Initials").text) + + +def __pages__(pagination: etree.Element) -> str: + start = pagination.find("StartPage") + end = pagination.find("EndPage") + return (start.text + ( + f"-{end.text}" if end is not None else "" + )) if start is not None else "" + + +def __abstract__(article: etree.Element) -> str: + abstract = article.find("Abstract/AbstractText") + return abstract.text if abstract is not None else None + + +def __article__(pubmed_article: etree.Element) -> dict: + article = pubmed_article.find("MedlineCitation/Article") + return { + "pubmed_id": int(pubmed_article.find("MedlineCitation/PMID").text), + "title": article.find("ArticleTitle").text, + **__journal__(article.find("Journal")), + "abstract": __abstract__(article), + "pages": __pages__(article.find("Pagination")), + "authors": ", ".join(__author__(author) + for author in article.findall("AuthorList/Author")) + } + + +def __process_pubmed_publication_data__(text) -> tuple[dict, ...]: + """Process the data from PubMed into usable data.""" + doc = etree.XML(text) + articles = doc.xpath("//PubmedArticle") + logger.debug("Retrieved %s publications from NCBI", len(articles)) + return tuple(__article__(article) for article in articles) + +def fetch_publications(pubmed_ids: tuple[int, ...]) -> tuple[dict, ...]: + """Retrieve data on new publications from NCBI.""" + # See whether we can retrieve multiple publications in one go + # Parse data and save to DB + # Return PublicationId(s) for new publication(s). + if len(pubmed_ids) == 0: + logger.debug("There are no new PubMed IDs to fetch") + return tuple() + + logger.info("Fetching publications data for the following PubMed IDs: %s", + ", ".join((str(pid) for pid in pubmed_ids))) + + # Should we, perhaps, pass this in from a config variable? + uri = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi" + try: + response = requests.get( + uri, + params={ + "db": "pubmed", + "retmode": "xml", + "id": ",".join(str(item) for item in pubmed_ids) + }) + + if response.status_code == 200: + return __process_pubmed_publication_data__(response.text) + + logger.error( + "Could not fetch the new publication from %s (status code: %s)", + uri, + response.status_code) + except requests.exceptions.ConnectionError: + logger.error("Could not find the domain %s", uri) + + return tuple() diff --git a/uploader/publications/views.py b/uploader/publications/views.py new file mode 100644 index 0000000..0608a35 --- /dev/null +++ b/uploader/publications/views.py @@ -0,0 +1,107 @@ +"""Endpoints for publications""" +import json + +from MySQLdb.cursors import DictCursor +from gn_libs.mysqldb import database_connection +from flask import ( + flash, + request, + url_for, + redirect, + Blueprint, + render_template, + current_app as app) + +from uploader.authorisation import require_login + +from .models import ( + fetch_publication_by_id, + create_new_publications, + fetch_publication_phenotypes) + +from .datatables import fetch_publications + +from gn_libs.debug import __pk__ + +pubbp = Blueprint("publications", __name__) + + +@pubbp.route("/", methods=["GET"]) +@require_login +def index(): + """Index page for publications.""" + with database_connection(app.config["SQL_URI"]) as conn: + return render_template("publications/index.html") + + +@pubbp.route("/list", methods=["GET"]) +@require_login +def list_publications(): + # request breakdown: + # https://datatables.net/manual/server-side + _page = int(request.args.get("draw")) + _length = int(request.args.get("length") or '-1') + _start = int(request.args.get("start") or '0') + _search = request.args["search[value]"] + with (database_connection(app.config["SQL_URI"]) as conn, + conn.cursor(cursorclass=DictCursor) as cursor): + _publications, _current_rows, _totalfiltered, _totalrows = fetch_publications( + conn, + _search, + offset=_start, + limit=_length) + + return json.dumps({ + "draw": _page, + "recordsTotal": _totalrows, + "recordsFiltered": _totalfiltered, + "publications": _publications, + "status": "success" + }) + + +@pubbp.route("/view/<int:publication_id>", methods=["GET"]) +@require_login +def view_publication(publication_id: int): + """View more details on a particular publication.""" + with database_connection(app.config["SQL_URI"]) as conn: + return render_template( + "publications/view-publication.html", + publication=fetch_publication_by_id(conn, publication_id), + linked_phenotypes=tuple(fetch_publication_phenotypes( + conn, publication_id))) + + +@pubbp.route("/create", methods=["GET", "POST"]) +@require_login +def create_publication(): + """Create a new publication.""" + if(request.method == "GET"): + return render_template("publications/create-publication.html") + form = request.form + authors = form.get("publication-authors").encode("utf8") + if authors is None or authors == "": + flash("The publication's author(s) MUST be provided!", "alert alert-danger") + return redirect(url_for("publications.create", **request.args)) + + with database_connection(app.config["SQL_URI"]) as conn: + publications = create_new_publications(conn, ({ + "pubmed_id": form.get("pubmed-id") or None, + "abstract": form.get("publication-abstract").encode("utf8") or None, + "authors": authors, + "title": form.get("publication-title").encode("utf8") or None, + "journal": form.get("publication-journal").encode("utf8") or None, + "volume": form.get("publication-volume").encode("utf8") or None, + "pages": form.get("publication-pages").encode("utf8") or None, + "month": (form.get("publication-month") or "").encode("utf8").capitalize() or None, + "year": form.get("publication-year").encode("utf8") or None + },)) + flash("New publication created!", "alert alert-success") + return redirect(url_for( + request.args.get("return_to") or "publications.view_publication", + publication_id=publications[0]["publication_id"], + **request.args)) + + flash("Publication creation failed!", "alert alert-danger") + app.logger.debug("Failed to create the new publication.", exc_info=True) + return redirect(url_for("publications.create_publication")) diff --git a/uploader/request_checks.py b/uploader/request_checks.py new file mode 100644 index 0000000..f1d8027 --- /dev/null +++ b/uploader/request_checks.py @@ -0,0 +1,75 @@ +"""Functions to perform common checks. + +These are useful for reusability, and hence maintainability of the code. +""" +from functools import wraps + +from gn_libs.mysqldb import database_connection +from flask import flash, url_for, redirect, current_app as app + +from uploader.species.models import species_by_id +from uploader.population.models import population_by_species_and_id + +def with_species(redirect_uri: str): + """Ensure the species actually exists.""" + def __decorator__(function): + @wraps(function) + def __with_species__(**kwargs): + try: + species_id = int(kwargs.get("species_id")) + if not bool(species_id): + flash("Expected species_id value to be present!", + "alert-danger") + return redirect(url_for(redirect_uri)) + with database_connection(app.config["SQL_URI"]) as conn: + species = species_by_id(conn, species_id) + if not bool(species): + flash("Could not find species with that ID", + "alert-danger") + return redirect(url_for(redirect_uri)) + except ValueError as _verr: + app.logger.debug( + "Exception converting value to integer: %s", + kwargs.get("species_id"), + exc_info=True) + flash("Expected an integer for 'species_id' value.", + "alert-danger") + return redirect(url_for(redirect_uri)) + return function(**{**kwargs, "species": species}) + return __with_species__ + return __decorator__ + + +def with_population(species_redirect_uri: str, redirect_uri: str): + """Ensure the population actually exists.""" + def __decorator__(function): + @wraps(function) + @with_species(redirect_uri=species_redirect_uri) + def __with_population__(**kwargs): + try: + species_id = int(kwargs["species_id"]) + population_id = int(kwargs.get("population_id")) + select_population_uri = redirect(url_for( + redirect_uri, species_id=species_id)) + if not bool(population_id): + flash("Expected population_id value to be present!", + "alert-danger") + return select_population_uri + with database_connection(app.config["SQL_URI"]) as conn: + population = population_by_species_and_id( + conn, species_id, population_id) + if not bool(population): + flash("Could not find population with that ID", + "alert-danger") + return select_population_uri + except ValueError as _verr: + app.logger.debug( + "Exception converting value to integer: %s", + kwargs.get("population_id"), + exc_info=True) + flash("Expected an integer for 'population_id' value.", + "alert-danger") + return select_population_uri + return function(**{**kwargs, "population": population}) + return __with_population__ + return __decorator__ diff --git a/uploader/route_utils.py b/uploader/route_utils.py new file mode 100644 index 0000000..ce718fb --- /dev/null +++ b/uploader/route_utils.py @@ -0,0 +1,42 @@ +"""Generic routing utilities.""" +from flask import flash, url_for, redirect, render_template, current_app as app + +from gn_libs.mysqldb import database_connection + +from uploader.population.models import (populations_by_species, + population_by_species_and_id) + +def generic_select_population( + # pylint: disable=[too-many-arguments, too-many-positional-arguments] + species: dict, + template: str, + population_id: str, + back_to: str, + forward_to: str, + activelink: str, + error_message: str = "No such population found!" +): + """Handles common flow for 'select population' step.""" + with database_connection(app.config["SQL_URI"]) as conn: + if not bool(population_id): + return render_template( + template, + species=species, + populations=populations_by_species(conn, species["SpeciesId"]), + activelink=activelink) + + if population_id == "CREATE-POPULATION": + return redirect(url_for( + "species.populations.create_population", + species_id=species["SpeciesId"], + return_to=forward_to)) + + population = population_by_species_and_id( + conn, species["SpeciesId"], int(population_id)) + if not bool(population): + flash(error_message, "alert-danger") + return redirect(url_for(back_to, species_id=species["SpeciesId"])) + + return redirect(url_for(forward_to, + species_id=species["SpeciesId"], + population_id=population["Id"])) diff --git a/uploader/samples/__init__.py b/uploader/samples/__init__.py new file mode 100644 index 0000000..1bd6d2d --- /dev/null +++ b/uploader/samples/__init__.py @@ -0,0 +1 @@ +"""Samples package. Handle samples uploads and editing.""" diff --git a/uploader/samples/models.py b/uploader/samples/models.py index d7d5384..b419d61 100644 --- a/uploader/samples/models.py +++ b/uploader/samples/models.py @@ -15,11 +15,11 @@ def samples_by_species_and_population( """Fetch the samples by their species and population.""" with conn.cursor(cursorclass=DictCursor) as cursor: cursor.execute( - "SELECT iset.InbredSetId, s.* FROM InbredSet AS iset " - "INNER JOIN StrainXRef AS sxr ON iset.InbredSetId=sxr.InbredSetId " - "INNER JOIN Strain AS s ON sxr.StrainId=s.Id " - "WHERE s.SpeciesId=%(species_id)s " - "AND iset.InbredSetId=%(population_id)s", + "SELECT InbredSet.InbredSetId, Strain.* FROM InbredSet " + "INNER JOIN StrainXRef ON InbredSet.InbredSetId=StrainXRef.InbredSetId " + "INNER JOIN Strain ON StrainXRef.StrainId=Strain.Id " + "WHERE Strain.SpeciesId=%(species_id)s " + "AND InbredSet.InbredSetId=%(population_id)s", {"species_id": species_id, "population_id": population_id}) return tuple(cursor.fetchall()) diff --git a/uploader/samples/views.py b/uploader/samples/views.py index 6e3dc4b..c0adb88 100644 --- a/uploader/samples/views.py +++ b/uploader/samples/views.py @@ -3,50 +3,51 @@ import os import sys import uuid from pathlib import Path -from typing import Iterator -import MySQLdb as mdb from redis import Redis -from MySQLdb.cursors import DictCursor -from flask import ( - flash, - request, - url_for, - redirect, - Blueprint, - render_template, - current_app as app) +from flask import (flash, + request, + url_for, + redirect, + Blueprint, + current_app as app) from uploader import jobs from uploader.files import save_file -from uploader.datautils import order_by_family +from uploader.ui import make_template_renderer from uploader.authorisation import require_login from uploader.input_validation import is_integer_input -from uploader.db_utils import ( - with_db_connection, - database_connection, - with_redis_connection) -from uploader.species.models import (all_species, - species_by_id, - order_species_by_family) -from uploader.population.models import(save_population, - population_by_id, - populations_by_species, - population_by_species_and_id) +from uploader.population.models import population_by_id +from uploader.route_utils import generic_select_population +from uploader.datautils import safe_int, enumerate_sequence +from uploader.species.models import all_species, species_by_id +from uploader.request_checks import with_species, with_population +from uploader.db_utils import (with_db_connection, + database_connection, + with_redis_connection) from .models import samples_by_species_and_population samplesbp = Blueprint("samples", __name__) +render_template = make_template_renderer("samples") @samplesbp.route("/samples", methods=["GET"]) +@require_login def index(): """Direct entry-point for uploading/handling the samples.""" with database_connection(app.config["SQL_URI"]) as conn: if not bool(request.args.get("species_id")): return render_template( "samples/index.html", - species=order_species_by_family(all_species(conn)), + species=all_species(conn), activelink="samples") + + species_id = request.args.get("species_id") + if species_id == "CREATE-SPECIES": + return redirect(url_for( + "species.create_species", + return_to="species.populations.samples.select_population")) + species = species_by_id(conn, request.args.get("species_id")) if not bool(species): flash("No such species!", "alert-danger") @@ -56,61 +57,35 @@ def index(): @samplesbp.route("<int:species_id>/samples/select-population", methods=["GET"]) -def select_population(species_id: int): +@require_login +@with_species(redirect_uri="species.populations.samples.index") +def select_population(species: dict, **kwargs):# pylint: disable=[unused-argument] """Select the population to use for the samples.""" - with database_connection(app.config["SQL_URI"]) as conn: - species = species_by_id(conn, species_id) - if not bool(species): - flash("Invalid species!", "alert-danger") - return redirect(url_for("species.populations.samples.index")) - - if not bool(request.args.get("population_id")): - return render_template("samples/select-population.html", - species=species, - populations=order_by_family( - populations_by_species( - conn, - species_id), - order_key="FamilyOrder"), - activelink="samples") - - population = population_by_id(conn, request.args.get("population_id")) - if not bool(population): - flash("Population not found!", "alert-danger") - return redirect(url_for( - "species.populations.samples.select_population", - species_id=species_id)) - - return redirect(url_for("species.populations.samples.list_samples", - species_id=species_id, - population_id=population["Id"])) + return generic_select_population( + species, + "samples/select-population.html", + request.args.get("population_id") or "", + "species.populations.samples.select_population", + "species.populations.samples.list_samples", + "samples", + "Population not found!") @samplesbp.route("<int:species_id>/populations/<int:population_id>/samples") -def list_samples(species_id: int, population_id: int): +@require_login +@with_population( + species_redirect_uri="species.populations.samples.index", + redirect_uri="species.populations.samples.select_population") +def list_samples(species: dict, population: dict, **kwargs):# pylint: disable=[unused-argument] """ List the samples in a particular population and give the ability to upload new ones. """ with database_connection(app.config["SQL_URI"]) as conn: - species = species_by_id(conn, species_id) - if not bool(species): - flash("Invalid species!", "alert-danger") - return redirect(url_for("species.populations.samples.index")) - - population = population_by_id(conn, population_id) - if not bool(population): - flash("Population not found!", "alert-danger") - return redirect(url_for( - "species.populations.samples.select_population", - species_id=species_id)) - - all_samples = samples_by_species_and_population( - conn, species_id, population_id) + all_samples = enumerate_sequence(samples_by_species_and_population( + conn, species["SpeciesId"], population["Id"])) total_samples = len(all_samples) - offset = int(request.args.get("from") or 0) - if offset < 0: - offset = 0 - count = int(request.args.get("count") or 10) + offset = max(safe_int(request.args.get("from") or 0), 0) + count = int(request.args.get("count") or 20) return render_template("samples/list-samples.html", species=species, population=population, @@ -197,6 +172,11 @@ def upload_samples(species_id: int, population_id: int):#pylint: disable=[too-ma redisuri = app.config["REDIS_URL"] with Redis.from_url(redisuri, decode_responses=True) as rconn: + #TODO: Add a QC step here — what do we check? + # 1. Does any sample in the uploaded file exist within the database? + # If yes, what is/are its/their species and population? + # 2. If yes 1. above, provide error with notes on which species and + # populations already own the samples. the_job = jobs.launch_job( jobs.initialise_job( rconn, @@ -224,56 +204,54 @@ def upload_samples(species_id: int, population_id: int):#pylint: disable=[too-ma @samplesbp.route("<int:species_id>/populations/<int:population_id>/" "upload-samples/status/<uuid:job_id>", methods=["GET"]) -def upload_status(species_id: int, population_id: int, job_id: uuid.UUID): +@require_login +@with_population(species_redirect_uri="species.populations.samples.index", + redirect_uri="species.populations.samples.select_population") +def upload_status(species: dict, population: dict, job_id: uuid.UUID, **kwargs):# pylint: disable=[unused-argument] """Check on the status of a samples upload job.""" - with database_connection(app.config["SQL_URI"]) as conn: - species = species_by_id(conn, species_id) - if not bool(species): - flash("You must provide a valid species.", "alert-danger") - return redirect(url_for("species.populations.samples.index")) + job = with_redis_connection(lambda rconn: jobs.job( + rconn, jobs.jobsnamespace(), job_id)) + if job: + status = job["status"] + if status == "success": + return render_template("samples/upload-success.html", + job=job, + species=species, + population=population,) - population = population_by_species_and_id( - conn, species_id, population_id) - if not bool(population): - flash("You must provide a valid population.", "alert-danger") + if status == "error": return redirect(url_for( - "species.populations.samples.select_population", - species_id=species_id)) - - job = with_redis_connection(lambda rconn: jobs.job( - rconn, jobs.jobsnamespace(), job_id)) - if job: - status = job["status"] - if status == "success": - return render_template("samples/upload-success.html", - job=job, - species=species, - population=population,) - - if status == "error": + "species.populations.samples.upload_failure", + species_id=species["SpeciesId"], + population_id=population["Id"], + job_id=job_id)) + + error_filename = Path(jobs.error_filename( + job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors")) + if error_filename.exists(): + stat = os.stat(error_filename) + if stat.st_size > 0: return redirect(url_for( - "species.populations.samples.upload_failure", job_id=job_id)) + "samples.upload_failure", job_id=job_id)) - error_filename = Path(jobs.error_filename( - job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors")) - if error_filename.exists(): - stat = os.stat(error_filename) - if stat.st_size > 0: - return redirect(url_for( - "samples.upload_failure", job_id=job_id)) + return render_template("samples/upload-progress.html", + species=species, + population=population, + job=job) # maybe also handle this? - return render_template("samples/upload-progress.html", - species=species, - population=population, - job=job) # maybe also handle this? + return render_template("no_such_job.html", + job_id=job_id, + species=species, + population=population), 400 - return render_template("no_such_job.html", - job_id=job_id, - species=species, - population=population), 400 -@samplesbp.route("/upload/failure/<uuid:job_id>", methods=["GET"]) -def upload_failure(job_id: uuid.UUID): +@samplesbp.route("<int:species_id>/populations/<int:population_id>/" + "upload-samples/failure/<uuid:job_id>", + methods=["GET"]) +@require_login +@with_population(species_redirect_uri="species.populations.samples.index", + redirect_uri="species.populations.samples.select_population") +def upload_failure(species: dict, population: dict, job_id: uuid.UUID, **kwargs): """Display the errors of the samples upload failure.""" job = with_redis_connection(lambda rconn: jobs.job( rconn, jobs.jobsnamespace(), job_id)) @@ -287,4 +265,7 @@ def upload_failure(job_id: uuid.UUID): if stat.st_size > 0: return render_template("worker_failure.html", job_id=job_id) - return render_template("samples/upload-failure.html", job=job) + return render_template("samples/upload-failure.html", + species=species, + population=population, + job=job) diff --git a/uploader/session.py b/uploader/session.py index 399f28c..5af5827 100644 --- a/uploader/session.py +++ b/uploader/session.py @@ -77,12 +77,15 @@ def set_user_token(token: str) -> SessionInfo: """Set the user's token.""" info = session_info() return save_session_info({ - **info, "user": {**info["user"], "token": Right(token)}})#type: ignore[misc] + **info, + "user": {**info["user"], "token": Right(token), "logged_in": True} + })#type: ignore[misc] def set_user_details(userdets: UserDetails) -> SessionInfo: """Set the user details information""" - return save_session_info({**session_info(), "user": userdets})#type: ignore[misc] + info = session_info() + return save_session_info({**info, "user": {**info["user"], **userdets}})#type: ignore[misc] def user_details() -> UserDetails: """Retrieve user details.""" @@ -96,7 +99,7 @@ def user_token() -> Either: def set_auth_server_jwks(keyset: KeySet) -> KeySet: """Update the JSON Web Keys in the session.""" save_session_info({ - **session_info(), + **session_info(),# type: ignore[misc] "auth_server_jwks": { "last-updated": datetime.now().timestamp(), "jwks": keyset.as_dict() diff --git a/uploader/species/models.py b/uploader/species/models.py index 4426b64..db53d48 100644 --- a/uploader/species/models.py +++ b/uploader/species/models.py @@ -28,7 +28,7 @@ def order_species_by_family(species: tuple[dict, ...]) -> list: **ordered, _key: ordered.get(_key, tuple()) + (current,) } - ordered = reduce(__order__, species, {}) + ordered = reduce(__order__, species, {})# type: ignore[var-annotated] return sorted(tuple(ordered.items()), key=lambda item: item[0][0]) @@ -47,7 +47,7 @@ def save_species(conn: mdb.Connection, common_name: str, scientific_name: str, family: str, - taxon_id: Optional[str] = None) -> int: + taxon_id: Optional[str] = None) -> dict: """ Save a new species to the database. @@ -58,17 +58,18 @@ def save_species(conn: mdb.Connection, common_name: The species' common name. scientific_name; The species' scientific name. """ - genus, species = scientific_name.split(" ") + genus, *species_parts = scientific_name.split(" ") + species_name: str = " ".join(species_parts) families = species_families(conn) with conn.cursor() as cursor: cursor.execute("SELECT MAX(OrderId) FROM Species") species = { "common_name": common_name, "common_name_lower": common_name.lower(), - "menu_name": f"{common_name} ({genus[0]}. {species.lower()})", + "menu_name": f"{common_name} ({genus[0]}. {species_name.lower()})", "scientific_name": scientific_name, "family": family, - "family_order": families[family], + "family_order": families.get(family, 999999), "taxon_id": taxon_id, "species_order": cursor.fetchone()[0] + 5 } @@ -91,6 +92,56 @@ def save_species(conn: mdb.Connection, } +def update_species(# pylint: disable=[too-many-arguments] + conn: mdb.Connection, + species_id: int, + common_name: str, + scientific_name: str, + family: str, + family_order: int, + species_order: int +): + """Update a species' details. + + Parameters + ---------- + conn: A connection to the MariaDB database. + species_id: The species identifier + + Key-Word Arguments + ------------------ + common_name: A layman's name for the species + scientific_name: A binomial nomenclature name for the species + family: The grouping under which the species falls + family_order: The ordering for the "family" above + species_order: The ordering of this species in relation to others + """ + with conn.cursor(cursorclass=DictCursor) as cursor: + genus, *species_parts = scientific_name.split(" ") + species_name = " ".join(species_parts) + species = { + "species_id": species_id, + "common_name": common_name, + "common_name_lower": common_name.lower(), + "menu_name": f"{common_name} ({genus[0]}. {species_name.lower()})", + "scientific_name": scientific_name, + "family": family, + "family_order": family_order, + "species_order": species_order + } + cursor.execute( + "UPDATE Species SET " + "SpeciesName=%(common_name)s, " + "Name=%(common_name_lower)s, " + "MenuName=%(menu_name)s, " + "FullName=%(scientific_name)s, " + "Family=%(family)s, " + "FamilyOrderId=%(family_order)s, " + "OrderId=%(species_order)s " + "WHERE Id=%(species_id)s", + species) + + def species_families(conn: mdb.Connection) -> dict: """Retrieve the families under which species are grouped.""" with conn.cursor(cursorclass=DictCursor) as cursor: diff --git a/uploader/species/views.py b/uploader/species/views.py index f39ca98..cea2f68 100644 --- a/uploader/species/views.py +++ b/uploader/species/views.py @@ -1,5 +1,7 @@ """Endpoints handling species.""" +from markupsafe import escape from pymonad.either import Left, Right, Either +from gn_libs.mysqldb import database_connection from flask import (flash, request, url_for, @@ -8,26 +10,32 @@ from flask import (flash, current_app as app) from uploader.population import popbp -from uploader.datautils import order_by_family +from uploader.platforms import platformsbp from uploader.ui import make_template_renderer -from uploader.db_utils import database_connection from uploader.oauth2.client import oauth2_get, oauth2_post from uploader.authorisation import require_login, require_token +from uploader.datautils import order_by_family, enumerate_sequence -from .models import all_species, save_species, species_by_id, species_families +from .models import (all_species, + save_species, + species_by_id, + update_species, + species_families) speciesbp = Blueprint("species", __name__) speciesbp.register_blueprint(popbp, url_prefix="/") +speciesbp.register_blueprint(platformsbp, url_prefix="/") render_template = make_template_renderer("species") @speciesbp.route("/", methods=["GET"]) +@require_login def list_species(): """List and display all the species in the database.""" with database_connection(app.config["SQL_URI"]) as conn: return render_template("species/list-species.html", - allspecies=all_species(conn)) + allspecies=enumerate_sequence(all_species(conn))) @speciesbp.route("/<int:species_id>", methods=["GET"]) @require_login @@ -55,6 +63,8 @@ def create_species(): if request.method == "GET": return render_template("species/create-species.html", families=species_families(conn), + return_to=( + request.args.get("return_to") or ""), activelink="create-species") error = False @@ -72,7 +82,7 @@ def create_species(): error = True parts = tuple(name.strip() for name in scientific_name.split(" ")) - if len(parts) != 2 or not all(bool(name) for name in parts): + if (len(parts) != 2 and len(parts) != 3) or not all(bool(name) for name in parts): flash("The scientific name you provided is invalid.", "alert-danger") error = True @@ -106,7 +116,15 @@ def create_species(): species = save_species( conn, common_name, scientific_name, family, taxon_id) - flash("Species saved successfully!", "alert-success") + flash( + f"You have successfully added species " + f"'{escape(species['scientific_name'])} " + f"({escape(species['common_name'])})'.", + "alert-success") + + return_to = request.form.get("return_to").strip() + if return_to: + return redirect(url_for(return_to, species_id=species["species_id"])) return redirect(url_for("species.view_species", species_id=species["species_id"])) @@ -177,9 +195,14 @@ def edit_species_extra(token: dict, species_id: int):# pylint: disable=[unused-a ).either(__failure__, lambda res: res) if bool(species) and request.method == "POST": - flash("We would have edited the species, but the feature is not " - "currently implemented …", - "alert-danger") + update_species(conn, + species_id, + request.form["species_name"], + request.form["species_fullname"], + request.form["species_family"], + int(request.form["species_familyorderid"]), + int(request.form["species_orderid"])) + flash("Updated species successfully.", "alert-success") return redirect(url_for("species.edit_species_extra", species_id=species_id)) diff --git a/uploader/static/css/styles.css b/uploader/static/css/styles.css index 834c563..df50dec 100644 --- a/uploader/static/css/styles.css +++ b/uploader/static/css/styles.css @@ -1,115 +1,187 @@ +* { + box-sizing: border-box; +} + body { margin: 0.7em; - box-sizing: border-box; display: grid; - grid-template-columns: 1fr 6fr; - grid-template-rows: 5em 100%; + grid-template-columns: 2fr 8fr; grid-gap: 20px; - font-family: Georgia, Garamond, serif; + font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-style: normal; + font-size: 20px; } #header { - grid-column: 1/3; - width: 100%; - /* background: cyan; */ - padding-top: 0.5em; - border-radius: 0.5em; + /* Place it in the parent element */ + grid-column-start: 1; + grid-column-end: 3; + /* Define layout for the children elements */ + display: grid; + grid-template-columns: 8fr 2fr; + + /* Content styling */ background-color: #336699; - border-color: #080808; color: #FFFFFF; - background-image: none; + border-radius: 3px; + min-height: 30px; } -#header .header { - font-size: 2em; - display: inline-block; - text-align: center; -} +#header #header-text { + /* Place it in the parent element */ + grid-column-start: 1; + grid-column-end: 2; -#header .header-nav { - display: inline-block; - color: #FFFFFF; + /* Content styling */ + padding-left: 1em; } -#header .header-nav li { - border-width: 1px; - border-color: #FFFFFF; - vertical-align: middle; - margin: 0.2em; +#header #header-nav { + /* Place it in the parent element */ + grid-column-start: 2; + grid-column-end: 3; } -#header .header-nav a { +#header #header-nav .nav li a { + /* Content styling */ color: #FFFFFF; - text-decoration: none; + background: #4477AA; + border: solid 5px #336699; + border-radius: 5px; + font-size: 0.7em; + text-align: center; + padding: 1px 7px; } #nav-sidebar { - grid-column: 1/2; - /* background: #e5e5ff; */ - padding-top: 0.5em; - border-radius: 0.5em; - font-size: 1.2em; + /* Place it in the parent element */ + grid-column-start: 1; + grid-column-end: 2; } -#main { - grid-column: 2/3; - width: 100%; - /* background: gray; */ +#nav-sidebar .nav li a:hover { border-radius: 0.5em; } -.pagetitle { - padding-top: 0.5em; - /* background: pink; */ +#nav-sidebar .nav .activemenu { + border-style: solid; border-radius: 0.5em; - /* background-color: #6699CC; */ - /* background-color: #77AADD; */ + border-color: #AAAAAA; + background-color: #EFEFEF; +} + +#main { + /* Place it in the parent element */ + grid-column-start: 2; + grid-column-end: 3; + + /* Define layout for the children elements */ + display: grid; + grid-template-columns: 1fr; + grid-template-rows: 4em 100%; + grid-gap: 1em; +} + +#main #pagetitle { + /* Place it in the parent element */ + grid-column-start: 1; + grid-column-end: 3; + + /* Content-styling */ + border-radius: 3px; background-color: #88BBEE; } -.pagetitle h1 { - text-align: center; +#main #pagetitle .title { + font-size: 1.4em; text-transform: capitalize; + padding-left: 0.5em; +} + +@media screen and (max-width: 20in) { + #main #all-content { + /* Place it in the parent element */ + grid-column-start: 1; + grid-column-end: 3; + + /* Define layout for the children elements */ + max-width: 80%; + } + + #sidebar-content { + display: none; + } +} + +@media screen and (min-width: 20.1in) { + #main #all-content { + /* Place it in the parent element */ + grid-column-start: 1; + grid-column-end: 3; + + /* Define layout for the children elements */ + display: grid; + grid-template-columns: 7fr 3fr; + grid-gap: 1.5em; + } +} + +#main #all-content .row { + margin: 0 2px; } -.pagetitle .breadcrumb { +#main #all-content #main-content { + background: #FFFFFF; + max-width: 950px; +} + +#pagetitle .breadcrumb { background: none; + text-transform: capitalize; + font-size: 0.75em; } -.pagetitle .breadcrumb .active a { +#pagetitle .breadcrumb .active a { color: #333333; } -.pagetitle .breadcrumb a { +#pagetitle .breadcrumb a { color: #666666; } -.main-content { - font-size: 1.275em; +.heading { + border-bottom: solid #EEBB88; + text-transform: capitalize; } -.breadcrumb { +.subheading { + padding: 1em 0 0.1em 0.5em; + border-bottom: solid #88BBEE; text-transform: capitalize; } -dd { - margin-left: 3em; - font-size: 0.88em; - padding-bottom: 1em; +input[type="search"] { + border-radius: 5px; } -input[type="submit"] { - text-transform: capitalize; +.btn { + text-transform: Capitalize; } -.card { - margin-top: 0.3em; - border-width: 1px; - border-style: solid; - border-radius: 0.3em; - border-color: #AAAAAA; - padding: 0.5em; +table.dataTable thead th, table.dataTable tfoot th{ + border-right: 1px solid white; + color: white; + background-color: #369 !important; +} + +table.dataTable tbody tr.selected td { + background-color: #ffee99 !important; +} + +.form-group { + margin-bottom: 2em; + padding-bottom: 0.2em; + border-bottom: solid gray 1px; } diff --git a/uploader/static/js/datatables.js b/uploader/static/js/datatables.js new file mode 100644 index 0000000..82fd696 --- /dev/null +++ b/uploader/static/js/datatables.js @@ -0,0 +1,69 @@ +/** Handlers for events in datatables **/ + +var addTableLength = (menuList, lengthToAdd, dataLength) => { + if(dataLength >= lengthToAdd) { + newList = structuredClone(menuList);//menuList.slice(0, menuList.length); // shallow copy + newList.push(lengthToAdd); + return newList; + } + return menuList; +}; + +var defaultLengthMenu = (data) => { + menuList = [] + var lengths = [10, 25, 50, 100, 1000, data.length]; + lengths.forEach((len) => { + menuList = addTableLength(menuList, len, data.length); + }); + return menuList; +}; + +var buildDataTable = (tableId, data = [], columns = [], userSettings = {}) => { + var defaultSettings = { + responsive: true, + layout: { + topStart: null, + topEnd: null, + bottomStart: null, + bottomEnd: null, + }, + select: true, + lengthMenu: defaultLengthMenu(data), + language: { + processing: "Processing… Please wait.", + loadingRecords: "Loading table data… Please wait.", + lengthMenu: "", + info: "" + }, + data: data, + columns: columns, + drawCallback: (settings) => { + $(this[0]).find("tbody tr").each((idx, row) => { + var arow = $(row); + var checkboxOrRadio = arow.find(".chk-row-select"); + if (checkboxOrRadio) { + if (arow.hasClass("selected")) { + checkboxOrRadio.prop("checked", true); + } else { + checkboxOrRadio.prop("checked", false); + } + } + }); + } + } + var theDataTable = $(tableId).DataTable({ + ...defaultSettings, + ...userSettings + }); + theDataTable.on("select", (event, datatable, type, cell, originalEvent) => { + datatable.rows({selected: true}).nodes().each((node, index) => { + $(node).find(".chk-row-select").prop("checked", true) + }); + }); + theDataTable.on("deselect", (event, datatable, type, cell, originalEvent) => { + datatable.rows({selected: false}).nodes().each((node, index) => { + $(node).find(".chk-row-select").prop("checked", false) + }); + }); + return theDataTable; +}; diff --git a/uploader/static/js/debug.js b/uploader/static/js/debug.js new file mode 100644 index 0000000..eb01209 --- /dev/null +++ b/uploader/static/js/debug.js @@ -0,0 +1,40 @@ +/** + * The entire purpose of this function is for use to debug values inline + * without changing the flow of the code too much. + * + * This **MUST** be a non-arrow function to allow access to the `arguments` + * object. + * + * This function expects at least one argument. + * + * If more than one argument is provided, then: + * a) the last argument is considered the value, and will be returned + * b) all other arguments will be converted to string and output + * + * If only one argument is provided, it is considered the value, and will be + * returned. + * + * Zero arguments is an error condition. + **/ +function __pk__(val) { + /* Handle zero arguments */ + if (arguments.length < 1) { + throw new Error("Invalid arguments: Expected at least one argument."); + } + + msg = "/********** DEBUG **********/"; + if (arguments.length > 1) { + msg = Array.from( + arguments + ).slice( + 0, + arguments.length - 1 + ).map((val) => { + return String(val); + }).join("; ") + } + + value = arguments[arguments.length - 1]; + console.debug("/********** " + msg + " **********/", value); + return value; +} diff --git a/uploader/static/js/files.js b/uploader/static/js/files.js new file mode 100644 index 0000000..0bde6f7 --- /dev/null +++ b/uploader/static/js/files.js @@ -0,0 +1,118 @@ +var readFirstNLines = (thefile, count, process_content_fns) => { + var reader = new FileReader(); + if(typeof thefile !== "undefined" && thefile !== null) { + reader.addEventListener("load", (event) => { + var content = event + .target + .result + .split("\n") + .slice(0, count) + .map((line) => {return line.trim("\r");}); + process_content_fns.forEach((fn) => {fn(content);}); + }); + reader.readAsText(thefile); + } +}; +var read_first_n_lines = readFirstNLines; + + +var readBinaryFile = (file) => { + return new Promise((resolve, reject) => { + var _reader = new FileReader(); + _reader.onload = (event) => {resolve(_reader.result);}; + _reader.readAsArrayBuffer(file); + }); +}; + + +var Uint8ArrayToHex = (arr) => { + var toHex = (val) => { + _hex = val.toString(16); + if(_hex.length < 2) { + return "0" + val; + } + return _hex; + }; + _hexstr = "" + arr.forEach((val) => {_hexstr += toHex(val)}); + return _hexstr +}; + + +var computeFileChecksum = (file) => { + return readBinaryFile(file) + .then((content) => { + return window.crypto.subtle.digest( + "SHA-256", new Uint8Array(content)); + }).then((digest) => { + return Uint8ArrayToHex(new Uint8Array(digest)) + }); +}; + + +var defaultResumableHandler = (event) => { + throw new Error("Please provide a valid event handler!"); +}; + +var addHandler = (resumable, handlername, handler) => { + if(resumable.support) { + resumable.on(handlername, (handler || defaultResumableHandler)); + } + return resumable; +}; + + +var makeResumableHandler = (handlername) => { + return (resumable, handler) => { + return addHandler(resumable, handlername, handler); + }; +}; + + +var fileSuccessHandler = makeResumableHandler("fileSuccess"); +var fileProgressHandler = makeResumableHandler("fileProgress"); +var fileAddedHandler = makeResumableHandler("fileAdded"); +var filesAddedHandler = makeResumableHandler("filesAdded"); +var filesRetryHandler = makeResumableHandler("filesRetry"); +var filesErrorHandler = makeResumableHandler("filesError"); +var uploadStartHandler = makeResumableHandler("uploadStart"); +var completeHandler = makeResumableHandler("complete"); +var progressHandler = makeResumableHandler("progress"); +var errorHandler = makeResumableHandler("error"); + + +var markResumableDragAndDropElement = (resumable, fileinput, droparea, browsebutton) => { + if(resumable.support) { + //Hide file input element and display drag&drop UI + add_class(fileinput, "visually-hidden"); + remove_class(droparea, "visually-hidden"); + + // Define UI elements for browse and drag&drop + resumable.assignDrop(droparea); + resumable.assignBrowse(browsebutton); + } + + return resumable; +}; + + +var makeResumableElement = (targeturi, fileinput, droparea, uploadbutton, filetype) => { + var resumable = Resumable({ + target: targeturi, + fileType: filetype, + maxFiles: 1, + forceChunkSize: true, + generateUniqueIdentifier: (file, event) => { + return computeFileChecksum(file).then((checksum) => { + var _relativePath = (file.webkitRelativePath + || file.relativePath + || file.fileName + || file.name); + return checksum + "-" + _relativePath.replace( + /[^a-zA-Z0-9_-]/img, ""); + }); + } + }); + + return resumable; +}; diff --git a/uploader/static/js/populations.js b/uploader/static/js/populations.js new file mode 100644 index 0000000..be1231f --- /dev/null +++ b/uploader/static/js/populations.js @@ -0,0 +1,21 @@ +$(() => { + var populationsDataTable = buildDataTable( + "#tbl-select-population", + JSON.parse( + $("#tbl-select-population").attr("data-populations-list")), + [ + { + data: (apopulation) => { + return `<input type="radio" name="population_id"` + + `id="rdo_population_id_${apopulation.InbredSetId}" ` + + `value="${apopulation.InbredSetId}" ` + + `class="chk-row-select">`; + } + }, + { + data: (apopulation) => { + return `${apopulation.FullName} (${apopulation.InbredSetName})`; + } + } + ]); +}); diff --git a/uploader/static/js/pubmed.js b/uploader/static/js/pubmed.js new file mode 100644 index 0000000..9afd4c3 --- /dev/null +++ b/uploader/static/js/pubmed.js @@ -0,0 +1,113 @@ +var extract_details = (pubmed_id, details) => { + var months = { + "jan": "January", + "feb": "February", + "mar": "March", + "apr": "April", + "may": "May", + "jun": "June", + "jul": "July", + "aug": "August", + "sep": "September", + "oct": "October", + "nov": "November", + "dec": "December" + }; + var _date = details[pubmed_id].pubdate.split(" "); + return { + "authors": details[pubmed_id].authors.map((authobj) => { + return authobj.name; + }), + "title": details[pubmed_id].title, + "journal": details[pubmed_id].fulljournalname, + "volume": details[pubmed_id].volume, + "pages": details[pubmed_id].pages, + "month": _date.length > 1 ? months[_date[1].toLowerCase()] : "jan", + "year": _date[0], + }; +}; + +var update_publication_details = (details) => { + Object.entries(details).forEach((entry) => {; + switch(entry[0]) { + case "authors": + $("#txt-publication-authors").val(entry[1].join(", ")); + break; + case "month": + $("#select-publication-month") + .children("option") + .each((index, child) => { + console.debug(entry[1].toLowerCase()); + child.selected = child.value == entry[1].toLowerCase(); + }); + default: + $("#txt-publication-" + entry[0]).val(entry[1]); + break; + } + }); +}; + +var fetch_publication_abstract = (pubmed_id, pub_details) => { + $.ajax("https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi", + { + "method": "GET", + "data": { + "db": "pubmed", + "id": pubmed_id, + "rettype": "abstract", + "retmode": "xml" + }, + "success": (data, textStatus, jqXHR) => { + update_publication_details({ + ...pub_details, + ...{ + "abstract": Array.from(data + .getElementsByTagName( + "Abstract")[0] + .children) + .map((elt) => {return elt.textContent.trim();}) + .join("\r\n") + }}); + }, + "error": (jqXHR, textStatus, errorThrown) => {}, + "complete": (jqXHR, textStatus) => {}, + "dataType": "xml" + }); +}; + +var fetch_publication_details = (pubmed_id, complete_thunks) => { + error_display = $("#search-pubmed-id-error"); + error_display.text(""); + add_class(error_display, "visually-hidden"); + $.ajax("https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi", + { + "method": "GET", + "data": {"db": "pubmed", "id": pubmed_id, "format": "json"}, + "success": (data, textStatus, jqXHR) => { + // process and update publication details + hasError = ( + Object.hasOwn(data, "error") || + Object.hasOwn(data.result[pubmed_id], "error")); + if(hasError) { + error_display.text( + "There was an error fetching a publication with " + + "the given PubMed ID! The error received " + + "was: '" + ( + data.error || + data.result[pubmed_id].error) + + "'. Please check ID you provided and try " + + "again."); + remove_class(error_display, "visually-hidden"); + } else { + fetch_publication_abstract( + pubmed_id, + extract_details(pubmed_id, data.result)); + } + }, + "error": (jqXHR, textStatus, errorThrown) => {}, + "complete": () => { + complete_thunks.forEach((thunk) => {thunk()}); + }, + "dataType": "json" + }); +}; diff --git a/uploader/static/js/species.js b/uploader/static/js/species.js new file mode 100644 index 0000000..9ea3017 --- /dev/null +++ b/uploader/static/js/species.js @@ -0,0 +1,20 @@ +$(() => { + var speciesDataTable = buildDataTable( + "#tbl-select-species", + JSON.parse( + $("#tbl-select-species").attr("data-species-list")), + [ + { + data: (aspecies) => { + return `<input type="radio" name="species_id"` + + `id="rdo_species_id_${aspecies.SpeciesId}" ` + + `value="${aspecies.SpeciesId}" class="chk-row-select">`; + } + }, + { + data: (aspecies) => { + return `${aspecies.FullName} (${aspecies.SpeciesName})`; + } + } + ]); +}); diff --git a/uploader/static/js/utils.js b/uploader/static/js/utils.js index 045dd47..1b31661 100644 --- a/uploader/static/js/utils.js +++ b/uploader/static/js/utils.js @@ -8,3 +8,30 @@ function trigger_change_event(element) { evt = new Event("change"); element.dispatchEvent(evt); } + + +var remove_class = (element, classvalue) => { + new_classes = (element.attr("class") || "").split(" ").map((val) => { + return val.trim(); + }).filter((val) => { + return ((val !== classvalue) && + (val !== "")) + }).join(" "); + + if(new_classes === "") { + element.removeAttr("class"); + } else { + element.attr("class", new_classes); + } +}; + + +var add_class = (element, classvalue) => { + remove_class(element, classvalue); + element.attr("class", (element.attr("class") || "") + " " + classvalue); +}; + +$(".not-implemented").click((event) => { + event.preventDefault(); + alert("This feature is not implemented yet. Please bear with us."); +}); diff --git a/uploader/templates/base.html b/uploader/templates/base.html index d68c6c0..3c0d0d4 100644 --- a/uploader/templates/base.html +++ b/uploader/templates/base.html @@ -8,14 +8,14 @@ <meta name="viewport" content="width=device-width, initial-scale=1.0" /> {%block extrameta%}{%endblock%} - <title>GN Uploader: {%block title%}{%endblock%}</title> + <title>Data Upload and Quality Control: {%block title%}{%endblock%}</title> <link rel="stylesheet" type="text/css" href="{{url_for('base.bootstrap', filename='css/bootstrap.min.css')}}" /> <link rel="stylesheet" type="text/css" - href="{{url_for('base.bootstrap', - filename='css/bootstrap-theme.min.css')}}" /> + href="{{url_for('base.datatables', + filename='css/dataTables.bootstrap5.min.css')}}" /> <link rel="stylesheet" type="text/css" href="/static/css/styles.css" /> {%block css%}{%endblock%} @@ -23,83 +23,139 @@ </head> <body> - <header id="header" class="container-fluid"> - <div class="row"> - <span class="header col-lg-9">GeneNetwork Data Quality Control and Upload</span> - <nav class="header-nav col-lg-3"> - <ul class="nav justify-content-end"> - <li class="btn"> - {%if user_logged_in()%} - <a href="{{url_for('oauth2.logout')}}" - title="Log out of the system">Log Out</a> - {%else%} - <a href="{{authserver_authorise_uri()}}" - title="Log in to the system">Log In</a> - {%endif%} - </li> - </ul> - </nav> + <header id="header"> + <span id="header-text">GeneNetwork</span> + <nav id="header-nav"> + <ul class="nav justify-content-end"> + <li> + {%if user_logged_in()%} + <a href="{{url_for('oauth2.logout')}}" + title="Log out of the system"> + <span class="glyphicon glyphicon-user"></span> + {{user_email()}} Sign Out</a> + {%else%} + <a href="{{authserver_authorise_uri()}}" + title="Log in to the system">Sign In</a> + {%endif%} + </li> + </ul> + </nav> </header> - <aside id="nav-sidebar" class="container-fluid"> + <aside id="nav-sidebar"> <ul class="nav flex-column"> - <li><a href="/" >Home</a></li> - <li><a href="{{url_for('species.list_species')}}" - title="View and manage species information.">Species</a></li> - <li><a href="#" - title="Upload Genotype data.">Genotype Data</a></li> - <li><a href="{{url_for('species.populations.index')}}" - title="View and manage species populations.">Populations</a></li> - <li><a href="{{url_for('species.populations.samples.index')}}" - title="Upload population samples.">Samples</a></li> - <li><a href="{{url_for('expression-data.index.index')}}" - title="Upload expression data.">Expression Data</a></li> - <li><a href="#" - title="Upload phenotype data.">Phenotype Data</a></li> - <li><a href="#" - title="Upload individual data.">Individual Data</a></li> - <li><a href="#" - title="Upload RNA-Seq data.">RNA-Seq Data</a></li> + <li {%if activemenu=="home"%}class="activemenu"{%endif%}> + <a href="/" >Home</a></li> + <li {%if activemenu=="publications"%}class="activemenu"{%endif%}> + <a href="{{url_for('publications.index')}}" + title="View and manage publications.">Publications</a></li> + <li {%if activemenu=="species"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.list_species')}}" + title="View and manage species information.">Species</a></li> + <li {%if activemenu=="platforms"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.platforms.index')}}" + title="View and manage species platforms.">Sequencing Platforms</a></li> + <li {%if activemenu=="populations"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.populations.index')}}" + title="View and manage species populations.">Populations</a></li> + <li {%if activemenu=="samples"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.populations.samples.index')}}" + title="Upload population samples.">Samples</a></li> + <li {%if activemenu=="genotypes"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.populations.genotypes.index')}}" + title="Upload Genotype data.">Genotype Data</a></li> + <!-- + TODO: Maybe include menus here for managing studies and dataset or + maybe have the studies/datasets managed under their respective + sections, e.g. "Publish*" studies/datasets under the "Phenotypes" + section, "ProbeSet*" studies/datasets under the "Expression Data" + sections, etc. + --> + <li {%if activemenu=="phenotypes"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.populations.phenotypes.index')}}" + title="Upload phenotype data.">Phenotype Data</a></li> + <!-- + <li {%if activemenu=="expression-data"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.populations.expression-data.index')}}" + title="Upload expression data." + class="not-implemented">Expression Data</a></li> + <li {%if activemenu=="individuals"%}class="activemenu"{%endif%}> + <a href="#" + class="not-implemented" + title="Upload individual data.">Individual Data</a></li> + <li {%if activemenu=="rna-seq"%}class="activemenu"{%endif%}> + <a href="#" + class="not-implemented" + title="Upload RNA-Seq data.">RNA-Seq Data</a></li> + <li {%if activemenu=="async-jobs"%}class="activemenu"{%endif%}> + <a href="#" + class="not-implemented" + title="View and manage the backgroud jobs you have running"> + Background Jobs</a></li> + --> </ul> </aside> - <main id="main" class="main container-fluid"> + <main id="main" class="main"> - <div class="pagetitle row"> - <h1>GN Uploader: {%block pagetitle%}{%endblock%}</h1> - <nav> - <ol class="breadcrumb"> - <li {%if activelink is not defined or activelink=="home"%} - class="breadcrumb-item active" - {%else%} - class="breadcrumb-item" - {%endif%}> - <a href="{{url_for('base.index')}}">Home</a> - </li> - {%block lvl1_breadcrumbs%}{%endblock%} - </ol> - </nav> + <div id="pagetitle" class="pagetitle"> + <span class="title">Data Upload and Quality Control: {%block pagetitle%}{%endblock%}</span> + <!-- + <nav> + <ol class="breadcrumb"> + <li {%if activelink is not defined or activelink=="home"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('base.index')}}">Home</a> + </li> + {%block lvl1_breadcrumbs%}{%endblock%} + </ol> + </nav> + --> </div> - <div class="row"> - <div class="container-fluid"> - <div class="col-md-8 main-content"> - {%block contents%}{%endblock%} - </div> - <div class="sidebar-content col-md-4"> - {%block sidebarcontents%}{%endblock%} - </div> + <div id="all-content"> + <div id="main-content"> + {%block contents%}{%endblock%} + </div> + <div id="sidebar-content"> + {%block sidebarcontents%}{%endblock%} </div> </div> </main> + <!-- + Core dependencies + --> <script src="{{url_for('base.jquery', filename='jquery.min.js')}}"></script> <script src="{{url_for('base.bootstrap', filename='js/bootstrap.min.js')}}"></script> - {%block javascript%}{%endblock%} - </body> + <!-- + DataTables dependencies + --> + <script type="text/javascript" + src="{{url_for('base.datatables', + filename='js/dataTables.min.js')}}"></script> + <script type="text/javascript" + src="{{url_for('base.datatables_extensions', + filename='scroller/js/dataTables.scroller.min.js')}}"></script> + <script type="text/javascript" + src="{{url_for('base.datatables_extensions', + filename='buttons/js/dataTables.buttons.min.js')}}"></script> + <script type="text/javascript" + src="{{url_for('base.datatables_extensions', + filename='select/js/dataTables.select.min.js')}}"></script> + <!-- + local dependencies + --> + <script type="text/javascript" src="/static/js/utils.js"></script> + <script type="text/javascript" src="/static/js/datatables.js"></script> + {%block javascript%}{%endblock%} + </body> </html> diff --git a/uploader/templates/cli-output.html b/uploader/templates/cli-output.html index 33fb73b..64b1a9a 100644 --- a/uploader/templates/cli-output.html +++ b/uploader/templates/cli-output.html @@ -1,7 +1,7 @@ {%macro cli_output(job, stream)%} -<h4>{{stream | upper}} Output</h4> -<div class="cli-output"> +<h4 class="subheading">{{stream | upper}} Output</h4> +<div class="cli-output" style="max-height: 10em; overflow: auto;"> <pre>{{job.get(stream, "")}}</pre> </div> diff --git a/uploader/templates/expression-data/base.html b/uploader/templates/expression-data/base.html new file mode 100644 index 0000000..d63fd7e --- /dev/null +++ b/uploader/templates/expression-data/base.html @@ -0,0 +1,13 @@ +{%extends "populations/base.html"%} + +{%block lvl3_breadcrumbs%} +<li {%if activelink=="expression-data"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.expression-data.index')}}"> + Expression Data</a> +</li> +{%block lvl4_breadcrumbs%}{%endblock%} +{%endblock%} diff --git a/uploader/templates/data_review.html b/uploader/templates/expression-data/data-review.html index 4e5c586..c985b03 100644 --- a/uploader/templates/data_review.html +++ b/uploader/templates/expression-data/data-review.html @@ -26,7 +26,7 @@ <small class="text-muted"> If you encounter an error saying your sample(s)/case(s) do not exist in the GeneNetwork database, then you will have to use the - <a href="{{url_for('expression-data.samples.select_species')}}" + <a href="{{url_for('species.populations.samples.index')}}" title="Upload samples/cases feature">Upload Samples/Cases</a> option on this system to upload them. </small> @@ -70,8 +70,8 @@ column</li> <li>The values of each field <strong>ARE NOT</strong> quoted.</li> <li>Here is an - <a href="https://gitlab.com/fredmanglis/gnqc_py/-/blob/main/tests/test_data/no_data_errors.tsv"> - example file</a> with a single data row.</li> + <a href="https://gitlab.com/fredmanglis/gnqc_py/-/blob/main/tests/test_data/no_data_errors.tsv" + target="_blank">example file</a> with a single data row.</li> </ul> </li> <li>.txt files: Content has the same format as .tsv file above</li> diff --git a/uploader/templates/expression-data/index.html b/uploader/templates/expression-data/index.html index ed5d8dd..9ba3582 100644 --- a/uploader/templates/expression-data/index.html +++ b/uploader/templates/expression-data/index.html @@ -1,5 +1,6 @@ -{%extends "base.html"%} +{%extends "expression-data/base.html"%} {%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-select-species.html" import select_species_form%} {%block title%}Expression Data{%endblock%} @@ -10,86 +11,23 @@ <a href="{{url_for('base.index')}}">Home</a> </li> <li class="breadcrumb-item active"> - <a href="{{url_for('expression-data.index.index')}}">Expression Data</a> + <a href="{{url_for('species.populations.expression-data.index')}}" + title="Upload expression data."> + Expression Data</a> </li> {%endblock%} {%block contents%} <div class="row"> - {{flash_all_messages()}} - - <h1 class="heading">data upload</h1> - - <div class="explainer"> - <p>Each of the sections below gives you a different option for data expression-data. - Please read the documentation for each section carefully to understand what - each section is about.</p> - </div> -</div> - -<div class="row"> - <h2 class="heading">R/qtl2 Bundles</h2> - - <div class="explainer"> - <p>This feature combines and extends the two upload methods below. Instead of - uploading one item at a time, the R/qtl2 bundle you upload can contain both - the genotypes data (samples/individuals/cases and their data) and the - expression data.</p> - <p>The R/qtl2 bundle, additionally, can contain extra metadata, that neither - of the methods below can handle.</p> - - <a href="{{url_for('expression-data.rqtl2.select_species')}}" - title="Upload a zip bundle of R/qtl2 files"> - <button class="btn btn-primary">upload R/qtl2 bundle</button></a> - </div> -</div> - - -<div class="row"> <h2 class="heading">Expression Data</h2> + {{flash_all_messages()}} - <div class="explainer"> - <p>This feature enables you to upload expression data. It expects the data to - be in <strong>tab-separated values (TSV)</strong> files. The data should be - a simple matrix of <em>phenotype × sample</em>, i.e. The first column is a - list of the <em>phenotypes</em> and the first row is a list of - <em>samples/cases</em>.</p> - - <p>If you haven't done so please go to this page to learn the requirements for - file formats and helpful suggestions to enter your data in a fast and easy - way.</p> - - <ol> - <li><strong>PLEASE REVIEW YOUR DATA.</strong>Make sure your data complies - with our system requirements. ( - <a href="{{url_for('expression-data.index.data_review')}}#data-concerns" - title="Details for the data expectations.">Help</a> - )</li> - <li><strong>UPLOAD YOUR DATA FOR DATA VERIFICATION.</strong> We accept - <strong>.csv</strong>, <strong>.txt</strong> and <strong>.zip</strong> - files (<a href="{{url_for('expression-data.index.data_review')}}#file-types" - title="Details for the data expectations.">Help</a>)</li> - </ol> - </div> - - <a href="{{url_for('expression-data.index.upload_file')}}" - title="Upload your expression data" - class="btn btn-primary">upload expression data</a> + <p>This section allows you to enter the expression data for your experiment. + You will need to select the species that your data concerns below.</p> </div> <div class="row"> - <h2 class="heading">samples/cases</h2> - - <div class="explainer"> - <p>For the expression data above, you need the samples/cases in your file to - already exist in the GeneNetwork database. If there are any samples that do - not already exist the upload of the expression data will fail.</p> - <p>This section gives you the opportunity to upload any missing samples</p> - </div> - - <a href="{{url_for('expression-data.samples.select_species')}}" - title="Upload samples/cases/individuals for your data" - class="btn btn-primary">upload Samples/Cases</a> + {{select_species_form(url_for("species.populations.expression-data.index"), + species)}} </div> - {%endblock%} diff --git a/uploader/templates/job_progress.html b/uploader/templates/expression-data/job-progress.html index 2feaa89..ef264e1 100644 --- a/uploader/templates/job_progress.html +++ b/uploader/templates/expression-data/job-progress.html @@ -1,5 +1,6 @@ {%extends "base.html"%} {%from "errors_display.html" import errors_display%} +{%from "populations/macro-display-population-card.html" import display_population_card%} {%block extrameta%} <meta http-equiv="refresh" content="5"> @@ -11,7 +12,9 @@ <h1 class="heading">{{job_name}}</h2> <div class="row"> - <form action="{{url_for('expression-data.parse.abort')}}" method="POST"> + <form action="{{url_for('species.populations.expression-data.abort', + species_id=species.SpeciesId, + population_id=population.Id)}}" method="POST"> <legend class="heading">Status</legend> <div class="form-group"> <label for="job_status" class="form-label">status:</label> @@ -38,3 +41,7 @@ </div> {%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/no_such_job.html b/uploader/templates/expression-data/no-such-job.html index 874d047..d22c429 100644 --- a/uploader/templates/no_such_job.html +++ b/uploader/templates/expression-data/no-such-job.html @@ -1,7 +1,8 @@ {%extends "base.html"%} {%block extrameta%} -<meta http-equiv="refresh" content="5;url={{url_for('expression-data.index.upload_file')}}"> +<meta http-equiv="refresh" + content="5;url={{url_for('species.populations.expression-data.index.upload_file')}}"> {%endblock%} {%block title%}No Such Job{%endblock%} diff --git a/uploader/templates/parse_failure.html b/uploader/templates/expression-data/parse-failure.html index 31f6be8..31f6be8 100644 --- a/uploader/templates/parse_failure.html +++ b/uploader/templates/expression-data/parse-failure.html diff --git a/uploader/templates/expression-data/parse-results.html b/uploader/templates/expression-data/parse-results.html new file mode 100644 index 0000000..03a23e2 --- /dev/null +++ b/uploader/templates/expression-data/parse-results.html @@ -0,0 +1,39 @@ +{%extends "base.html"%} +{%from "errors_display.html" import errors_display%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Parse Results{%endblock%} + +{%block contents%} + +<div class="row"> + <h2 class="heading">{{job_name}}: parse results</h2> + + {%if user_aborted%} + <span class="alert-warning">Job aborted by the user</span> + {%endif%} + + {{errors_display(errors, "No errors found in the file", "We found the following errors", True)}} + + {%if errors | length == 0 and not user_aborted %} + <form method="post" action="{{url_for('dbinsert.select_platform')}}"> + <input type="hidden" name="job_id" value="{{job_id}}" /> + <input type="submit" value="update database" class="btn btn-primary" /> + </form> + {%endif%} + + {%if errors | length > 0 or user_aborted %} + <br /> + <a href="{{url_for('species.populations.expression-data.upload_file', + species_id=species.SpeciesId, + population_id=population.Id)}}" + title="Back to index page." + class="btn btn-primary">Go back</a> + + {%endif%} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/expression-data/select-file.html b/uploader/templates/expression-data/select-file.html new file mode 100644 index 0000000..4ca461e --- /dev/null +++ b/uploader/templates/expression-data/select-file.html @@ -0,0 +1,115 @@ +{%extends "expression-data/base.html"%} +{%from "flash_messages.html" import flash_messages%} +{%from "upload_progress_indicator.html" import upload_progress_indicator%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Expression Data — Upload Data{%endblock%} + +{%block pagetitle%}Expression Data — Upload Data{%endblock%} + +{%block contents%} +{{upload_progress_indicator()}} + +<div class="row"> + <h2 class="heading">Upload Expression Data</h2> + + <p>This feature enables you to upload expression data. It expects the data to + be in <strong>tab-separated values (TSV)</strong> files. The data should be + a simple matrix of <em>phenotype × sample</em>, i.e. The first column is a + list of the <em>phenotypes</em> and the first row is a list of + <em>samples/cases</em>.</p> + + <p>If you haven't done so please go to this page to learn the requirements for + file formats and helpful suggestions to enter your data in a fast and easy + way.</p> + + <ol> + <li><strong>PLEASE REVIEW YOUR DATA.</strong>Make sure your data complies + with our system requirements. ( + <a href="{{url_for('species.populations.expression-data.data_review')}}#data-concerns" + title="Details for the data expectations.">Help</a> + )</li> + <li><strong>UPLOAD YOUR DATA FOR DATA VERIFICATION.</strong> We accept + <strong>.csv</strong>, <strong>.txt</strong> and <strong>.zip</strong> + files (<a href="{{url_for('species.populations.expression-data.data_review')}}#file-types" + title="Details for the data expectations.">Help</a>)</li> + </ol> +</div> + +<div class="row"> + <form action="{{url_for( + 'species.populations.expression-data.upload_file', + species_id=species.SpeciesId, + population_id=population.Id)}}" + method="POST" + enctype="multipart/form-data" + id="frm-upload-expression-data"> + {{flash_messages("error-expr-data")}} + + <div class="form-group"> + <legend class="heading">File Type</legend> + + <div class="radio"> + <label for="filetype_average" class="form-check-label"> + <input type="radio" name="filetype" value="average" id="filetype_average" + required="required" class="form-check-input" /> + Average</label> + <p class="form-text text-muted"> + <small>The averages data …</small></p> + </div> + + <div class="radio"> + <label for="filetype_standard_error" class="form-check-label"> + <input type="radio" name="filetype" value="standard-error" + id="filetype_standard_error" required="required" + class="form-check-input" /> + Standard Error + </label> + <p class="form-text text-muted"> + <small>The standard errors computed from the averages …</small></p> + </div> + </div> + + <div class="form-group"> + <span id="no-file-error" class="alert-danger" style="display: none;"> + No file selected + </span> + <label for="file_upload" class="form-label">Select File</label> + <input type="file" name="qc_text_file" id="file_upload" + accept="text/plain, text/tab-separated-values, application/zip" + class="form-control"/> + <p class="form-text text-muted"> + <small>Select the file to upload.</small></p> + </div> + + <button type="submit" + class="btn btn-primary" + data-toggle="modal" + data-target="#upload-progress-indicator">upload file</button> + </form> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} + +{%block javascript%} +<script type="text/javascript" src="/static/js/upload_progress.js"></script> +<script type="text/javascript"> + function setup_formdata(form) { + var formdata = new FormData(); + formdata.append( + "qc_text_file", + form.querySelector("input[type='file']").files[0]); + formdata.append( + "filetype", + selected_filetype( + Array.from(form.querySelectorAll("input[type='radio']")))); + return formdata; + } + + setup_upload_handlers( + "frm-upload-expression-data", make_data_uploader(setup_formdata)); +</script> +{%endblock%} diff --git a/uploader/templates/expression-data/select-population.html b/uploader/templates/expression-data/select-population.html new file mode 100644 index 0000000..8555e27 --- /dev/null +++ b/uploader/templates/expression-data/select-population.html @@ -0,0 +1,29 @@ +{%extends "expression-data/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} +{%from "populations/macro-select-population.html" import select_population_form%} + +{%block title%}Expression Data{%endblock%} + +{%block pagetitle%}Expression Data{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>You have selected the species. Now you need to select the population that + the expression data belongs to.</p> +</div> + +<div class="row"> + {{select_population_form(url_for( + "species.populations.expression-data.select_population", + species_id=species.SpeciesId), + populations)}} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} diff --git a/uploader/templates/genotypes/base.html b/uploader/templates/genotypes/base.html new file mode 100644 index 0000000..7d61312 --- /dev/null +++ b/uploader/templates/genotypes/base.html @@ -0,0 +1,23 @@ +{%extends "populations/base.html"%} + +{%block lvl3_breadcrumbs%} +<li {%if activelink=="genotypes"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + {%if population is mapping%} + <a href="{{url_for('species.populations.genotypes.list_genotypes', + species_id=species.SpeciesId, + population_id=population.Id)}}"> + {%if dataset is defined and dataset is mapping%} + {{dataset.Name}} + {%else%} + Genotypes + {%endif%}</a> + {%else%} + <a href="{{url_for('species.populations.genotypes.index')}}">Genotypes</a> + {%endif%} +</li> +{%block lvl4_breadcrumbs%}{%endblock%} +{%endblock%} diff --git a/uploader/templates/genotypes/create-dataset.html b/uploader/templates/genotypes/create-dataset.html new file mode 100644 index 0000000..10331c1 --- /dev/null +++ b/uploader/templates/genotypes/create-dataset.html @@ -0,0 +1,82 @@ +{%extends "genotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Genotypes — Create Dataset{%endblock%} + +{%block pagetitle%}Genotypes — Create Dataset{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="create-dataset"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.genotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}">Create Dataset</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <form id="frm-geno-create-dataset" + method="POST" + action="{{url_for('species.populations.genotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}"> + <legend>Create a new Genotype Dataset</legend> + + <div class="form-group"> + <label for="txt-geno-dataset-name" class="form-label">Name</label> + <input type="text" + id="txt-geno-dataset-name" + name="geno-dataset-name" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>This is a short representative, but constrained name for the genotype + dataset.<br /> + The field will only accept letters ('A-Za-z'), numbers (0-9), hyphens + and underscores. Any other character will cause the name to be + rejected.</p></small> + </div> + + <div class="form-group"> + <label for="txt-geno-dataset-fullname" class="form-label">Full Name</label> + <input type="text" + id="txt-geno-dataset-fullname" + name="geno-dataset-fullname" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>This is a longer, more descriptive name for your dataset.</p></small> + </div> + + <div class="form-group"> + <label for="txt-geno-dataset-shortname" + class="form-label">Short Name</label> + <input type="text" + id="txt-geno-dataset-shortname" + name="geno-dataset-shortname" + class="form-control" /> + <small class="form-text text-muted"> + <p>A short name for your dataset. If you leave this field blank, the + short name will be set to the same value as the + "<strong>Name</strong>" field above.</p></small> + </div> + + <div class="form-group"> + <input type="submit" + class="btn btn-primary" + value="create dataset" /> + </div> + </form> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/genotypes/index.html b/uploader/templates/genotypes/index.html new file mode 100644 index 0000000..b50ebc5 --- /dev/null +++ b/uploader/templates/genotypes/index.html @@ -0,0 +1,32 @@ +{%extends "genotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-select-species.html" import select_species_form%} + +{%block title%}Genotypes{%endblock%} + +{%block pagetitle%}Genotypes{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p> + This section allows you to upload genotype information for your experiments, + in the case that you have not previously done so. + </p> + <p> + We'll need to link the genotypes to the species and population, so do please + go ahead and select those in the next two steps. + </p> +</div> + +<div class="row"> + {{select_species_form(url_for("species.populations.genotypes.index"), + species)}} +</div> +{%endblock%} + +{%block javascript%} +<script type="text/javascript" src="/static/js/species.js"></script> +{%endblock%} diff --git a/uploader/templates/genotypes/list-genotypes.html b/uploader/templates/genotypes/list-genotypes.html new file mode 100644 index 0000000..0f074fd --- /dev/null +++ b/uploader/templates/genotypes/list-genotypes.html @@ -0,0 +1,149 @@ +{%extends "genotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Genotypes{%endblock%} + +{%block pagetitle%}Genotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="list-genotypes"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.genotypes.list_genotypes', + species_id=species.SpeciesId, + population_id=population.Id)}}">List genotypes</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <h2>Genetic Markers</h2> + <p>There are a total of {{total_markers}} currently registered genetic markers + for the "{{species.FullName}}" species. You can click + <a href="{{url_for('species.populations.genotypes.list_markers', + species_id=species.SpeciesId, + population_id=population.Id)}}" + title="View genetic markers for species '{{species.FullName}}"> + this link to view the genetic markers + </a>. + </p> +</div> + +<div class="row"> + <h2>Genotype Encoding</h2> + <p> + The genotype encoding used for the "{{population.FullName}}" population from + the "{{species.FullName}}" species is as shown in the table below. + </p> + <table class="table"> + + <thead> + <tr> + <th>Allele Type</th> + <th>Allele Symbol</th> + <th>Allele Value</th> + </tr> + </thead> + + <tbody> + {%for row in genocode%} + <tr> + <td>{{row.AlleleType}}</td> + <td>{{row.AlleleSymbol}}</td> + <td>{{row.DatabaseValue if row.DatabaseValue is not none else "NULL"}}</td> + </tr> + {%else%} + <tr> + <td colspan="7" class="text-info"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + There is no explicit genotype encoding defined for this population. + </td> + </tr> + {%endfor%} + </tbody> + </table> + + {%if genocode | length < 1%} + <a href="#add-genotype-encoding" + title="Add a genotype encoding system for this population" + class="btn btn-primary not-implemented"> + add genotype encoding + </a> + {%endif%} +</div> + +<div class="row text-danger"> + <h3>Some Important Concepts to Consider/Remember</h3> + <ul> + <li>Reference vs. Non-reference alleles</li> + <li>In <em>GenoCode</em> table, items are ordered by <strong>InbredSet</strong></li> + </ul> + <h3>Possible references</h3> + <ul> + <li>https://mr-dictionary.mrcieu.ac.uk/term/genotype/</li> + <li>https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7363099/</li> + </ul> +</div> + +<div class="row"> + <h2>Genotype Datasets</h2> + + <p>The genotype data is organised under various genotype datasets. You can + click on the link for the relevant dataset to view a little more information + about it.</p> + + {%if dataset is not none%} + <table class="table"> + <thead> + <tr> + <th>Name</th> + <th>Full Name</th> + </tr> + </thead> + + <tbody> + <tr> + <td>{{dataset.Name}}</td> + <td><a href="{{url_for('species.populations.genotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}" + title="View details regarding and manage dataset '{{dataset.FullName}}'"> + {{dataset.FullName}}</a></td> + </tr> + </tbody> + </table> + {%else%} + <p class="text-warning"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + There is no genotype dataset defined for this population. + </p> + <p> + <a href="{{url_for('species.populations.genotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}" + title="Create a new genotype dataset for the '{{population.FullName}}' population for the '{{species.FullName}}' species." + class="btn btn-primary"> + create new genotype dataset</a></p> + {%endif%} +</div> +<div class="row text-warning"> + <p> + <span class="glyphicon glyphicon-exclamation-sign"></span> + <strong>NOTE</strong>: Currently the GN2 (and related) system(s) expect a + single genotype dataset. If there is more than one, the system apparently + fails in unpredictable ways. + </p> + <p>Fix this to allow multiple datasets, each with a different assembly from + all the rest.</p> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/genotypes/list-markers.html b/uploader/templates/genotypes/list-markers.html new file mode 100644 index 0000000..a705ae3 --- /dev/null +++ b/uploader/templates/genotypes/list-markers.html @@ -0,0 +1,105 @@ +{%extends "genotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} + +{%block title%}Genotypes: List Markers{%endblock%} + +{%block pagetitle%}Genotypes: List Markers{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="list-markers"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.genotypes.list_markers', + species_id=species.SpeciesId, + population_id=population.Id)}}">List markers</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +{%if markers | length > 0%} +<div class="row"> + <p> + There are a total of {{total_markers}} genotype markers for this species. + </p> + <div class="row"> + <div class="col-md-2" style="text-align: start;"> + {%if start_from > 0%} + <a href="{{url_for('species.populations.genotypes.list_markers', + species_id=species.SpeciesId, + population_id=population.Id, + start_from=start_from-count, + count=count)}}"> + <span class="glyphicon glyphicon-backward"></span> + Previous + </a> + {%endif%} + </div> + <div class="col-md-8" style="text-align: center;"> + Displaying markers {{start_from+1}} to {{start_from+count if start_from+count < total_markers else total_markers}} of + {{total_markers}} + </div> + <div class="col-md-2" style="text-align: end;"> + {%if start_from + count < total_markers%} + <a href="{{url_for('species.populations.genotypes.list_markers', + species_id=species.SpeciesId, + population_id=population.Id, + start_from=start_from+count, + count=count)}}"> + Next + <span class="glyphicon glyphicon-forward"></span> + </a> + {%endif%} + </div> + </div> + <table class="table"> + <thead> + <tr> + <th title="">#</th> + <th title="">Marker Name</th> + <th title="Chromosome">Chr</th> + <th title="Physical location of the marker in megabasepairs"> + Location (Mb)</th> + <th title="">Source</th> + <th title="">Source2</th> + </thead> + + <tbody> + {%for marker in markers%} + <tr> + <td>{{marker.sequence_number}}</td> + <td>{{marker.Marker_Name}}</td> + <td>{{marker.Chr}}</td> + <td>{{marker.Mb}}</td> + <td>{{marker.Source}}</td> + <td>{{marker.Source2}}</td> + </tr> + {%endfor%} + </tbody> + </table> +</div> +{%else%} +<div class="row"> + <p class="text-warning"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + This species does not currently have any genetic markers uploaded, therefore, + there is nothing to display here. + </p> + <p> + <a href="#add-genetic-markers-for-species-{{species.SpeciesId}}" + title="Add genetic markers for this species" + class="btn btn-primary"> + add genetic markers + </a> + </p> +</div> +{%endif%} +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} diff --git a/uploader/templates/genotypes/select-population.html b/uploader/templates/genotypes/select-population.html new file mode 100644 index 0000000..acdd063 --- /dev/null +++ b/uploader/templates/genotypes/select-population.html @@ -0,0 +1,25 @@ +{%extends "genotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} +{%from "populations/macro-select-population.html" import select_population_form%} + +{%block title%}Genotypes{%endblock%} + +{%block pagetitle%}Genotypes{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + {{select_population_form(url_for("species.populations.genotypes.select_population", species_id=species.SpeciesId), species, populations)}} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} + +{%block javascript%} +<script type="text/javascript" src="/static/js/populations.js"></script> +{%endblock%} diff --git a/uploader/templates/genotypes/view-dataset.html b/uploader/templates/genotypes/view-dataset.html new file mode 100644 index 0000000..e7ceb36 --- /dev/null +++ b/uploader/templates/genotypes/view-dataset.html @@ -0,0 +1,61 @@ +{%extends "genotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Genotypes: View Dataset{%endblock%} + +{%block pagetitle%}Genotypes: View Dataset{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="view-dataset"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.genotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">view dataset</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <h2>Genotype Dataset Details</h2> + <table class="table"> + <thead> + <tr> + <th>Name</th> + <th>Full Name</th> + </tr> + </thead> + + <tbody> + <tr> + <td>{{dataset.Name}}</td> + <td>{{dataset.FullName}}</td> + </tr> + </tbody> + </table> +</div> + +<div class="row text-warning"> + <h2>Assembly Details</h2> + + <p>Maybe include the assembly details here if found to be necessary.</p> +</div> + +<div class="row"> + <h2>Genotype Data</h2> + + <p class="text-danger"> + Provide link to enable uploading of genotype data here.</p> +</div> + +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/index.html b/uploader/templates/index.html index e3f5af4..aa1414e 100644 --- a/uploader/templates/index.html +++ b/uploader/templates/index.html @@ -5,23 +5,103 @@ {%block pagetitle%}Home{%endblock%} -{%block breadcrumb%} -<li class="breadcrumb-item active"> - <a href="{{url_for('base.index')}}">Home</a> -</li> -{%endblock%} - {%block contents%} <div class="row"> {{flash_all_messages()}} <div class="explainer"> - <p>Welcome to the <strong>GeneNetwork Data Quality Control and Upload System</strong>. This system is provided to help in uploading your data onto GeneNetwork where you can do analysis on it.</p> + <p>Welcome to the <strong>GeneNetwork Data Upload and Quality Control + System</strong>.</p> + <p>This tool helps you prepare and upload research data to GeneNetwork for + analysis.</p> - <p>Click on the menu items on the left to select the kind of data you want to upload.</p> + <h2 class="heading">Getting Started</h2> + <p>The sections below explain the features of the system. Review this guide + to learn how to use the system.</p> {%block extrapageinfo%}{%endblock%} + + <h3 class="subheading">Species</h3> + + <p>GeneNetwork supports genetic studies across multiple species (e.g. mice + [Mus musculus], human [homo sapiens], rats [Rattus norvegicus], etc.) . + Here you can:</p> + <ul> + <li>View all species that are currently supported</li> + <li>Add new species not yet in the system</li> + </ul> + + <h3 class="subheading">Populations</h3> + + <p>A "population" refers to a specific subgroup within a species that you’re + studying (e.g., BXD mice). Here you can:</p> + <ul> + <li>View the populations that exist for a selected species</li> + <li>Add new populations of study for a selected species</li> + </ul> + + <h3 class="subheading">Samples</h3> + + <p>Manage individual specimens or cases used in your experiments. These + include:</p> + + <ul> + <li>Experimental subjects</li> + <li>Data sources (e.g., tissue samples, clinical cases)</li> + <li>Strain means (instead of entering multiple BXD1 individuals, for + example, the mean would be entered for a single BXD1 strain)</li> + </ul> + + + <h3 class="subheading">Genotype Data</h3> + + <p>Upload and review genetic markers and allele encodings for your + population. Key details:</p> + + <ul> + <li>Markers are species-level (e.g., mouse SNP databases).</li> + <li>Allele data is population-specific (tied to your experimental + samples).</li> + </ul> + + <p><strong>Requirement</strong>: Samples must already have been registered + in the system before uploading genotype data.</p> + + <h3 class="subheading">Phenotype Data</h3> + + <p>Phenotypes are the visible traits or features of a living thing. For + example, phenotypes include:</p> + + <ul> + <li>Weight</li> + <li>Height</li> + <li>Color (such as the color of fur or eyes)</li> + </ul> + + <p>This part of the system will allow you to upload and manage the values + for different phenotypes from various samples in your studies.</p> + + <!-- + + <h3 class="subheading">Expression Data</h3> + + <p class="text-danger"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + <strong>TODO</strong>: Document this …</p> + + <h3 class="subheading">Individual Data</h3> + + <p class="text-danger"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + <strong>TODO</strong>: Document this …</p> + + <h3 class="subheading">RNA-Seq Data</h3> + + <p class="text-danger"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + <strong>TODO</strong>: Document this …</p> </div> + --> </div> {%endblock%} diff --git a/uploader/templates/jobs/job-error.html b/uploader/templates/jobs/job-error.html new file mode 100644 index 0000000..b3015fc --- /dev/null +++ b/uploader/templates/jobs/job-error.html @@ -0,0 +1,17 @@ +{%extends "base.html"%} + +{%from "flash_messages.html" import flash_all_messages%} + +{%block title%}Background Jobs: Error{%endblock%} + +{%block pagetitle%}Background Jobs: Error{%endblock%} + +{%block contents%} + +<h1>Background Jobs: Error</h1> +<p>Job <strong>{{job["job_id"]}}</strong> failed!</p> +<p>The error details are in the "STDERR" section below.</p> + +<h2>STDERR</h2> +<pre>{{job["stderr"]}}</pre> +{%endblock%} diff --git a/uploader/templates/jobs/job-not-found.html b/uploader/templates/jobs/job-not-found.html new file mode 100644 index 0000000..a71e66f --- /dev/null +++ b/uploader/templates/jobs/job-not-found.html @@ -0,0 +1,11 @@ +{%extends "base.html"%} + +{%from "flash_messages.html" import flash_all_messages%} + +{%block title%}Background Jobs{%endblock%} + +{%block pagetitle%}Background Jobs{%endblock%} + +{%block contents%} +<p>Could not find job with ID: {{job_id}}</p> +{%endblock%} diff --git a/uploader/templates/jobs/job-status.html b/uploader/templates/jobs/job-status.html new file mode 100644 index 0000000..83c02fd --- /dev/null +++ b/uploader/templates/jobs/job-status.html @@ -0,0 +1,24 @@ +{%extends "base.html"%} + +{%from "flash_messages.html" import flash_all_messages%} + +{%block extrameta%} +<meta http-equiv="refresh" content="5" /> +{%endblock%} + +{%block title%}Background Jobs{%endblock%} + +{%block pagetitle%}Background Jobs{%endblock%} + +{%block contents%} + +<p>Status: {{job["metadata"]["status"]}}</p> +<p>Job Type: {{job["metadata"]["job-type"]}}</p> + +<h2>STDOUT</h2> +<pre>{{job["stdout"]}}</pre> + +<h2>STDERR</h2> +<pre>{{job["stderr"]}}</pre> + +{%endblock%} diff --git a/uploader/templates/login.html b/uploader/templates/login.html index bbca42f..e76c644 100644 --- a/uploader/templates/login.html +++ b/uploader/templates/login.html @@ -5,7 +5,8 @@ {%block pagetitle%}log in{%endblock%} {%block extrapageinfo%} -<p> - You <strong>do need to be logged in</strong> to upload data onto this system. - Please do that by clicking the "Log In" button at the top of the page.</p> +<p class="text-dark"> + You <strong>need to + <a href="{{authserver_authorise_uri()}}" + title="Sign in to the system">sign in</a></strong> to use this system.</p> {%endblock%} diff --git a/uploader/templates/macro-step-indicator.html b/uploader/templates/macro-step-indicator.html new file mode 100644 index 0000000..ac0be77 --- /dev/null +++ b/uploader/templates/macro-step-indicator.html @@ -0,0 +1,15 @@ +{%macro step_indicator(step, width=100)%} +<svg width="{{width}}" height="{{width}}" xmlns="http://www.w3.org/2000/svg"> + <circle cx="{{0.5*width}}" + cy="{{0.5*width}}" + r="{{0.5*width}}" + fill="#E5E5FF" /> + <text x="{{0.5*width}}" + y="{{0.6*width}}" + font-size="{{0.2*width}}" + text-anchor="middle" + fill="#555555"> + Step {{step}} + </text> +</svg> +{%endmacro%} diff --git a/uploader/templates/macro-table-pagination.html b/uploader/templates/macro-table-pagination.html new file mode 100644 index 0000000..292c531 --- /dev/null +++ b/uploader/templates/macro-table-pagination.html @@ -0,0 +1,26 @@ +{%macro table_pagination(start_at, page_count, total_count, base_uri, name)%} +{%set ns = namespace(forward_uri=base_uri, back_uri=base_uri)%} +{%set ns.forward_uri="brr"%} + <div class="row"> + <div class="col-md-2" style="text-align: start;"> + {%if start_at > 0%} + <a href="{{base_uri + + '?start_at='+((start_at-page_count)|string) + + '&count='+(page_count|string)}}"> + <span class="glyphicon glyphicon-backward"></span> + Previous + </a> + {%endif%} + </div> + <div class="col-md-8" style="text-align: center;"> + Displaying {{name}} {{start_at+1}} to {{start_at+page_count if start_at+page_count < total_count else total_count}} of {{total_count}}</div> + <div class="col-md-2" style="text-align: end;"> + {%if start_at + page_count < total_count%} + <a href="{{base_uri + + '?start_at='+((start_at+page_count)|string) + + '&count='+(page_count|string)}}"> + Next<span class="glyphicon glyphicon-forward"></span></a> + {%endif%} + </div> + </div> +{%endmacro%} diff --git a/uploader/templates/parse_results.html b/uploader/templates/parse_results.html deleted file mode 100644 index 46fbaaf..0000000 --- a/uploader/templates/parse_results.html +++ /dev/null @@ -1,30 +0,0 @@ -{%extends "base.html"%} -{%from "errors_display.html" import errors_display%} - -{%block title%}Parse Results{%endblock%} - -{%block contents%} -<h1 class="heading">{{job_name}}: parse results</h2> - -{%if user_aborted%} -<span class="alert-warning">Job aborted by the user</span> -{%endif%} - -{{errors_display(errors, "No errors found in the file", "We found the following errors", True)}} - -{%if errors | length == 0 and not user_aborted %} -<form method="post" action="{{url_for('dbinsert.select_platform')}}"> - <input type="hidden" name="job_id" value="{{job_id}}" /> - <input type="submit" value="update database" class="btn btn-primary" /> -</form> -{%endif%} - -{%if errors | length > 0 or user_aborted %} -<br /> -<a href="{{url_for('expression-data.index.upload_file')}}" title="Back to index page." - class="btn btn-primary"> - Go back -</a> -{%endif%} - -{%endblock%} diff --git a/uploader/templates/phenotypes/add-phenotypes-base.html b/uploader/templates/phenotypes/add-phenotypes-base.html new file mode 100644 index 0000000..9909c20 --- /dev/null +++ b/uploader/templates/phenotypes/add-phenotypes-base.html @@ -0,0 +1,166 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "phenotypes/macro-display-pheno-dataset-card.html" import display_pheno_dataset_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="add-phenotypes"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">Add Phenotypes</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <form id="frm-add-phenotypes" + method="POST" + enctype="multipart/form-data" + action="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id, + use_bundle=use_bundle)}}" + data-resumable-target="{{url_for('files.resumable_upload_post')}}"> + <legend>Add New Phenotypes</legend> + + <div class="form-text help-block"> + {%block frm_add_phenotypes_documentation%}{%endblock%} + <p><strong class="text-warning">This will not update any existing phenotypes!</strong></p> + </div> + + {%block frm_add_phenotypes_elements%}{%endblock%} + + <fieldset id="fldset-publication-info"> + <legend>Publication Information</legend> + <input type="hidden" name="publication-id" id="txt-publication-id" /> + <span class="form-text text-muted"> + Select a publication for your data. <br /> + Can't find a publication you can use? Go ahead and + <a href="{{url_for( + 'publications.create_publication', + return_to='species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">create a new publication</a>.</span> + <table id="tbl-select-publication" class="table compact stripe"> + <thead> + <tr> + <th>#</th> + <th>PubMed ID</th> + <th>Title</th> + <th>Authors</th> + </tr> + </thead> + + <tbody></tbody> + </table> + </fieldset> + + <div class="form-group"> + <input type="submit" + value="upload phenotypes" + class="btn btn-primary" /> + </div> + </form> +</div> + +<div class="row"> + {%block page_documentation%}{%endblock%} +</div> + +{%endblock%} + + +{%block javascript%} +<script type="text/javascript"> + $(function() { + var publicationsDataTable = buildDataTable( + "#tbl-select-publication", + [], + [ + {data: "index"}, + { + searchable: true, + data: (pub) => { + if(pub.PubMed_ID) { + return `<a href="https://pubmed.ncbi.nlm.nih.gov/` + + `${pub.PubMed_ID}/" target="_blank" ` + + `title="Link to publication on NCBI.">` + + `${pub.PubMed_ID}</a>`; + } + return ""; + } + }, + { + searchable: true, + data: (pub) => { + var title = "⸻"; + if(pub.Title) { + title = pub.Title + } + return `<a href="/publications/view/${pub.Id}" ` + + `target="_blank" ` + + `title="Link to view publication details">` + + `${title}</a>`; + } + }, + { + searchable: true, + data: (pub) => { + authors = pub.Authors.split(",").map( + (item) => {return item.trim();}); + if(authors.length > 1) { + return authors[0] + ", et. al."; + } + return authors[0]; + } + } + ], + { + serverSide: true, + ajax: { + url: "/publications/list", + dataSrc: "publications" + }, + select: "single", + paging: true, + scrollY: 700, + deferRender: true, + scroller: true, + scrollCollapse: true, + layout: { + topStart: "info", + topEnd: "search" + } + }); + publicationsDataTable.on("select", (event, datatable, type, indexes) => { + indexes.forEach((element, index, thearray) => { + let row = datatable.row(element).node(); + console.debug(datatable.row(element).data()); + $("#frm-add-phenotypes #txt-publication-id").val( + datatable.row(element).data().Id); + }); + }); + publicationsDataTable.on("deselect", (event, datatable, type, indexes) => { + indexes.forEach((element, index, thearray) => { + let row = datatable.row(element).node(); + $("#frm-add-phenotypes #txt-publication-id").val(null); + }); + }); + }); +</script> + +{%block more_javascript%}{%endblock%} +{%endblock%} diff --git a/uploader/templates/phenotypes/add-phenotypes-raw-files.html b/uploader/templates/phenotypes/add-phenotypes-raw-files.html new file mode 100644 index 0000000..67b56e3 --- /dev/null +++ b/uploader/templates/phenotypes/add-phenotypes-raw-files.html @@ -0,0 +1,847 @@ +{%extends "phenotypes/add-phenotypes-base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "phenotypes/macro-display-pheno-dataset-card.html" import display_pheno_dataset_card%} +{%from "phenotypes/macro-display-preview-table.html" import display_preview_table%} +{%from "phenotypes/macro-display-resumable-elements.html" import display_resumable_elements%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="add-phenotypes"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">Add Phenotypes</a> +</li> +{%endblock%} + +{%block frm_add_phenotypes_documentation%} +<p>This page will allow you to upload all the separate files that make up your + phenotypes. Here, you will have to upload each separate file individually. If + you want instead to upload all your files as a single ZIP file, + <a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id, + use_bundle=true)}}" + title="">click here</a>.</p> +{%endblock%} + +{%block frm_add_phenotypes_elements%} +<fieldset id="fldset-file-metadata"> + <legend>File(s) Metadata</legend> + <div class="form-group"> + <label for="txt-file-separator" class="form-label">File Separator</label> + <div class="input-group"> + <input id="txt-file-separator" + name="file-separator" + type="text" + value="	" + class="form-control" + maxlength="1" /> + <span class="input-group-btn"> + <button id="btn-reset-file-separator" class="btn btn-info">Reset Default</button> + </span> + </div> + <span class="form-text text-muted"> + Provide the character that separates the fields in your file(s). It should + be the same character for all files (if more than one is provided).<br /> + A tab character will be assumed if you leave this field blank. See + <a href="#docs-file-separator" + title="Documentation for file-separator characters"> + documentation for more information</a>. + </span> + </div> + + <div class="form-group"> + <label for="txt-file-comment-character" class="form-label">File Comment-Character</label> + <div class="input-group"> + <input id="txt-file-comment-character" + name="file-comment-character" + type="text" + value="#" + class="form-control" + maxlength="1" /> + <span class="input-group-btn"> + <button id="btn-reset-file-comment-character" class="btn btn-info"> + Reset Default</button> + </span> + </div> + <span class="form-text text-muted"> + This specifies that lines that begin with the character provided will be + considered comment lines and ignored in their entirety. See + <a href="#docs-file-comment-character" + title="Documentation for comment characters"> + documentation for more information</a>. + </span> + </div> + + <div class="form-group"> + <label for="txt-file-na" class="form-label">File "No-Value" Indicators</label> + <div class="input-group"> + <input id="txt-file-na" + name="file-na" + type="text" + value="- NA N/A" + class="form-control" /> + <span class="input-group-btn"> + <button id="btn-reset-file-na" class="btn btn-info">Reset Default</button> + </span> + </div> + <span class="form-text text-muted"> + This specifies strings in your file indicate that there is no value for a + particular cell (a cell is where a column and row intersect). Provide a + space-separated list of strings if you have more than one way of + indicating no values. See + <a href="#docs-file-na" title="Documentation for no-value fields"> + documentation for more information</a>.</span> + </div> +</fieldset> + +<fieldset id="fldset-files"> + <legend>Data File(s)</legend> + + <fieldset id="fldset-descriptions-file"> + <div class="form-group"> + <div class="form-check"> + <input id="chk-phenotype-descriptions-transposed" + name="phenotype-descriptions-transposed" + type="checkbox" + class="form-check-input" + style="border: solid #8EABF0" /> + <label for="chk-phenotype-descriptions-transposed" + class="form-check-label"> + Description file transposed?</label> + </div> + + <div class="non-resumable-elements"> + <label for="finput-phenotype-descriptions" class="form-label"> + Phenotype Descriptions</label> + <input id="finput-phenotype-descriptions" + name="phenotype-descriptions" + class="form-control" + type="file" + data-preview-table="tbl-preview-pheno-desc" + required="required" /> + <span class="form-text text-muted"> + Provide a file that contains only the phenotype descriptions, + <a href="#docs-file-phenotype-description" + title="Documentation of the phenotype data file format."> + the documentation for the expected format of the file</a>.</span> + </div> + {{display_resumable_elements( + "resumable-phenotype-descriptions", + "phenotype descriptions", + '<p>Drag and drop the CSV file that contains the descriptions of your + phenotypes here.</p> + + <p>The CSV file should be a matrix of + <strong>phenotypes × descriptions</strong> i.e. The first column + contains the phenotype names/identifiers whereas the first row is a list + of metadata fields like, "description", "units", etc.</p> + + <p>If the format is transposed (i.e. + <strong>descriptions × phenotypes</strong>) select the checkbox above. + </p> + + <p>Please see the + <a href="#docs-file-phenotype-description" + title="Documentation of the phenotype data file format."> + "Phenotypes Descriptions" documentation</a> section below for more + information on the expected format of the file provided here.</p>')}} + {{display_preview_table( + "tbl-preview-pheno-desc", "phenotype descriptions")}} + </div> + </fieldset> + + + <fieldset id="fldset-data-file"> + <div class="form-group"> + <div class="form-check"> + <input id="chk-phenotype-data-transposed" + name="phenotype-data-transposed" + type="checkbox" + class="form-check-input" + style="border: solid #8EABF0" /> + <label for="chk-phenotype-data-transposed" class="form-check-label"> + Data file transposed?</label> + </div> + + <div class="non-resumable-elements"> + <label for="finput-phenotype-data" class="form-label">Phenotype Data</label> + <input id="finput-phenotype-data" + name="phenotype-data" + class="form-control" + type="file" + data-preview-table="tbl-preview-pheno-data" + required="required" /> + <span class="form-text text-muted"> + Provide a file that contains only the phenotype data. See + <a href="#docs-file-phenotype-data" + title="Documentation of the phenotype data file format."> + the documentation for the expected format of the file</a>.</span> + </div> + + {{display_resumable_elements( + "resumable-phenotype-data", + "phenotype data", + '<p>Drag and drop a CSV file that contains the phenotypes numerical data + here. You can click the "Browse" button (below and to the right) to + select the file from your computer.</p> + + <p>The CSV should be a matrix of <strong>samples × phenotypes</strong>, + i.e. The first column contains the samples identifiers while the first + row is the list of phenotypes identifiers occurring in the phenotypes + descriptions file.</p> + + <p>If the format is transposed (i.e <strong>phenotypes × samples</strong>) + select the checkbox above.</p> + <p>Please see the + <a href="#docs-file-phenotype-data" + title="Documentation of the phenotype data file format."> + "Phenotypes Data" documentation</a> section below for more information + on the expected format for the file provided here.</p>')}} + {{display_preview_table("tbl-preview-pheno-data", "phenotype data")}} + </div> + </fieldset> + + + {%if population.Family in families_with_se_and_n%} + <fieldset id="fldset-se-file"> + <div class="form-group"> + <div class="form-check"> + <input id="chk-phenotype-se-transposed" + name="phenotype-se-transposed" + type="checkbox" + class="form-check-input" + style="border: solid #8EABF0" /> + <label for="chk-phenotype-se-transposed" class="form-check-label"> + Standard-Errors file transposed?</label> + </div> + <div class="group non-resumable-elements"> + <label for="finput-phenotype-se" class="form-label">Phenotype: Standard Errors</label> + <input id="finput-phenotype-se" + name="phenotype-se" + class="form-control" + type="file" + data-preview-table="tbl-preview-pheno-se" + required="required" /> + <span class="form-text text-muted"> + Provide a file that contains only the standard errors for the phenotypes, + computed from the data above.</span> + </div> + + {{display_resumable_elements( + "resumable-phenotype-se", + "standard errors", + '<p>Drag and drop a CSV file that contains the phenotypes standard-errors + data here. You can click the "Browse" button (below and to the right) to + select the file from your computer.</p> + + <p>The CSV should be a matrix of <strong>samples × phenotypes</strong>, + i.e. The first column contains the samples identifiers while the first + row is the list of phenotypes identifiers occurring in the phenotypes + descriptions file.</p> + + <p>If the format is transposed (i.e <strong>phenotypes × samples</strong>) + select the checkbox above.</p> + + <p>Please see the + <a href="#docs-file-phenotype-se" + title="Documentation of the phenotype data file format."> + "Phenotypes Data" documentation</a> section below for more information + on the expected format of the file provided here.</p>')}} + + {{display_preview_table("tbl-preview-pheno-se", "standard errors")}} + </div> + </fieldset> + + + <fieldset id="fldset-n-file"> + <div class="form-group"> + <div class="form-check"> + <input id="chk-phenotype-n-transposed" + name="phenotype-n-transposed" + type="checkbox" + class="form-check-input" + style="border: solid #8EABF0" /> + <label for="chk-phenotype-n-transposed" class="form-check-label"> + Counts file transposed?</label> + </div> + <div class="non-resumable-elements"> + <label for="finput-phenotype-n" class="form-label">Phenotype: Number of Samples/Individuals</label> + <input id="finput-phenotype-n" + name="phenotype-n" + class="form-control" + type="file" + data-preview-table="tbl-preview-pheno-n" + required="required" /> + <span class="form-text text-muted"> + Provide a file that contains only the number of samples/individuals used in + the computation of the standard errors above.</span> + </div> + + {{display_resumable_elements( + "resumable-phenotype-n", + "number of samples/individuals", + '<p>Drag and drop a CSV file that contains the samples\' phenotypes counts + data here. You can click the "Browse" button (below and to the right) to + select the file from your computer.</p> + + <p>The CSV should be a matrix of <strong>samples × phenotypes</strong>, + i.e. The first column contains the samples identifiers while the first + row is the list of phenotypes identifiers occurring in the phenotypes + descriptions file.</p> + + <p>If the format is transposed (i.e <strong>phenotypes × samples</strong>) + select the checkbox above.</p> + + <p>Please see the + <a href="#docs-file-phenotype-se" + title="Documentation of the phenotype data file format."> + "Phenotypes Data" documentation</a> section below for more information + on the expected format of the file provided here.</p>')}} + + {{display_preview_table("tbl-preview-pheno-n", "number of samples/individuals")}} + </div> + </fieldset> +</fieldset> +{%endif%} +{%endblock%} + + +{%block page_documentation%} +<div class="row"> + <h2 class="heading" id="docs-help">Help</h2> + <h3 class="subheading">Common Features</h3> + <p>The following are the common expectations for <strong>ALL</strong> the + files provided in the form above: + <ul> + <li>The file <strong>MUST</strong> be character-separated values (CSV) + text file</li> + <li>The first row in the file <strong>MUST</strong> be a heading row, and + will be composed of the list identifiers for all of + samples/individuals/cases involved in your study.</li> + <li>The first column of data in the file <strong>MUST</strong> be the + identifiers for all of the phenotypes you wish to upload.</li> + </ul> + </p> + + <p>If you do not specify the separator character, then we will assume a + <strong>TAB</strong> character was used as your separator.</p> + + <p>We also assume you might include comments lines in your files. In that + case, if you do not specify what character denotes that a line in your files + is a comment line, we will assume the <strong>#</strong> character.<br /> + A comment <strong>MUST ALWAYS</strong> begin at the start of the line marked + with the comment character specified.</p> + + <h3 class="subheading" id="docs-file-metadata">File Metadata</h3> + <p>We request some details about your files to help us parse and process the + files correctly. The details we collect are:</p> + <dl> + <dt id="docs-file-separator">File separator</dt> + <dd>The files you provide should be character-separated value (CSV) files. + We need to know what character you used to separate the values in your + file. Some common ones are the Tab character, the comma, etc.<br /> + Providing that information makes it possible for the system to parse and + process your files correctly.<br> + <strong>NOTE:</strong> All the files you upload MUST use the same + separator.</dd> + + <dt id="docs-file-comment-character">Comment character</dt> + <dd>We support use of comment lines in your files. We only support one type + of comment style, the <em>line comment</em>.<br /> + This mean the comment begins at the start of the line, and the end of that + line indicates the end of that comment. If you have a really long comment, + then you need to break it across multiple lines, marking each line a + comment line.<br /> + The "comment character" is the character at the start of the line that + indicates that the line is a line comment.</dd> + + <dt id="docs-file-na">No-Value indicator(s)</dt> + <dd>Data in the real world is messy, and in some cases, entirely absent. You + need to indicate, in your files, that a particular field did not have a + value, and once you do that, you then need to let the system know how you + mark such fields. Common ways of indicating "empty values" are, leaving + the field blank, using a character such as '-', or using strings like + "NA", "N/A", "NULL", etc.<br /> + Providing this information will help with parsing and processing such + no-value fields the correct way.</dd> + </dl> + + <h3 class="subheading" id="docs-file-phenotype-description"> + file: Phenotypes Descriptions</h3> + <p>The data in this file is a matrix of <em>phenotypes × metadata-fields</em>. + Please note we use the term "metadata-fields" above loosely, due to lack of + a good word for this.</p> + <p>The file <strong>MUST</strong> have columns in this order: + <dl> + <dt>Phenotype Identifiers</dt> + <dd>These are the names/identifiers for your phenotypes. These + names/identifiers are the same ones you will have in all the other files you are + uploading.</dd> + + <dt>Descriptions</dt> + <dd>Each phenotype will need a description. Good description are necessary + to inform other people of what the data is about. Good description are + hard to construct, so we provide + <a href="https://info.genenetwork.org/faq.php#q-22" + title="How to write phenotype descriptions"> + advice on describing your phenotypes.</a></dd> + + <dt>Units</dt> + <dd>Each phenotype will need units for the measurements taken. If there are + none, then indicate the field is a no-value field.</dd> + </dl></p> + <p>You can add more columns after those three if you want to, but these 3 + <strong>MUST</strong> be present.</p> + <p>The file would, for example, look like the following:</p> + <code>id,description,units,…<br /> + pheno10001|Central nervous system, behavior, cognition; …|mg|…<br /> + pheno10002|Aging, metabolism, central nervous system: …|mg|…<br /> + â‹®<br /></code> + + <p><strong>Note 01</strong>: The first usable row is the heading row.</p> + <p><strong>Note 02: </strong>This example demonstrates a subtle issue that + could make your CSV file invalid — the choice of your field separator + character.<br > + In the example above, we use the pipe character (<code>|</code>) as our + field separator. This is because, if we follow the advice on how to write + good descriptions, then we cannot use the comma as our separator – if + we did, then our CSV file would be invalid because the system would have no + way to tell the difference between the comma as a field separator, and the + comma as a way to separate the "general category and ontology terms".</p> + + <h3 class="subheading">file: Phenotype Data, Standard Errors and/or Sample Counts</h3> + <span id="docs-file-phenotype-data"></span> + <span id="docs-file-phenotype-se"></span> + <span id="docs-file-phenotype-n"></span> + <p>The data is a matrix of <em>samples(or individuals) × phenotypes</em>, e.g.</p> + <code> + # num-cases: 2549 + # num-phenos: 13 + id,pheno10001,pheno10002,pheno10003,pheno10004,53.099998,…<br /> + IND001,61.400002,49,62.5,55.099998,…<br /> + IND002,54.099998,50.099998,53.299999,55.099998,…<br /> + IND003,483,403,501,403,…<br /> + IND004,49.799999,45.5,62.900002,NA,…<br /> + â‹®<br /></code> + + <p>where <code>IND001,IND002,IND003,IND004,…</code> are the + samples/individuals/cases in your study, and + <code>pheno10001,pheno10002,pheno10004,pheno10004,…</code> are the + identifiers for your phenotypes.</p> + <p>The lines beginning with the "<em>#</em>" symbol (i.e. + <code># num-cases: 2549</code> and <code># num-phenos: 13</code> are comment + lines and will be ignored</p> + <p>In this example, the comma (,) is used as the file separator.</p> +</div> + +{%endblock%} + +{%block sidebarcontents%} +{{display_pheno_dataset_card(species, population, dataset)}} +{%endblock%} + + +{%block more_javascript%} +<script src="{{url_for('base.node_modules', + filename='resumablejs/resumable.js')}}"></script> +<script type="text/javascript" src="/static/js/files.js"></script> + +<script type="text/javascript"> + $("#btn-reset-file-separator").on("click", (event) => { + event.preventDefault(); + $("#txt-file-separator").val("\t"); + $("#txt-file-separator").trigger("change"); + }); + $("#btn-reset-file-comment-character").on("click", (event) => { + event.preventDefault(); + $("#txt-file-comment-character").val("#"); + $("#txt-file-comment-character").trigger("change"); + }); + $("#btn-reset-file-na").on("click", (event) => { + event.preventDefault(); + $("#txt-file-na").val("- NA N/A"); + $("#txt-file-na").trigger("change"); + }); + + var update_preview = (table, filedata, formdata, numrows) => { + table.find("thead tr").remove() + table.find(".data-row").remove(); + var linenum = 0; + var tableheader = table.find("thead"); + var tablebody = table.find("tbody"); + var numheadings = 0; + var navalues = formdata + .na_strings + .split(" ") + .map((v) => {return v.trim();}) + .filter((v) => {return Boolean(v);}); + filedata.forEach((line) => { + if(line.startsWith(formdata.comment_char) || linenum >= numrows) { + return false; + } + var row = $("<tr></tr>"); + line.split(formdata.separator) + .map((field) => { + var value = field.trim(); + if(navalues.includes(value)) { + return "⋘NULâ‹™"; + } + return value; + }) + .filter((field) => { + return (field !== "" && field != undefined && field != null); + }) + .forEach((field) => { + if(linenum == 0) { + numheadings += 1; + var tablefield = $("<th></th>"); + tablefield.text(field); + row.append(tablefield); + } else { + add_class(row, "data-row"); + var tablefield = $("<td></td>"); + tablefield.text(field); + row.append(tablefield); + } + }); + + if(linenum == 0) { + tableheader.append(row); + } else { + tablebody.append(row); + } + linenum += 1; + }); + + if(table.find("tbody tr.data-row").length > 0) { + add_class(table.find(".data-row-template"), "visually-hidden"); + } else { + remove_class(table.find(".data-row-template"), "visually-hidden"); + } + }; + + var makePreviewUpdater = (preview_table) => { + return (data) => { + update_preview( + preview_table, + data, + filesMetadata(), + PREVIEW_ROWS); + }; + }; + + var preview_tables_to_elements_map = { + "#tbl-preview-pheno-desc": "#finput-phenotype-descriptions", + "#tbl-preview-pheno-data": "#finput-phenotype-data", + "#tbl-preview-pheno-se": "#finput-phenotype-se", + "#tbl-preview-pheno-n": "#finput-phenotype-n" + }; + + var filesMetadata = () => { + return { + "separator": $("#txt-file-separator").val(), + "comment_char": $( + "#txt-file-comment-character").val(), + "na_strings": $("#txt-file-na").val() + } + }; + + var PREVIEW_ROWS = 5; + + var handler_update_previews = (event) => { + Object.entries(preview_tables_to_elements_map).forEach((mapentry) => { + var preview_table = $(mapentry[0]); + var file_input = $(mapentry[1]); + if(file_input[0].files.length > 0) { + readFirstNLines( + file_input[0].files[0], + 10, + [makePreviewUpdater(preview_table)]); + } + }); + + if(typeof(resumables) !== "undefined") { + resumables.forEach((resumable) => { + if(resumable.files.length > 0) { + readFirstNLines( + resumable.files[0].file, + 10, + [makePreviewUpdater(resumable.preview_table)]); + } + }); + } + }; + + [ + "#txt-file-separator", + "#txt-file-comment-character", + "#txt-file-na" + ].forEach((elementid) => { + $(elementid).on("change", handler_update_previews); + }); + + [ + "#finput-phenotype-descriptions", + "#finput-phenotype-data", + "#finput-phenotype-se", + "#finput-phenotype-n" + ].forEach((elementid) => { + $(elementid).on("change", (event) => { + readFirstNLines( + event.target.files[0], + 10, + [makePreviewUpdater( + $("#" + event.target.getAttribute("data-preview-table")))]); + }); + }); + + + var resumableDisplayFiles = (display_area, files) => { + files.forEach((file) => { + display_area.find(".file-display").remove(); + var display_element = display_area + .find(".file-display-template") + .clone(); + remove_class(display_element, "visually-hidden"); + remove_class(display_element, "file-display-template"); + add_class(display_element, "file-display"); + display_element.find(".filename").text(file.name + || file.fileName + || file.relativePath + || file.webkitRelativePath); + display_element.find(".filesize").text( + (file.size / (1024*1024)).toFixed(2) + "MB"); + display_element.find(".fileuniqueid").text(file.uniqueIdentifier); + display_element.find(".filemimetype").text(file.file.type); + display_area.append(display_element); + }); + }; + + + var indicateProgress = (resumable, progress_bar) => { + return () => {/*Has no event!*/ + var progress = (resumable.progress() * 100).toFixed(2); + var pbar = progress_bar.find(".progress-bar"); + remove_class(progress_bar, "visually-hidden"); + pbar.css("width", progress+"%"); + pbar.attr("aria-valuenow", progress); + pbar.text("Uploading: " + progress + "%"); + }; + }; + + var retryUpload = (retry_button, cancel_button) => { + retry_button.on("click", (event) => { + resumable.files.forEach((file) => {file.retry();}); + add_class(retry_button, "visually-hidden"); + remove_class(cancel_button, "visually-hidden"); + add_class(browse_button, "visually-hidden"); + }); + }; + + var cancelUpload = (cancel_button, retry_button) => { + cancel_button.on("click", (event) => { + resumable.files.forEach((file) => { + if(file.isUploading()) { + file.abort(); + } + }); + add_class(cancel_button, "visually-hidden"); + remove_class(retry_button, "visually-hidden"); + remove_class(browse_button, "visually-hidden"); + }); + }; + + + var startUpload = (browse_button, retry_button, cancel_button) => { + return (event) => { + remove_class(cancel_button, "visually-hidden"); + add_class(retry_button, "visually-hidden"); + add_class(browse_button, "visually-hidden"); + }; + }; + + var processForm = (form) => { + var formdata = new FormData(form); + uploaded_files.forEach((msg) => { + formdata.delete(msg["file-input-name"]); + formdata.append(msg["file-input-name"], JSON.stringify({ + "uploaded-file": msg["uploaded-file"], + "original-name": msg["original-name"] + })); + }); + formdata.append("resumable-upload", "true"); + formdata.append("publication-id", $("#txt-publication-id").val()); + return formdata; + } + + var uploaded_files = new Set(); + var submitForm = (new_file) => { + uploaded_files.add(new_file); + if(uploaded_files.size === resumables.length) { + var form = $("#frm-add-phenotypes"); + if(form.length !== 1) { + // TODO: Handle error somehow? + alert("Could not find form!!!"); + return false; + } + + $.ajax({ + "url": form.attr("action"), + "type": "POST", + "data": processForm(form[0]), + "processData": false, + "contentType": false, + "success": (data, textstatus, jqxhr) => { + // TODO: Redirect to endpoint that should come as part of the + // success/error message. + console.log("SUCCESS DATA: ", data); + console.log("SUCCESS STATUS: ", textstatus); + console.log("SUCCESS jqXHR: ", jqxhr); + window.location.assign(window.location.origin + data["redirect-to"]); + }, + }); + return false; + } + return false; + }; + + var uploadSuccess = (file_input_name) => { + return (file, message) => { + submitForm({...JSON.parse(message), "file-input-name": file_input_name}); + }; + }; + + + var uploadError = () => { + return (message, file) => { + $("#frm-add-phenotypes input[type=submit]").removeAttr("disabled"); + console.log("THE FILE:", file); + console.log("THE ERROR MESSAGE:", message); + }; + }; + + + + var makeResumableObject = (form_id, file_input_id, resumable_element_id, preview_table_id) => { + var the_form = $("#" + form_id); + var file_input = $("#" + file_input_id); + var submit_button = the_form.find("input[type=submit]"); + if(file_input.length != 1) { + return false; + } + var r = errorHandler( + fileSuccessHandler( + uploadStartHandler( + filesAddedHandler( + markResumableDragAndDropElement( + makeResumableElement( + the_form.attr("data-resumable-target"), + file_input.parent(), + $("#" + resumable_element_id), + submit_button, + ["csv", "tsv", "txt"]), + file_input.parent(), + $("#" + resumable_element_id), + $("#" + resumable_element_id + "-browse-button")), + (files) => { + // TODO: Also trigger preview! + resumableDisplayFiles( + $("#" + resumable_element_id + "-selected-files"), files); + files.forEach((file) => { + readFirstNLines( + file.file, + 10, + [makePreviewUpdater( + $("#" + preview_table_id))]) + }); + }), + startUpload($("#" + resumable_element_id + "-browse-button"), + $("#" + resumable_element_id + "-retry-button"), + $("#" + resumable_element_id + "-cancel-button"))), + uploadSuccess(file_input.attr("name"))), + uploadError()); + + /** Setup progress indicator **/ + progressHandler( + r, + indicateProgress(r, $("#" + resumable_element_id + "-progress-bar"))); + + return r; + }; + + var resumables = [ + ["frm-add-phenotypes", "finput-phenotype-descriptions", "resumable-phenotype-descriptions", "tbl-preview-pheno-desc"], + ["frm-add-phenotypes", "finput-phenotype-data", "resumable-phenotype-data", "tbl-preview-pheno-data"], + ["frm-add-phenotypes", "finput-phenotype-se", "resumable-phenotype-se", "tbl-preview-pheno-se"], + ["frm-add-phenotypes", "finput-phenotype-n", "resumable-phenotype-n", "tbl-preview-pheno-n"], + ].map((row) => { + r = makeResumableObject(row[0], row[1], row[2], row[3]); + r.preview_table = $("#" + row[3]); + return r; + }).filter((val) => { + return Boolean(val); + }); + + $("#frm-add-phenotypes input[type=submit]").on("click", (event) => { + event.preventDefault(); + console.debug(); + if ($("#txt-publication-id").val() == "") { + alert("You MUST provide a publication for the phenotypes."); + return false; + } + // TODO: Check all the relevant files exist + // TODO: Verify that files are not duplicated + var filenames = []; + var nondupfiles = []; + resumables.forEach((r) => { + var fname = r.files[0].file.name; + filenames.push(fname); + if(!nondupfiles.includes(fname)) { + nondupfiles.push(fname); + } + }); + + // Check that all files were provided + if(resumables.length !== filenames.length) { + window.alert("You MUST provide all the files requested."); + event.target.removeAttribute("disabled"); + return false; + } + + // Check that there are no duplicate files + var duplicates = Object.entries(filenames.reduce( + (acc, curr, idx, arr) => { + acc[curr] = (acc[curr] || 0) + 1; + return acc; + }, + {})).filter((entry) => {return entry[1] !== 1;}); + if(duplicates.length > 0) { + var msg = "The file(s):\r\n"; + msg = msg + duplicates.reduce( + (msgstr, afile) => { + return msgstr + " • " + afile[0] + "\r\n"; + }, + ""); + msg = msg + "is(are) duplicated. Please fix and try again."; + window.alert(msg); + event.target.removeAttribute("disabled"); + return false; + } + // TODO: Check all fields + // Start the uploads. + event.target.setAttribute("disabled", "disabled"); + resumables.forEach((r) => {r.upload();}); + }); +</script> +{%endblock%} diff --git a/uploader/templates/phenotypes/add-phenotypes-with-rqtl2-bundle.html b/uploader/templates/phenotypes/add-phenotypes-with-rqtl2-bundle.html new file mode 100644 index 0000000..898fc0c --- /dev/null +++ b/uploader/templates/phenotypes/add-phenotypes-with-rqtl2-bundle.html @@ -0,0 +1,207 @@ +{%extends "phenotypes/add-phenotypes-base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "phenotypes/macro-display-pheno-dataset-card.html" import display_pheno_dataset_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="add-phenotypes"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">Add Phenotypes</a> +</li> +{%endblock%} + +{%block frm_add_phenotypes_documentation%} +<p>Select the zip file bundle containing information on the phenotypes you + wish to upload, then click the "Upload Phenotypes" button below to + upload the data.</p> +<p>If you wish to upload the files individually instead, + <a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}" + title="">click here</a>.</p> +<p>See the <a href="#section-file-formats">File Formats</a> section below + to get an understanding of what is expected of the bundle files you + upload.</p> +{%endblock%} + +{%block frm_add_phenotypes_elements%} +<div class="form-group"> + <label for="finput-phenotypes-bundle" class="form-label"> + Phenotypes Bundle</label> + <input type="file" + id="finput-phenotypes-bundle" + name="phenotypes-bundle" + accept="application/zip, .zip" + required="required" + class="form-control" /> +</div> +{%endblock%} + +{%block page_documentation%} +<div class="row"> + <h2 class="heading" id="section-file-formats">File Formats</h2> + <p>We accept an extended form of the + <a href="https://kbroman.org/qtl2/assets/vignettes/input_files.html#format-of-the-data-files" + title="R/qtl2 software input file format documentation"> + input files' format used with the R/qtl2 software</a> as a single ZIP + file</p> + <p>The files that are used for this feature are: + <ul> + <li>the <em>control</em> file</li> + <li><em>pheno</em> file(s)</li> + <li><em>phenocovar</em> file(s)</li> + <li><em>phenose</em> files(s)</li> + </ul> + </p> + <p>Other files within the bundle will be ignored, for this feature.</p> + <p>The following section will detail the expectations for each of the + different file types within the uploaded ZIP file bundle for phenotypes:</p> + + <h3 class="subheading">Control File</h3> + <p>There <strong>MUST be <em>one, and only one</em></strong> file that acts + as the control file. This file can be: + <ul> + <li>a <em>JSON</em> file, or</li> + <li>a <em>YAML</em> file.</li> + </ul> + </p> + + <p>The control file is useful for defining things about the bundle such as:</p> + <ul> + <li>The field separator value (default: <code>sep: ','</code>). There can + only ever be one field separator and it <strong>MUST</strong> be the same + one for <strong>ALL</strong> files in the bundle.</li> + <li>The comment character (default: <code>comment.char: '#'</code>). Any + line that starts with this character will be considered a comment line and + be ignored in its entirety.</li> + <li>Code for missing values (default: <code>na.strings: 'NA'</code>). You + can specify more than one code to indicate missing values, e.g. + <code>{…, "na.strings": ["NA", "N/A", "-"], …}</code></li> + </ul> + + <h3 class="subheading"><em>pheno</em> File(s)</h3> + <p>These files are the main data files. You must have at least one of these + files in your bundle for it to be valid for this step.</p> + <p>The data is a matrix of <em>individuals × phenotypes</em> by default, as + below:<br /> + <code> + id,10001,10002,10003,10004,…<br /> + BXD1,61.400002,54.099998,483,49.799999,…<br /> + BXD2,49,50.099998,403,45.5,…<br /> + BXD5,62.5,53.299999,501,62.900002,…<br /> + BXD6,53.099998,55.099998,403,NA,…<br /> + â‹®<br /></code> + </p> + <p>If the <code>pheno_transposed</code> value is set to <code>True</code>, + then the data will be a <em>phenotypes × individuals</em> matrix as in the + example below:<br /> + <code> + id,BXD1,BXD2,BXD5,BXD6,…<br /> + 10001,61.400002,49,62.5,53.099998,…<br /> + 10002,54.099998,50.099998,53.299999,55.099998,…<br /> + 10003,483,403,501,403,…<br /> + 10004,49.799999,45.5,62.900002,NA,…<br /> + â‹® + </code> + </p> + + + <h3 class="subheading"><em>phenocovar</em> File(s)</h3> + <p>At least one phenotypes metadata file with the metadata values such as + descriptions, PubMed Identifier, publication titles (if present), etc.</p> + <p>The data in this/these file(s) is a matrix of + <em>phenotypes × phenotypes-covariates</em>. The first column is always the + phenotype names/identifiers — same as in the R/qtl2 format.</p> + <p><em>phenocovar</em> files <strong>should never be transposed</strong>!</p> + <p>This file <strong>MUST</strong> be present in the bundle, and have data for + the bundle to be considered valid by our system for this step.<br /> + In addition to that, the following are the fields that <strong>must be + present</strong>, and + have values, in the file before the file is considered valid: + <ul> + <li><em>description</em>: A description for each phenotype. Useful + for users to know what the phenotype is about.</li> + <li><em>units</em>: The units of measurement for the phenotype, + e.g. milligrams for brain weight, centimetres/millimetres for + tail-length, etc.</li> + </ul></p> + + <p>The following <em>optional</em> fields can also be provided: + <ul> + <li><em>pubmedid</em>: A PubMed Identifier for the publication where + the phenotype is published. If this field is not provided, the system will + assume your phenotype is not published.</li> + </ul> + </p> + <p>These files will be marked up in the control file with the + <code>phenocovar</code> key, as in the examples below: + <ol> + <li>JSON: single file<br /> + <code>{<br /> + â‹®,<br /> + "phenocovar": "your_covariates_file.csv",<br /> + â‹®<br /> + } + </code> + </li> + <li>JSON: multiple files<br /> + <code>{<br /> + â‹®,<br /> + "phenocovar": [<br /> + "covariates_file_01.csv",<br /> + "covariates_file_01.csv",<br /> + â‹®<br /> + ],<br /> + â‹®<br /> + } + </code> + </li> + <li>YAML: single file or<br /> + <code> + â‹®<br /> + phenocovar: your_covariates_file.csv<br /> + â‹® + </code> + </li> + <li>YAML: multiple files<br /> + <code> + â‹®<br /> + phenocovar:<br /> + - covariates_file_01.csv<br /> + - covariates_file_02.csv<br /> + - covariates_file_03.csv<br /> + …<br /> + â‹® + </code> + </li> + </ol> + </p> + + <h3 class="subheading"><em>phenose</em> and <em>phenonum</em> File(s)</h3> + <p>These are extensions to the R/qtl2 standard, i.e. these types ofs file are + not supported by the original R/qtl2 file format</p> + <p>We use these files to upload the standard errors (<em>phenose</em>) when + the data file (<em>pheno</em>) is average data. In that case, the + <em>phenonum</em> file(s) contains the number of individuals that were + involved when computing the averages.</p> + <p>Both types of files are matrices of <em>individuals × phenotypes</em> by + default. Like the related <em>pheno</em> files, if + <code>pheno_transposed: True</code>, then the file will be a matrix of + <em>phenotypes × individuals</em>.</p> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_pheno_dataset_card(species, population, dataset)}} +{%endblock%} diff --git a/uploader/templates/phenotypes/base.html b/uploader/templates/phenotypes/base.html new file mode 100644 index 0000000..adbc012 --- /dev/null +++ b/uploader/templates/phenotypes/base.html @@ -0,0 +1,19 @@ +{%extends "populations/base.html"%} + +{%block lvl3_breadcrumbs%} +<li {%if activelink=="phenotypes"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + {%if dataset is mapping%} + <a href="{{url_for('species.populations.phenotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">{{dataset.Name}}</a> + {%else%} + <a href="{{url_for('species.populations.phenotypes.index')}}">Phenotypes</a> + {%endif%} +</li> +{%block lvl4_breadcrumbs%}{%endblock%} +{%endblock%} diff --git a/uploader/templates/phenotypes/bulk-edit-upload.html b/uploader/templates/phenotypes/bulk-edit-upload.html new file mode 100644 index 0000000..d0f38f5 --- /dev/null +++ b/uploader/templates/phenotypes/bulk-edit-upload.html @@ -0,0 +1,62 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="view-dataset"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">View</a> +</li> +{%endblock%} + +{%block contents%} +<div class="row"> + <p>Upload the edited file you downloaded and edited.</p> +</div> + +<div class="row"> + <form id="frm-bulk-edit-upload" + class="form-horizontal" + method="POST" + action="{{url_for( + 'species.populations.phenotypes.edit_upload_phenotype_data', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}" + enctype="multipart/form-data"> + + <div class="form-group row"> + <label for="file-upload-bulk-edit-upload" + class="form-label col-form-label col-sm-2"> + Edited File</label> + <div class="col-sm-10"> + <input id="file-upload-bulk-edit-upload" + name="file-upload-bulk-edit-upload" + class="form-control" + type="file" + accept="text/tab-separated-values" + required="required" /> + </div> + </div> + + <input type="submit" class="btn btn-primary" + value="upload to edit" /> + + </form> +</div> +{%endblock%} + + +{%block javascript%} +{%endblock%} diff --git a/uploader/templates/phenotypes/create-dataset.html b/uploader/templates/phenotypes/create-dataset.html new file mode 100644 index 0000000..19a2b34 --- /dev/null +++ b/uploader/templates/phenotypes/create-dataset.html @@ -0,0 +1,108 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="create-dataset"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}">Create Datasets</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>Create a new phenotype dataset.</p> +</div> + +<div class="row"> + <form id="frm-create-pheno-dataset" + action="{{url_for('species.populations.phenotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}" + method="POST"> + + <div class="form-group"> + <label class="form-label" for="txt-dataset-name">Name</label> + {%if errors["dataset-name"] is defined%} + <small class="form-text text-muted danger"> + <p>{{errors["dataset-name"]}}</p></small> + {%endif%} + <input type="text" + name="dataset-name" + id="txt-dataset-name" + value="{{original_formdata.get('dataset-name') or (population.Name + 'Publish')}}" + {%if errors["dataset-name"] is defined%} + class="form-control danger" + {%else%} + class="form-control" + {%endif%} + required="required" /> + <small class="form-text text-muted"> + <p>A short representative name for the dataset.</p> + <p>Recommended: Use the population name and append "Publish" at the end. + <br />This field will only accept names composed of + letters ('A-Za-z'), numbers (0-9), hyphens and underscores.</p> + </small> + </div> + + <div class="form-group"> + <label class="form-label" for="txt-dataset-fullname">Full Name</label> + {%if errors["dataset-fullname"] is defined%} + <small class="form-text text-muted danger"> + <p>{{errors["dataset-fullname"]}}</p></small> + {%endif%} + <input id="txt-dataset-fullname" + name="dataset-fullname" + type="text" + value="{{original_formdata.get('dataset-fullname', '')}}" + {%if errors["dataset-fullname"] is defined%} + class="form-control danger" + {%else%} + class="form-control" + {%endif%} + required="required" /> + <small class="form-text text-muted"> + <p>A longer, descriptive name for the dataset. The name is meant for use + by humans, and therefore, it should be clear what the dataset contains + from the name.</p> + </small> + </div> + + <div class="form-group"> + <label class="form-label" for="txt-dataset-shortname">Short Name</label> + <input id="txt-dataset-shortname" + name="dataset-shortname" + type="text" + class="form-control" + value="{{original_formdata.get('dataset-shortname') or (population.Name + 'Publish')}}" /> + <small class="form-text text-muted"> + <p>An optional, short name for the dataset. <br /> + If this is not provided, it will default to the value provided for the + <strong>Name</strong> field above.</p></small> + </div> + + <div class="form-group"> + <input type="submit" + class="btn btn-primary" + value="create phenotype dataset" /> + </div> + + </form> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/phenotypes/edit-phenotype.html b/uploader/templates/phenotypes/edit-phenotype.html new file mode 100644 index 0000000..32c903f --- /dev/null +++ b/uploader/templates/phenotypes/edit-phenotype.html @@ -0,0 +1,332 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="edit-phenotype"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.edit_phenotype_data', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id, + xref_id=xref_id)}}">View Datasets</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <h2 class="heading">edit phenotype data</h2> + <p>The forms provided in this page help you update the data for the + phenotypes, and the publication information for the phenotype, + respectively.</p> +</div> + +<div class="row"> + <h3 class="subheading">Basic metadata</h3> + <form name="frm-phenotype-basic-metadata" + class="form-horizontal" + method="POST" + action="{{url_for( + 'species.populations.phenotypes.edit_phenotype_data', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id, + xref_id=xref_id)}}"> + <input type="hidden" name="phenotype-id" value="{{phenotype.Id}}" /> + <div class="form-group"> + <label for="txt-pre-publication-description" + class="control-label col-sm-2">Pre-Publication Description</label> + <div class="col-sm-10"> + <input type="text" + id="txt-pre-publication-description" + name="pre-publication-description" + class="form-control" + value="{{phenotype['Pre_publication_description'] or ''}}" /> + </div> + </div> + + <div class="form-group"> + <label for="txt-pre-publication-abbreviation" + class="control-label col-sm-2">Pre-Publication Abbreviation</label> + <div class="col-sm-10"> + <input type="text" + id="txt-pre-publication-abbreviation" + name="pre-publication-abbreviation" + class="form-control" + value="{{phenotype['Pre_publication_abbreviation'] or ''}}" /> + </div> + </div> + + <div class="form-group"> + <label for="txt-post-publication-description" + class="control-label col-sm-2">Post-Publication Description</label> + <div class="col-sm-10"> + <input type="text" + id="txt-post-publication-description" + name="post-publication-description" + class="form-control" + value="{{phenotype['Post_publication_description'] or ''}}" /> + </div> + </div> + + <div class="form-group"> + <label for="txt-post-publication-abbreviation" + class="control-label col-sm-2">Post-Publication Abbreviation</label> + <div class="col-sm-10"> + <input type="text" + id="txt-post-publication-abbreviation" + name="post-publication-abbreviation" + class="form-control" + value="{{phenotype['Post_publication_abbreviation'] or ''}}" /> + </div> + </div> + + <div class="form-group"> + <label for="txt-original-description" + class="control-label col-sm-2">Original Description</label> + <div class="col-sm-10"> + <input type="text" + id="txt-original-description" + name="original-description" + class="form-control" + value="{{phenotype['Original_description'] or ''}}" /> + </div> + </div> + + <div class="form-group"> + <label for="txt-units" + class="control-label col-sm-2">units</label> + <div class="col-sm-10"> + <input type="text" + id="txt-units" + name="units" + class="form-control" + required="required" + value="{{phenotype['Units']}}" /> + </div> + </div> + + <div class="form-group"> + <div class="col-sm-offset-2 col-sm-10"> + <input type="submit" + name="submit" + class="btn btn-primary" + value="update basic metadata"> + </div> + </div> + </form> +</div> + + +<div class="row"> + <h3 class="subheading">phenotype data</h3> + <form id="frm-edit-phenotype-data" + class="form-horizontal" + method="POST" + action="{{url_for( + 'species.populations.phenotypes.edit_phenotype_data', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id, + xref_id=xref_id)}}"> + <div style="max-height: 23.37em;overflow-y: scroll;"> + <table class="table table-striped table-responsive table-form-table"> + <thead style="position: sticky; top: 0;"> + <tr> + <th>#</th> + <th>Sample</th> + <th>Value</th> + {%if population.Family in families_with_se_and_n%} + <th>Standard-Error</th> + <th>Number of Samples</th> + {%endif%} + </tr> + </thead> + + <tbody> + {%for item in phenotype.data%} + <tr> + <td>{{loop.index}}</td> + <td>{{item.StrainName}}</td> + <td> + <input type="text" + name="value-new::{{item.DataId}}::{{item.StrainId}}" + value="{{item.value}}" + class="form-control" /> + <input type="hidden" + name="value-original::{{item.DataId}}::{{item.StrainId}}" + value="{{item.value}}" /></td> + {%if population.Family in families_with_se_and_n%} + <td> + <input type="text" + name="se-new::{{item.DataId}}::{{item.StrainId}}" + value="{{item.error or ''}}" + data-original-value="{{item.error or ''}}" + class="form-control" /> + <input type="hidden" + name="se-original::{{item.DataId}}::{{item.StrainId}}" + value="{{item.error or ''}}" /></td> + <td> + <input type="text" + name="n-new::{{item.DataId}}::{{item.StrainId}}" + value="{{item.count or ''}}" + data-original-value="{{item.count or "-"}}" + class="form-control" /> + <input type="hidden" + name="n-original::{{item.DataId}}::{{item.StrainId}}" + value="{{item.count or ''}}" /></td> + {%endif%} + </tr> + {%endfor%} + </tbody> + </table> + </div> + <div class="form-group"> + <div class="col-sm-offset-2 col-sm-10"> + <input type="submit" + name="submit" + class="btn btn-primary" + value="update data" /> + </div> + </div> + </form> +</div> + + +<div class="row"> + <h3 class="subheading">publication information</h3> + <p>Use the form below to update the publication information for this + phenotype.</p> + <form id="frm-edit-phenotype-pub-data" + class="form-horizontal" + method="POST" + action="#"> + <div class="form-group"> + <label for="txt-pubmed-id" class="control-label col-sm-2">Pubmed ID</label> + <div class="col-sm-10"> + <input id="txt-pubmed-id" name="pubmed-id" type="text" + class="form-control" /> + <span class="form-text text-muted"> + Enter your publication's PubMed ID.</span> + </div> + </div> + + <div class="form-group"> + <label for="txt-publication-authors" class="control-label col-sm-2">Authors</label> + <div class="col-sm-10"> + <input id="txt-publication-authors" name="publication-authors" + type="text" class="form-control" /> + <span class="form-text text-muted"> + Enter the authors.</span> + </div> + </div> + + <div class="form-group"> + <label for="txt-publication-title" class="control-label col-sm-2"> + Publication Title</label> + <div class="col-sm-10"> + <input id="txt-publication-title" name="publication-title" type="text" + class="form-control" /> + <span class="form-text text-muted"> + Enter your publication's title.</span> + </div> + </div> + + <div class="form-group"> + <label for="txt-publication-abstract" class="control-label col-sm-2"> + Publication Abstract</label> + <div class="col-sm-10"> + <textarea id="txt-publication-abstract" name="publication-abstract" + class="form-control" rows="10"></textarea> + <span class="form-text text-muted"> + Enter the abstract for your publication.</span> + </div> + </div> + + <div class="form-group"> + <label for="txt-publication-journal" class="control-label col-sm-2">Journal</label> + <div class="col-sm-10"> + <input id="txt-publication-journal" name="journal" type="text" + class="form-control" /> + <span class="form-text text-muted"> + Enter the name of the journal where your work was published.</span> + </div> + </div> + + <div class="form-group"> + <label for="txt-publication-volume" class="control-label col-sm-2">Volume</label> + <div class="col-sm-10"> + <input id="txt-publication-volume" name="publication-volume" type="text" + class="form-control" /> + <span class="form-text text-muted"> + Enter the volume in the following format …</span> + </div> + </div> + + <div class="form-group"> + <label for="txt-publication-pages" class="control-label col-sm-2">Pages</label> + <div class="col-sm-10"> + <input id="txt-publication-pages" name="publication-pages" type="text" + class="form-control" /> + <span class="form-text text-muted"> + Enter the journal volume where your work was published.</span> + </div> + </div> + + <div class="form-group"> + <label for="select-publication-month" class="control-label col-sm-2"> + Publication Month</label> + <div class="col-sm-10"> + <select id="select-publication-month" name="publication-month" + class="form-control"> + {%for month in monthnames%} + <option value="{{month | lower}}" + {%if current_month | lower == month | lower%} + selected="selected" + {%endif%}>{{month | capitalize}}</option> + {%endfor%} + </select> + <span class="form-text text-muted"> + Select the month when the work was published. + <span class="text-danger"> + This cannot be before, say 1600 and cannot be in the future!</span></span> + </div> + </div> + + <div class="form-group"> + <label for="txt-publication-year" class="control-label col-sm-2">Publication Year</label> + <div class="col-sm-10"> + <input id="txt-publication-year" name="publication-year" type="text" + class="form-control" value="{{current_year}}" /> + <span class="form-text text-muted"> + Enter the year your work was published. + <span class="text-danger"> + This cannot be before, say 1600 and cannot be in the future!</span> + </span> + </div> + </div> + <div class="form-group"> + <div class="col-sm-offset-2 col-sm-10"> + <input type="submit" + name="submit" + class="btn btn-primary not-implemented" + value="update publication" /> + </div> + </div> + </form> +</div> + +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/phenotypes/index.html b/uploader/templates/phenotypes/index.html new file mode 100644 index 0000000..689c28e --- /dev/null +++ b/uploader/templates/phenotypes/index.html @@ -0,0 +1,21 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-select-species.html" import select_species_form%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + {{select_species_form(url_for("species.populations.phenotypes.index"), species)}} +</div> +{%endblock%} + + +{%block javascript%} +<script type="text/javascript" src="/static/js/species.js"></script> +{%endblock%} diff --git a/uploader/templates/phenotypes/job-status.html b/uploader/templates/phenotypes/job-status.html new file mode 100644 index 0000000..12963c1 --- /dev/null +++ b/uploader/templates/phenotypes/job-status.html @@ -0,0 +1,155 @@ +{%extends "phenotypes/base.html"%} +{%from "cli-output.html" import cli_output%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "phenotypes/macro-display-pheno-dataset-card.html" import display_pheno_dataset_card%} + +{%block extrameta%} +{%if job and job.status not in ("success", "completed:success", "error", "completed:error")%} +<meta http-equiv="refresh" content="5" /> +{%endif%} +{%endblock%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="add-phenotypes"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">View Datasets</a> +</li> +{%endblock%} + +{%block contents%} + +{%if job%} +<h4 class="subheading">Progress</h4> +<div class="row" style="overflow:scroll;"> + <p><strong>Process Status:</strong> {{job.status}}</p> + {%if metadata%} + <table class="table table-responsive"> + <thead> + <tr> + <th>File</th> + <th>Status</th> + <th>Lines Processed</th> + <th>Total Errors</th> + </tr> + </thead> + + <tbody> + {%for file,meta in metadata.items()%} + <tr> + <td>{{file}}</td> + <td>{{meta.status}}</td> + <td>{{meta.linecount}}</td> + <td>{{meta["total-errors"]}}</td> + </tr> + {%endfor%} + </tbody> + </table> + {%endif%} +</div> + +<div class="row"> + {%if job.status in ("completed:success", "success")%} + <p> + {%if errors | length == 0%} + <a href="{{url_for('species.populations.phenotypes.review_job_data', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id, + job_id=job_id)}}" + class="btn btn-primary" + title="Continue to process data">Continue</a> + {%else%} + <span class="text-muted" + disabled="disabled" + style="border: solid 2px;border-radius: 5px;padding: 0.3em;"> + Cannot continue due to errors. Please fix the errors first. + </span> + {%endif%} + </p> + {%endif%} +</div> + +<h4 class="subheading">Errors</h4> +<div class="row" style="max-height: 20em; overflow: scroll;"> + {%if errors | length == 0 %} + <p class="text-info"> + <span class="glyphicon glyphicon-info-sign"></span> + No errors found so far + </p> + {%else%} + <table class="table table-responsive"> + <thead style="position: sticky; top: 0; background: white;"> + <tr> + <th>File</th> + <th>Row</th> + <th>Column</th> + <th>Value</th> + <th>Message</th> + </tr> + </thead> + + <tbody style="font-size: 0.9em;"> + {%for error in errors%} + <tr> + <td>{{error.filename}}</td> + <td>{{error.rowtitle}}</td> + <td>{{error.coltitle}}</td> + <td>{%if error.cellvalue | length > 25%} + {{error.cellvalue[0:24]}}… + {%else%} + {{error.cellvalue}} + {%endif%} + </td> + <td> + {%if error.message | length > 250 %} + {{error.message[0:249]}}… + {%else%} + {{error.message}} + {%endif%} + </td> + </tr> + {%endfor%} + </tbody> + </table> + {%endif%} +</div> + +<div class="row"> + {{cli_output(job, "stdout")}} +</div> + +<div class="row"> + {{cli_output(job, "stderr")}} +</div> + +{%else%} +<div class="row"> + <h3 class="text-danger">No Such Job</h3> + <p>Could not find a job with the ID: {{job_id}}</p> + <p> + Please go back to + <a href="{{url_for('species.populations.phenotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}" + title="'{{dataset.Name}}' dataset page"> + the '{{dataset.Name}}' dataset page</a> + to upload new phenotypes or edit existing ones.</p> +</div> +{%endif%} +{%endblock%} + +{%block sidebarcontents%} +{{display_pheno_dataset_card(species, population, dataset)}} +{%endblock%} diff --git a/uploader/templates/phenotypes/list-datasets.html b/uploader/templates/phenotypes/list-datasets.html new file mode 100644 index 0000000..2cf2c7f --- /dev/null +++ b/uploader/templates/phenotypes/list-datasets.html @@ -0,0 +1,68 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="list-datasets"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.list_datasets', + species_id=species.SpeciesId, + population_id=population.Id)}}">List Datasets</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + {%if datasets | length > 0%} + <p>The dataset(s) available for this population is/are:</p> + + <table class="table"> + <thead> + <tr> + <th>Name</th> + <th>Full Name</th> + <th>Short Name</th> + </tr> + </thead> + + <tbody> + {%for dataset in datasets%} + <tr> + <td><a href="{{url_for('species.populations.phenotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">{{dataset.Name}}</a></td> + <td>{{dataset.FullName}}</td> + <td>{{dataset.ShortName}}</td> + </tr> + {%endfor%} + </tbody> + </table> + {%else%} + <p>Phenotypes need to go into a dataset. We do not currently have a dataset + for species <strong>'{{species["FullName"]}} ({{species["Name"]}})'</strong> + phenotypes.</p> + + <p>Do, please, create a new dataset by clicking on the "Create Dataset" button + below and following the prompts/instructions.</p> + <p><a href="{{url_for('species.populations.phenotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}" + class="btn btn-primary" + title="Create a new phenotype dataset.">create dataset</a></p> + {%endif%} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/phenotypes/load-phenotypes-success.html b/uploader/templates/phenotypes/load-phenotypes-success.html new file mode 100644 index 0000000..645be16 --- /dev/null +++ b/uploader/templates/phenotypes/load-phenotypes-success.html @@ -0,0 +1,42 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "phenotypes/macro-display-pheno-dataset-card.html" import display_pheno_dataset_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="load-phenotypes-success"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">Add Phenotypes</a> +</li> +{%endblock%} + +{%block contents%} +<div class="row"> + <p>You have successfully loaded + <!-- maybe indicate the number of phenotypes here? -->your + new phenotypes into the database.</p> + <!-- TODO: Maybe notify user that they have sole access. --> + <!-- TODO: Maybe provide a link to go to GeneNetwork to view the data. --> + <p>View your data + <a href="{{search_page_uri}}" + target="_blank">on GeneNetwork2</a>. + You might need to login to GeneNetwork2 to view specific traits.</p> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_pheno_dataset_card(species, population, dataset)}} +{%endblock%} + + +{%block more_javascript%}{%endblock%} diff --git a/uploader/templates/phenotypes/macro-display-pheno-dataset-card.html b/uploader/templates/phenotypes/macro-display-pheno-dataset-card.html new file mode 100644 index 0000000..11b108b --- /dev/null +++ b/uploader/templates/phenotypes/macro-display-pheno-dataset-card.html @@ -0,0 +1,31 @@ +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%macro display_pheno_dataset_card(species, population, dataset)%} +{{display_population_card(species, population)}} + +<div class="card"> + <div class="card-body"> + <h5 class="card-title">Phenotypes' Dataset</h5> + <div class="card-text"> + <table class="table"> + <tbody> + <tr> + <td>Name</td> + <td>{{dataset.Name}}</td> + </tr> + + <tr> + <td>Full Name</td> + <td>{{dataset.FullName}}</td> + </tr> + + <tr> + <td>Short Name</td> + <td>{{dataset.ShortName}}</td> + </tr> + </tbody> + </table> + </div> + </div> +</div> +{%endmacro%} diff --git a/uploader/templates/phenotypes/macro-display-preview-table.html b/uploader/templates/phenotypes/macro-display-preview-table.html new file mode 100644 index 0000000..5a4c422 --- /dev/null +++ b/uploader/templates/phenotypes/macro-display-preview-table.html @@ -0,0 +1,19 @@ +{%macro display_preview_table(tableid, filetype)%} +<div class="card"> + <div class="card-body"> + <h5 class="card-title">{{filetype | title}}: File Preview</h5> + <div class="card-text" style="overflow: scroll;"> + <table id="{{tableid}}" class="table table-condensed table-responsive"> + <thead> + <tr> + </tr> + <tbody> + <tr> + <td class="data-row-template text-info"></td> + </tr> + </tbody> + </table> + </div> + </div> +</div> +{%endmacro%} diff --git a/uploader/templates/phenotypes/macro-display-resumable-elements.html b/uploader/templates/phenotypes/macro-display-resumable-elements.html new file mode 100644 index 0000000..ed14ea5 --- /dev/null +++ b/uploader/templates/phenotypes/macro-display-resumable-elements.html @@ -0,0 +1,60 @@ +{%macro display_resumable_elements(id, title, help)%} +<div id="{{id}}" + class="resumable-elements visually-hidden" + style="background:#D4D4EE;border-radius: 5px;;padding: 1em;border-left: solid #B2B2CC 1px;border-bottom: solid #B2B2CC 2px;margin-top:0.3em;"> + <strong style="line-height: 1.2em;">{{title | title}}</strong> + + <span class="form-text text-muted">{{help | safe}}</span> + + <div id="{{id}}-selected-files" + class="resumable-selected-files" + style="display:flex;flex-direction:row;flex-wrap: wrap;justify-content:space-around;gap:10px 20px;"> + <div class="panel panel-info file-display-template visually-hidden"> + <div class="panel-heading filename">The Filename Goes Here!</div> + <div class="panel-body"> + <ul> + <li> + <strong>Name</strong>: + <span class="filename">the file's name</span></li> + + <li><strong>Size</strong>: <span class="filesize">0 MB</span></li> + + <li> + <strong>Unique Identifier</strong>: + <span class="fileuniqueid">brrr</span></li> + + <li> + <strong>Mime</strong>: + <span class="filemimetype">text/csv</span></li> + </ul> + </div> + </div> + </div> + + <a id="{{id}}-browse-button" + class="resumable-browse-button btn btn-info" + href="#{{id}}" + style="margin-left: 80%;">Browse</a> + + <div id="{{id}}-progress-bar" class="progress visually-hidden"> + <div class="progress-bar" + role="progress-bar" + aria-valuenow="60" + aria-valuemin="0" + aria-valuemax="100" + style="width: 0%;"> + Uploading: 60% + </div> + </div> + + <div id="{{id}}-cancel-resume-buttons"> + <a id="{{id}}-resume-button" + class="resumable-resume-button btn btn-info visually-hidden" + href="#">resume upload</a> + + <a id="{{id}}-cancel-button" + class="resumable-cancel-button btn btn-danger visually-hidden" + href="#">cancel upload</a> + </div> +</div> +{%endmacro%} diff --git a/uploader/templates/phenotypes/review-job-data.html b/uploader/templates/phenotypes/review-job-data.html new file mode 100644 index 0000000..859df74 --- /dev/null +++ b/uploader/templates/phenotypes/review-job-data.html @@ -0,0 +1,125 @@ +{%extends "phenotypes/base.html"%} +{%from "cli-output.html" import cli_output%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "phenotypes/macro-display-pheno-dataset-card.html" import display_pheno_dataset_card%} + +{%block extrameta%} +{%if not job%} +<meta http-equiv="refresh" + content="20; url={{url_for('species.populations.phenotypes.view_dataset', species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}" /> +{%endif%} +{%endblock%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="add-phenotypes"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">View Datasets</a> +</li> +{%endblock%} + +{%block contents%} + +{%if job%} +<div class="row"> + <h3 class="heading">Data Review</h3> + <p class="text-info"><strong> + The data has <em>NOT</em> been added/saved yet. Review the details below + and click "Continue" to save the data.</strong></p> + <p>The “<strong>{{dataset.FullName}}</strong>” dataset from the + “<strong>{{population.FullName}}</strong>” population of the + species “<strong>{{species.SpeciesName}} ({{species.FullName}})</strong>” + will be updated as follows:</p> + + <ul> + {%if publication%} + <li>All {{summary.get("pheno", {}).get("total-data-rows", "0")}} phenotypes + are linked to the following publication: + <ul> + <li><strong>Publication Title:</strong> + {{publication.Title or "—"}}</li> + <li><strong>Author(s):</strong> + {{publication.Authors or "—"}}</li> + </ul> + </li> + {%endif%} + {%for ftype in ("phenocovar", "pheno", "phenose", "phenonum")%} + {%if summary.get(ftype, False)%} + <li>A total of {{summary[ftype]["number-of-files"]}} files will be processed + adding {%if ftype == "phenocovar"%}(possibly){%endif%} + {{summary[ftype]["total-data-rows"]}} new + {%if ftype == "phenocovar"%} + phenotypes + {%else%} + {{summary[ftype]["description"]}} rows + {%endif%} + to the database. + </li> + {%endif%} + {%endfor%} + </ul> + + <form id="frm-review-phenotype-data" + method="POST" + action="{{url_for('species.populations.phenotypes.load_data_to_database', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}"> + <input type="hidden" name="data-qc-job-id" value="{{job.jobid}}" /> + <input type="submit" + value="continue" + class="btn btn-primary" /> + </form> +</div> +{%else%} +<div class="row"> + <h4 class="subheading">Invalid Job</h3> + <p class="text-danger"> + Could not find a job with the ID: <strong>{{job_id}}.</p> + <p>You will be redirected in + <span id="countdown-element" class="text-info">20</span> second(s)</p> + <p class="text-muted"> + <small> + If you are not redirected, please + <a href="{{url_for( + 'species.populations.phenotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">click here</a> to continue + </small> + </p> +</div> +{%endif%} +{%endblock%} + +{%block sidebarcontents%} +{{display_pheno_dataset_card(species, population, dataset)}} +{%endblock%} + + +{%block javascript%} +<script type="text/javascript"> + $(document).ready(function() { + var countdown = 20; + var countdown_element = $("#countdown-element"); + if(countdown_element.length === 1) { + intv = window.setInterval(function() { + countdown = countdown - 1; + countdown_element.html(countdown); + }, 1000); + } + }); +</script> +{%endblock%} diff --git a/uploader/templates/phenotypes/select-population.html b/uploader/templates/phenotypes/select-population.html new file mode 100644 index 0000000..48c19b1 --- /dev/null +++ b/uploader/templates/phenotypes/select-population.html @@ -0,0 +1,26 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} +{%from "populations/macro-select-population.html" import select_population_form%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + + +<div class="row"> + {{select_population_form(url_for("species.populations.phenotypes.select_population", species_id=species.SpeciesId), species, populations)}} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} + +{%block javascript%} +<script type="text/javascript" src="/static/js/populations.js"></script> +{%endblock%} diff --git a/uploader/templates/phenotypes/view-dataset.html b/uploader/templates/phenotypes/view-dataset.html new file mode 100644 index 0000000..21563d6 --- /dev/null +++ b/uploader/templates/phenotypes/view-dataset.html @@ -0,0 +1,244 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="view-dataset"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">View</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>The basic dataset details are:</p> + + <table class="table"> + <thead> + <tr> + <th>Name</th> + <th>Full Name</th> + <th>Short Name</th> + </tr> + </thead> + + <tbody> + <tr> + <td>{{dataset.Name}}</td> + <td>{{dataset.FullName}}</td> + <td>{{dataset.ShortName}}</td> + </tr> + </tbody> + </table> +</div> + +<div class="row"> + <p><a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}" + title="Add a bunch of phenotypes" + class="btn btn-primary">Add phenotypes</a></p> +</div> + +<div class="row"> + <h2>Phenotype Data</h2> + + <p>Click on any of the phenotypes in the table below to view and edit that + phenotype's data.</p> + <p>Use the search to filter through all the phenotypes and find specific + phenotypes of interest.</p> +</div> + + +<div class="row"> + + <table id="tbl-phenotypes-list" class="table compact stripe cell-border"> + <thead> + <tr> + <th></th> + <th>Index</th> + <th>Record</th> + <th>Description</th> + </tr> + </thead> + + <tbody></tbody> + </table> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} + + +{%block javascript%} +<script type="text/javascript"> + $(function() { + var species_id = {{species.SpeciesId}}; + var population_id = {{population.Id}}; + var dataset_id = {{dataset.Id}}; + var dataset_name = "{{dataset.Name}}"; + var data = {{phenotypes | tojson}}; + + var dtPhenotypesList = buildDataTable( + "#tbl-phenotypes-list", + data, + [ + { + data: function(pheno) { + return `<input type="checkbox" name="selected-phenotypes" ` + + `id="chk-selected-phenotypes-${pheno.InbredSetCode}_${pheno.xref_id}" ` + + `value="${pheno.InbredSetCode}_${pheno.xref_id}" ` + + `class="chk-row-select" />` + } + }, + {data: "sequence_number"}, + { + data: function(pheno, type, set, meta) { + var spcs_id = {{species.SpeciesId}}; + var pop_id = {{population.Id}}; + var dtst_id = {{dataset.Id}}; + return `<a href="/species/${spcs_id}` + + `/populations/${pop_id}` + + `/phenotypes/datasets/${dtst_id}` + + `/phenotype/${pheno.xref_id}` + + `" target="_blank">` + + `${pheno.InbredSetCode}_${pheno.xref_id}` + + `</a>`; + } + }, + { + data: function(pheno) { + return (pheno.Post_publication_description || + pheno.Original_description || + pheno.Pre_publication_description); + } + } + ], + { + select: "multi+shift", + layout: { + top2: { + buttons: [ + { + extend: "selectAll", + className: "btn btn-info", + titleAttr: "Click to select ALL records in the table." + }, + { + extend: "selectNone", + className: "btn btn-info", + titleAttr: "Click to deselect ANY selected record(s) in the table." + }, + { + text: "Bulk Edit (Download Data)", + className: "btn btn-info btn-bulk-edit", + titleAttr: "Click to download data for editing.", + action: (event, dt, node, config) => { + var phenoids = []; + var selected = dt.rows({selected: true, page: "all"}).data(); + for(var idx = 0; idx < selected.length; idx++) { + phenoids.push({ + phenotype_id: selected[idx].Id, + xref_id: selected[idx].xref_id + }); + } + if(phenoids.length == 0) { + alert("No record selected. Nothing to do!"); + return false; + } + + $(".btn-bulk-edit").prop("disabled", true); + $(".btn-bulk-edit").addClass("d-none"); + var spinner = $( + "<div id='bulk-edit-spinner' class='spinner-grow text-info'>"); + spinner_content = $( + "<span class='visually-hidden'>"); + spinner_content.html( + "Downloading data …"); + spinner.append(spinner_content) + $(".btn-bulk-edit").parent().append( + spinner); + + $.ajax( + (`/species/${species_id}/populations/` + + `${population_id}/phenotypes/datasets/` + + `${dataset_id}/edit-download`), + { + method: "POST", + data: JSON.stringify(phenoids), + xhrFields: { + responseType: "blob" + }, + success: (data, textStatus, jqXHR) => { + var link = document.createElement("a"); + uri = window.URL.createObjectURL(data); + link.href = uri; + link.download = `${dataset_name}_data.tsv`; + + document.body.appendChild(link); + link.click(); + window.URL.revokeObjectURL(uri); + link.remove(); + }, + error: (jQXHR, textStatus, errorThrown) => { + console.log("Experienced an error: ", textStatus); + console.log("The ERROR: ", errorThrown); + }, + complete: (jqXHR, textStatus) => { + $("#bulk-edit-spinner").remove(); + $(".btn-bulk-edit").removeClass( + "d-none"); + $(".btn-bulk-edit").prop( + "disabled", false); + }, + contentType: "application/json" + }); + } + }, + { + text: "Bulk Edit (Upload Data)", + className: "btn btn-info btn-bulk-edit", + titleAttr: "Click to upload edited data you got by clicking the `Bulk Edit (Download Data)` button.", + action: (event, dt, node, config) => { + window.location.assign( + `${window.location.protocol}//` + + `${window.location.host}` + + `/species/${species_id}` + + `/populations/${population_id}` + + `/phenotypes/datasets/${dataset_id}` + + `/edit-upload`) + } + } + ] + }, + top1Start: { + pageLength: { + text: "Show _MENU_ of _TOTAL_" + } + }, + topStart: "info", + top1End: null + }, + rowId: function(pheno) { + return `${pheno.InbredSetCode}_${pheno.xref_id}`; + } + }); + }); +</script> +{%endblock%} diff --git a/uploader/templates/phenotypes/view-phenotype.html b/uploader/templates/phenotypes/view-phenotype.html new file mode 100644 index 0000000..21ac501 --- /dev/null +++ b/uploader/templates/phenotypes/view-phenotype.html @@ -0,0 +1,135 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="view-phenotype"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.view_phenotype', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id, + xref_id=xref_id)}}">View Phenotype</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <div class="panel panel-default"> + <div class="panel-heading"><strong>Basic Phenotype Details</strong></div> + + <table class="table"> + <tbody> + <tr> + <td><strong>Phenotype</strong></td> + <td>{{phenotype.Post_publication_description or phenotype.Pre_publication_abbreviation or phenotype.Original_description}} + </tr> + <tr> + <td><strong>Database</strong></td> + <td>{{dataset.FullName}}</td> + </tr> + <tr> + <td><strong>Units</strong></td> + <td>{{phenotype.Units}}</td> + </tr> + {%for key,value in publish_data.items()%} + <tr> + <td><strong>{{key}}</strong></td> + <td>{{value}}</td> + </tr> + {%else%} + <tr> + <td colspan="2" class="text-muted"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + No publication data found. + </td> + </tr> + {%endfor%} + </tbody> + </table> + </div> +</div> + +{%if "group:resource:edit-resource" in privileges +or "group:resource:delete-resource" in privileges%} +<div class="row"> + <div class="btn-group btn-group-justified"> + <div class="btn-group"> + {%if "group:resource:edit-resource" in privileges%} + <a href="{{url_for('species.populations.phenotypes.edit_phenotype_data', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id, + xref_id=xref_id)}}" + title="Edit the values for the phenotype. This is meant to be used when you need to update only a few values." + class="btn btn-primary">Edit</a> + {%endif%} + </div> + <div class="btn-group"></div> + <div class="btn-group"> + {%if "group:resource:delete-resource" in privileges%} + <a href="#" + title="Delete the entire phenotype. This is useful when you need to change data for most or all of the fields for this phenotype." + class="btn btn-danger not-implemented" + disabled="disabled">delete</a> + {%endif%} + </div> + </div> +</div> +{%endif%} + +<div class="row"> + <div class="panel panel-default"> + <div class="panel-heading"><strong>Phenotype Data</strong></div> + {%if "group:resource:view-resource" in privileges%} + <table class="table"> + <thead> + <tr> + <th>#</th> + <th>Sample</th> + <th>Value</th> + {%if has_se%} + <th>SE</th> + <th>N</th> + {%endif%} + </tr> + </thead> + + <tbody> + {%for item in phenotype.data%} + <tr> + <td>{{loop.index}}</td> + <td>{{item.StrainName}}</td> + <td>{{item.value}}</td> + {%if has_se%} + <td>{{item.error or "-"}}</td> + <td>{{item.count or "-"}}</td> + {%endif%} + </tr> + {%endfor%} + </tbody> + </table> + {%else%} + <p class="text-danger"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + You do not currently have privileges to view this phenotype in greater + detail. + </p> + {%endif%} + </div> +</div> + +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/platforms/base.html b/uploader/templates/platforms/base.html new file mode 100644 index 0000000..dac965f --- /dev/null +++ b/uploader/templates/platforms/base.html @@ -0,0 +1,13 @@ +{%extends "species/base.html"%} + +{%block lvl3_breadcrumbs%} +<li {%if activelink=="platforms"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.platforms.index')}}"> + Sequencing Platforms</a> +</li> +{%block lvl4_breadcrumbs%}{%endblock%} +{%endblock%} diff --git a/uploader/templates/platforms/create-platform.html b/uploader/templates/platforms/create-platform.html new file mode 100644 index 0000000..0866d5e --- /dev/null +++ b/uploader/templates/platforms/create-platform.html @@ -0,0 +1,124 @@ +{%extends "platforms/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} + +{%block title%}Platforms — Create Platforms{%endblock%} + +{%block pagetitle%}Platforms — Create Platforms{%endblock%} + +{%block lvl3_breadcrumbs%} +<li {%if activelink=="create-platform"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.platforms.create_platform', + species_id=species.SpeciesId)}}">create platform</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <h2>Create New Platform</h2> + + <p>You can create a new genetic sequencing platform below.</p> +</div> + +<div class="row"> + <form id="frm-create-platform" + method="POST" + action="{{url_for('species.platforms.create_platform', + species_id=species.SpeciesId)}}"> + + <div class="form-group"> + <label for="txt-geo-platform" class="form-label">GEO Platform</label> + <input type="text" + id="txt-geo-platform" + name="geo-platform" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>This is the platform's + <a href="https://www.ncbi.nlm.nih.gov/geo/browse/?view=platforms&tax={{species.TaxonomyId}}" + title="Platforms for '{{species.FullName}}' on NCBI"> + accession value on NCBI</a>. If you do not know the value, click the + link and search on NCBI for species '{{species.FullName}}'.</p></small> + </div> + + <div class="form-group"> + <label for="txt-platform-name" class="form-label">Platform Name</label> + <input type="text" + id="txt-platform-name" + name="platform-name" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>This is name of the genetic sequencing platform.</p></small> + </div> + + <div class="form-group"> + <label for="txt-platform-shortname" class="form-label"> + Platform Short Name</label> + <input type="text" + id="txt-platform-shortname" + name="platform-shortname" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>Use the following conventions for this field: + <ol> + <li>Start with a 4-letter vendor code, e.g. "Affy" for "Affymetrix", "Illu" for "Illumina", etc.</li> + <li>Append an underscore to the 4-letter vendor code</li> + <li>Use the name of the array given by the vendor, e.g. U74AV2, MOE430A, etc.</li> + </ol> + </p> + </small> + </div> + + <div class="form-group"> + <label for="txt-platform-title" class="form-label">Platform Title</label> + <input type="text" + id="txt-platform-title" + name="platform-title" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>The full platform title. Sometimes, this is the same as the Platform + Name above.</p></small> + </div> + + <div class="form-group"> + <label for="txt-go-tree-value" class="form-label">GO Tree Value</label> + <input type="text" + id="txt-go-tree-value" + name="go-tree-value" + class="form-control" /> + <small class="form-text text-muted"> + <p>This is a Chip identification value useful for analysis with the + <strong> + <a href="https://www.geneweaver.org/" + title="Go to the GeneWeaver site." + target="_blank">GeneWeaver</a></strong> + and + <strong> + <a href="https://www.webgestalt.org/" + title="Go to the WEB-based GEne SeT AnaLysis Toolkit site." + target="_blank">WebGestalt</a></strong> + tools.<br /> + This can be left blank for custom platforms.</p></small> + </div> + + <div class="form-group"> + <input type="submit" + value="create new platform" + class="btn btn-primary" /> + </div> + </form> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} diff --git a/uploader/templates/platforms/index.html b/uploader/templates/platforms/index.html new file mode 100644 index 0000000..555b444 --- /dev/null +++ b/uploader/templates/platforms/index.html @@ -0,0 +1,25 @@ +{%extends "platforms/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-select-species.html" import select_species_form%} + +{%block title%}Platforms{%endblock%} + +{%block pagetitle%}Platforms{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>In this section, you will be able to view and manage the sequencing + platforms that are currently supported by GeneNetwork.</p> +</div> + +<div class="row"> + {{select_species_form(url_for("species.platforms.index"), species)}} +</div> +{%endblock%} + +{%block javascript%} +<script type="text/javascript" src="/static/js/species.js"></script> +{%endblock%} diff --git a/uploader/templates/platforms/list-platforms.html b/uploader/templates/platforms/list-platforms.html new file mode 100644 index 0000000..a6bcfdc --- /dev/null +++ b/uploader/templates/platforms/list-platforms.html @@ -0,0 +1,93 @@ +{%extends "platforms/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} + +{%block title%}Platforms — List Platforms{%endblock%} + +{%block pagetitle%}Platforms — List Platforms{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>View the list of the genetic sequencing platforms that are currently + supported by GeneNetwork.</p> + <p>If you cannot find the platform you wish to use, you can add it by clicking + the "New Platform" button below.</p> + <p><a href="{{url_for('species.platforms.create_platform', + species_id=species.SpeciesId)}}" + title="Create a new genetic sequencing platform for species {{species.FullName}}" + class="btn btn-primary">Create Platform</a></p> +</div> + +<div class="row"> + <h2>Supported Platforms</h2> + {%if platforms is defined and platforms | length > 0%} + <p>There are {{total_platforms}} platforms supported by GeneNetwork</p> + + <div class="row"> + <div class="col-md-2" style="text-align: start;"> + {%if start_from > 0%} + <a href="{{url_for('species.platforms.list_platforms', + species_id=species.SpeciesId, + start_from=start_from-count, + count=count)}}"> + <span class="glyphicon glyphicon-backward"></span> + Previous + </a> + {%endif%} + </div> + <div class="col-md-8" style="text-align: center;"> + Displaying platforms {{start_from+1}} to {{start_from+count if start_from+count < total_platforms else total_platforms}} of + {{total_platforms}} + </div> + <div class="col-md-2" style="text-align: end;"> + {%if start_from + count < total_platforms%} + <a href="{{url_for('species.platforms.list_platforms', + species_id=species.SpeciesId, + start_from=start_from+count, + count=count)}}"> + Next + <span class="glyphicon glyphicon-forward"></span> + </a> + {%endif%} + </div> + </div> + + <table class="table"> + <thead> + <tr> + <th></th> + <th>Platform Name</th> + <th><a href="https://www.ncbi.nlm.nih.gov/geo/browse/?view=platforms&tax={{species.TaxonomyId}}" + title="Gene Expression Omnibus: Platforms section" + target="_blank">GEO Platform</a></th> + <th>Title</th> + </tr> + </thead> + + <tbody> + {%for platform in platforms%} + <tr> + <td>{{platform.sequence_number}}</td> + <td>{{platform.GeneChipName}}</td> + <td><a href="https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc={{platform.GeoPlatform}}" + title="View platform on the Gene Expression Omnibus" + target="_blank">{{platform.GeoPlatform}}</a></td> + <td>{{platform.Title}}</td> + </tr> + {%endfor%} + </tbody> + </table> + {%else%} + <p class="text-warning"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + There are no platforms supported at this time!</p> + {%endif%} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} diff --git a/uploader/templates/populations/base.html b/uploader/templates/populations/base.html index d763fc1..9db8083 100644 --- a/uploader/templates/populations/base.html +++ b/uploader/templates/populations/base.html @@ -6,7 +6,13 @@ {%else%} class="breadcrumb-item" {%endif%}> + {%if population is mapping%} + <a href="{{url_for('species.populations.view_population', + species_id=species.SpeciesId, + population_id=population.Id)}}">{{population.Name}}</a> + {%else%} <a href="{{url_for('species.populations.index')}}">Populations</a> + {%endif%} </li> {%block lvl3_breadcrumbs%}{%endblock%} {%endblock%} diff --git a/uploader/templates/populations/create-population.html b/uploader/templates/populations/create-population.html index b57afba..c0c4f45 100644 --- a/uploader/templates/populations/create-population.html +++ b/uploader/templates/populations/create-population.html @@ -37,12 +37,15 @@ <div class="row"> <form method="POST" action="{{url_for('species.populations.create_population', - species_id=species.SpeciesId)}}"> + species_id=species.SpeciesId, + return_to=return_to)}}"> <legend>Create Population</legend> {{flash_all_messages()}} + <input type="hidden" name="return_to" value="{{return_to}}"> + <div {%if errors.population_fullname%} class="form-group has-error" {%else%} @@ -107,7 +110,13 @@ value="{{error_values.population_code or ''}}" class="form-control" /> <small class="form-text text-muted"> - … document what this field is for … + <p class="form-text text-muted"> + This is a 3-character code for your population, that is prepended to + the phenotype identifiers. e.g. For the "BXD Family" population, the + code is "BXD" and therefore, the phenotype identifiers for the + population look like the following examples: <em>BXD_10148</em>, + <em>BXD_10180</em>, <em>BXD_10197</em>, etc. + </p> </small> </div> @@ -159,7 +168,10 @@ {%endfor%} </select> <small class="form-text text-muted"> - <p>… provide some documentation on what this field does …</p> + <p> + This is a rough grouping of the populations in GeneNetwork into lists + of common types of populations. + </p> </small> </div> @@ -207,6 +219,28 @@ {%endif%}>{{gtype}}</option> {%endfor%} </select> + <small class="form-text text-muted text-danger"> + <p> + <span class="glyphicon glyphicon-exclamation-sign"></span> + This might be a poorly named field. + </p> + <p> + It probably has more to do with the mating crosses/crossings used to + produce the individuals in the population. I am no biologist, however, + and I'm leaving this here to remind myself to confirm this. + </p> + <p> + I still don't know what riset is.<br /> + … probably something to do with Recombinant Inbred Strains + </p> + <p> + Possible resources for this: + <ul> + <li>https://www.informatics.jax.org/silver/chapters/3-2.shtml</li> + <li>https://www.informatics.jax.org/silver/chapters/9-2.shtml</li> + </ul> + </p> + </small> </div> <div class="form-group"> diff --git a/uploader/templates/populations/index.html b/uploader/templates/populations/index.html index 3314516..d2bee77 100644 --- a/uploader/templates/populations/index.html +++ b/uploader/templates/populations/index.html @@ -11,8 +11,18 @@ {{flash_all_messages()}} <div class="row"> - To continue, you need to select the species: + <p> + Your experiment data will relate to a particular population from a + particular species. Let us know what species it is you want to work with + below. + </p> +</div> +<div class="row"> {{select_species_form(url_for("species.populations.index"), species)}} </div> {%endblock%} + +{%block javascript%} +<script type="text/javascript" src="/static/js/species.js"></script> +{%endblock%} diff --git a/uploader/templates/populations/list-populations.html b/uploader/templates/populations/list-populations.html index c83c18c..f780e94 100644 --- a/uploader/templates/populations/list-populations.html +++ b/uploader/templates/populations/list-populations.html @@ -51,6 +51,7 @@ <caption>Populations for {{species.FullName}}</caption> <thead> <tr> + <th></th> <th>Name</th> <th>Full Name</th> <th>Description</th> @@ -60,6 +61,7 @@ <tbody> {%for population in populations%} <tr> + <td>{{population["sequence_number"]}}</td> <td> <a href="{{url_for('species.populations.view_population', species_id=species.SpeciesId, diff --git a/uploader/templates/populations/macro-display-population-card.html b/uploader/templates/populations/macro-display-population-card.html index e68f8e3..16b477f 100644 --- a/uploader/templates/populations/macro-display-population-card.html +++ b/uploader/templates/populations/macro-display-population-card.html @@ -7,25 +7,34 @@ <div class="card-body"> <h5 class="card-title">Population</h5> <div class="card-text"> - <dl> - <dt>Name</dt> - <dd>{{population.Name}}</dd> + <table class="table"> + <tbody> + <tr> + <td>Name</td> + <td>{{population.Name}}</td> + </tr> - <dt>Full Name</dt> - <dd>{{population.FullName}}</dd> + <tr> + <td>Full Name</td> + <td>{{population.FullName}}</td> + </tr> - <dt>Code</dt> - <dd>{{population.InbredSetCode}}</dd> + <tr> + <td>Code</td> + <td>{{population.InbredSetCode}}</td> + </tr> - <dt>Genetic Type</dt> - <dd>{{population.GeneticType}}</dd> + <tr> + <td>Genetic Type</td> + <td>{{population.GeneticType}}</td> + </tr> - <dt>Family</dt> - <dd>{{population.Family}}</dd> - - <dt>Description</dt> - <dd>{{population.Description or "-"}}</dd> - </dl> + <tr> + <td>Family</td> + <td>{{population.Family}}</td> + </tr> + </tbody> + </table> </div> </div> </div> diff --git a/uploader/templates/populations/macro-select-population.html b/uploader/templates/populations/macro-select-population.html index af4fd3a..14b0510 100644 --- a/uploader/templates/populations/macro-select-population.html +++ b/uploader/templates/populations/macro-select-population.html @@ -1,30 +1,52 @@ -{%macro select_population_form(form_action, populations)%} -<form method="GET" action="{{form_action}}"> - <legend>Select Population</legend> - - <div class="form-group"> - <label for="select-population" class="form-label">Select Population</label> - <select id="select-population" - name="population_id" - class="form-control" - required="required"> - <option value="">Select Population</option> - {%for family in populations%} - <optgroup {%if family[0][1] is not none%} - label="{{family[0][1]}}" - {%else%} - label="Undefined" - {%endif%}> - {%for population in family[1]%} - <option value="{{population.Id}}">{{population.FullName}}</option> - {%endfor%} - </optgroup> - {%endfor%} - </select> +{%from "macro-step-indicator.html" import step_indicator%} + +{%macro select_population_form(form_action, species, populations)%} +<form method="GET" action="{{form_action}}" class="form-horizontal"> + + <h2>{{step_indicator("2")}} What population do you want to work with?</h2> + + {%if populations | length != 0%} + + <p class="form-text">Search for, and select the population from the table + below and click "Continue"</p> + + <div class="radio"> + <label class="control-label" for="rdo-cant-find-population"> + <input type="radio" id="rdo-cant-find-population" + name="population_id" value="CREATE-POPULATION" /> + I cannot find the population I want — create it! + </label> + </div> + + <div class="col-sm-offset-10 col-sm-2"> + <input type="submit" value="continue" class="btn btn-primary" /> + </div> + + <div style="margin-top:3em;"> + <table id="tbl-select-population" class="table compact stripe" + data-populations-list='{{populations | tojson}}'> + <thead> + <tr> + <th></th> + <th>Population</th> + </tr> + </thead> + + <tbody></tbody> + </table> </div> - <div class="form-group"> - <input type="submit" value="Select" class="btn btn-primary" /> + {%else%} + <p class="form-text"> + There are no populations currently defined for {{species['FullName']}} + ({{species['SpeciesName']}}).<br /> + Click "Continue" to create the first!</p> + <input type="hidden" name="population_id" value="CREATE-POPULATION" /> + + <div class="col-sm-offset-10 col-sm-2"> + <input type="submit" value="continue" class="btn btn-primary" /> </div> + {%endif%} + </form> {%endmacro%} diff --git a/uploader/templates/rqtl2/create-tissue-success.html b/uploader/templates/populations/rqtl2/create-tissue-success.html index d6fe154..d6fe154 100644 --- a/uploader/templates/rqtl2/create-tissue-success.html +++ b/uploader/templates/populations/rqtl2/create-tissue-success.html diff --git a/uploader/templates/populations/rqtl2/index.html b/uploader/templates/populations/rqtl2/index.html new file mode 100644 index 0000000..ec6ffb8 --- /dev/null +++ b/uploader/templates/populations/rqtl2/index.html @@ -0,0 +1,54 @@ +{%extends "base.html"%} +{%from "flash_messages.html" import flash_messages%} + +{%block title%}Data Upload{%endblock%} + +{%block contents%} +<h1 class="heading">R/qtl2 data upload</h1> + +<h2>R/qtl2 Upload</h2> + +<div class="row"> + <form method="POST" action="{{url_for('expression-data.rqtl2.select_species')}}" + id="frm-rqtl2-upload"> + <legend class="heading">upload R/qtl2 bundle</legend> + {{flash_messages("error-rqtl2")}} + + <div class="form-group"> + <label for="select:species" class="form-label">Species</label> + <select id="select:species" + name="species_id" + required="required" + class="form-control"> + <option value="">Select species</option> + {%for spec in species%} + <option value="{{spec.SpeciesId}}">{{spec.MenuName}}</option> + {%endfor%} + </select> + <small class="form-text text-muted"> + Data that you upload to the system should belong to a know species. + Here you can select the species that you wish to upload data for. + </small> + </div> + + <input type="submit" class="btn btn-primary" value="submit" /> + </form> +</div> + +<div class="row"> + <h2 class="heading">R/qtl2 Bundles</h2> + + <div class="explainer"> + <p>This feature combines and extends the two upload methods below. Instead of + uploading one item at a time, the R/qtl2 bundle you upload can contain both + the genotypes data (samples/individuals/cases and their data) and the + expression data.</p> + <p>The R/qtl2 bundle, additionally, can contain extra metadata, that neither + of the methods below can handle.</p> + + <a href="{{url_for('expression-data.rqtl2.select_species')}}" + title="Upload a zip bundle of R/qtl2 files"> + <button class="btn btn-primary">upload R/qtl2 bundle</button></a> + </div> +</div> +{%endblock%} diff --git a/uploader/templates/rqtl2/no-such-job.html b/uploader/templates/populations/rqtl2/no-such-job.html index b17004f..b17004f 100644 --- a/uploader/templates/rqtl2/no-such-job.html +++ b/uploader/templates/populations/rqtl2/no-such-job.html diff --git a/uploader/templates/rqtl2/rqtl2-job-error.html b/uploader/templates/populations/rqtl2/rqtl2-job-error.html index 9817518..9817518 100644 --- a/uploader/templates/rqtl2/rqtl2-job-error.html +++ b/uploader/templates/populations/rqtl2/rqtl2-job-error.html diff --git a/uploader/templates/rqtl2/rqtl2-job-results.html b/uploader/templates/populations/rqtl2/rqtl2-job-results.html index 4ecd415..4ecd415 100644 --- a/uploader/templates/rqtl2/rqtl2-job-results.html +++ b/uploader/templates/populations/rqtl2/rqtl2-job-results.html diff --git a/uploader/templates/rqtl2/rqtl2-job-status.html b/uploader/templates/populations/rqtl2/rqtl2-job-status.html index e896f88..e896f88 100644 --- a/uploader/templates/rqtl2/rqtl2-job-status.html +++ b/uploader/templates/populations/rqtl2/rqtl2-job-status.html diff --git a/uploader/templates/rqtl2/rqtl2-qc-job-error.html b/uploader/templates/populations/rqtl2/rqtl2-qc-job-error.html index 90e8887..90e8887 100644 --- a/uploader/templates/rqtl2/rqtl2-qc-job-error.html +++ b/uploader/templates/populations/rqtl2/rqtl2-qc-job-error.html diff --git a/uploader/templates/rqtl2/rqtl2-qc-job-results.html b/uploader/templates/populations/rqtl2/rqtl2-qc-job-results.html index b3c3a8f..b3c3a8f 100644 --- a/uploader/templates/rqtl2/rqtl2-qc-job-results.html +++ b/uploader/templates/populations/rqtl2/rqtl2-qc-job-results.html diff --git a/uploader/templates/rqtl2/rqtl2-qc-job-status.html b/uploader/templates/populations/rqtl2/rqtl2-qc-job-status.html index f4a6266..f4a6266 100644 --- a/uploader/templates/rqtl2/rqtl2-qc-job-status.html +++ b/uploader/templates/populations/rqtl2/rqtl2-qc-job-status.html diff --git a/uploader/templates/rqtl2/rqtl2-qc-job-success.html b/uploader/templates/populations/rqtl2/rqtl2-qc-job-success.html index f126835..f126835 100644 --- a/uploader/templates/rqtl2/rqtl2-qc-job-success.html +++ b/uploader/templates/populations/rqtl2/rqtl2-qc-job-success.html diff --git a/uploader/templates/populations/rqtl2/select-geno-dataset.html b/uploader/templates/populations/rqtl2/select-geno-dataset.html new file mode 100644 index 0000000..3233abc --- /dev/null +++ b/uploader/templates/populations/rqtl2/select-geno-dataset.html @@ -0,0 +1,69 @@ +{%extends "base.html"%} +{%from "flash_messages.html" import flash_messages%} + +{%block title%}Upload R/qtl2 Bundle{%endblock%} + +{%block contents%} +<h2 class="heading">Select Genotypes Dataset</h2> + +<div class="row"> + <p>Your R/qtl2 files bundle could contain a "geno" specification. You will + therefore need to select from one of the existing Genotype datasets or + create a new one.</p> + <p>This is the dataset where your data will be organised under.</p> +</div> + +<div class="row"> + <form id="frm-upload-rqtl2-bundle" + action="{{url_for('expression-data.rqtl2.select_geno_dataset', + species_id=species.SpeciesId, + population_id=population.InbredSetId)}}" + method="POST" + enctype="multipart/form-data"> + <legend class="heading">select from existing genotype datasets</legend> + + <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> + <input type="hidden" name="population_id" + value="{{population.InbredSetId}}" /> + <input type="hidden" name="rqtl2_bundle_file" + value="{{rqtl2_bundle_file}}" /> + + {{flash_messages("error-rqtl2-select-geno-dataset")}} + + <div class="form-group"> + <legend>Datasets</legend> + <label for="select:geno-datasets" class="form-label">Dataset</label> + <select id="select:geno-datasets" + name="geno-dataset-id" + required="required" + {%if datasets | length == 0%} + disabled="disabled" + {%endif%} + class="form-control" + aria-describedby="help-geno-dataset-select-dataset"> + <option value="">Select dataset</option> + {%for dset in datasets%} + <option value="{{dset['Id']}}">{{dset["Name"]}} ({{dset["FullName"]}})</option> + {%endfor%} + </select> + <span id="help-geno-dataset-select-dataset" class="form-text text-muted"> + Select from the existing genotype datasets for species + {{species.SpeciesName}} ({{species.FullName}}). + </span> + </div> + + <button type="submit" class="btn btn-primary">select dataset</button> + </form> +</div> + +<div class="row"> + <p>If the genotype dataset you need does not currently exist for your dataset, + go the <a href="{{url_for( + 'species.populations.genotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}" + title="Create a new genotypes dataset for {{species.FullName}}"> + genotypes page to create the genotype dataset</a></p> +</div> + +{%endblock%} diff --git a/uploader/templates/populations/rqtl2/select-population.html b/uploader/templates/populations/rqtl2/select-population.html new file mode 100644 index 0000000..ded425f --- /dev/null +++ b/uploader/templates/populations/rqtl2/select-population.html @@ -0,0 +1,57 @@ +{%extends "expression-data/index.html"%} +{%from "flash_messages.html" import flash_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} + +{%block title%}Select Grouping/Population{%endblock%} + +{%block contents%} +<h1 class="heading">Select grouping/population</h1> + +<div class="row"> + <p>The data is organised in a hierarchical form, beginning with + <em>species</em> at the very top. Under <em>species</em> the data is + organised by <em>population</em>, sometimes referred to as <em>grouping</em>. + (In some really old documents/systems, you might see this referred to as + <em>InbredSet</em>.)</p> + <p>In this section, you get to define what population your data is to be + organised by.</p> +</div> + +<div class="row"> + <form method="POST" + action="{{url_for('expression-data.rqtl2.select_population', + species_id=species.SpeciesId)}}"> + <legend class="heading">select grouping/population</legend> + {{flash_messages("error-select-population")}} + + <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> + + <div class="form-group"> + <label for="select:inbredset" class="form-label">population</label> + <select id="select:inbredset" + name="inbredset_id" + required="required" + class="form-control"> + <option value="">Select a grouping/population</option> + {%for pop in populations%} + <option value="{{pop.InbredSetId}}"> + {{pop.InbredSetName}} ({{pop.FullName}})</option> + {%endfor%} + </select> + <span class="form-text text-muted">Select the population for your data from + the list below.</span> + </div> + + <button type="submit" class="btn btn-primary" />select population</button> +</form> +</div> + +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} + + +{%block javascript%} +{%endblock%} diff --git a/uploader/templates/rqtl2/select-probeset-dataset.html b/uploader/templates/populations/rqtl2/select-probeset-dataset.html index 74f8f69..74f8f69 100644 --- a/uploader/templates/rqtl2/select-probeset-dataset.html +++ b/uploader/templates/populations/rqtl2/select-probeset-dataset.html diff --git a/uploader/templates/rqtl2/select-probeset-study-id.html b/uploader/templates/populations/rqtl2/select-probeset-study-id.html index e3fd9cc..e3fd9cc 100644 --- a/uploader/templates/rqtl2/select-probeset-study-id.html +++ b/uploader/templates/populations/rqtl2/select-probeset-study-id.html diff --git a/uploader/templates/rqtl2/select-tissue.html b/uploader/templates/populations/rqtl2/select-tissue.html index fe3080a..fe3080a 100644 --- a/uploader/templates/rqtl2/select-tissue.html +++ b/uploader/templates/populations/rqtl2/select-tissue.html diff --git a/uploader/templates/rqtl2/summary-info.html b/uploader/templates/populations/rqtl2/summary-info.html index 0adba2e..0adba2e 100644 --- a/uploader/templates/rqtl2/summary-info.html +++ b/uploader/templates/populations/rqtl2/summary-info.html diff --git a/uploader/templates/rqtl2/upload-rqtl2-bundle-step-01.html b/uploader/templates/populations/rqtl2/upload-rqtl2-bundle-step-01.html index 9d45c5f..9d45c5f 100644 --- a/uploader/templates/rqtl2/upload-rqtl2-bundle-step-01.html +++ b/uploader/templates/populations/rqtl2/upload-rqtl2-bundle-step-01.html diff --git a/uploader/templates/rqtl2/upload-rqtl2-bundle-step-02.html b/uploader/templates/populations/rqtl2/upload-rqtl2-bundle-step-02.html index 8210ed0..8210ed0 100644 --- a/uploader/templates/rqtl2/upload-rqtl2-bundle-step-02.html +++ b/uploader/templates/populations/rqtl2/upload-rqtl2-bundle-step-02.html diff --git a/uploader/templates/populations/view-population.html b/uploader/templates/populations/view-population.html index 31db54f..b23caeb 100644 --- a/uploader/templates/populations/view-population.html +++ b/uploader/templates/populations/view-population.html @@ -15,7 +15,7 @@ {%endif%}> <a href="{{url_for('species.populations.view_population', species_id=species.SpeciesId, - population_id=population.InbredSetId)}}">view population</a> + population_id=population.InbredSetId)}}">view</a> </li> {%endblock%} @@ -23,6 +23,9 @@ {%block contents%} <div class="row"> <h2>Population Details</h2> + + {{flash_all_messages()}} + <dl> <dt>Name</dt> <dd>{{population.Name}}</dd> @@ -66,16 +69,28 @@ manage samples</a> </li> <li> - <a href="#" title="Upload expression data">upload expression data</a> + <a href="{{url_for('species.populations.genotypes.list_genotypes', + species_id=species.SpeciesId, + population_id=population.Id)}}" + title="Manage genotypes for {{species.FullName}}">Manage Genotypes</a> + </li> + <li> + <a href="{{url_for('species.populations.phenotypes.list_datasets', + species_id=species.SpeciesId, + population_id=population.Id)}}" + title="Manage phenotype data.">manage phenotype data</a> </li> <li> - <a href="#" title="Upload phenotype data">upload phenotype data</a> + <a href="#" title="Manage expression data" + class="not-implemented">manage expression data</a> </li> <li> - <a href="#" title="Upload individual data">upload individual data</a> + <a href="#" title="Manage individual data" + class="not-implemented">manage individual data</a> </li> <li> - <a href="#" title="Upload RNA-Seq data">upload RNA-Seq data</a> + <a href="#" title="Manage RNA-Seq data" + class="not-implemented">manage RNA-Seq data</a> </li> </ul> </nav> diff --git a/uploader/templates/publications/base.html b/uploader/templates/publications/base.html new file mode 100644 index 0000000..db80bfa --- /dev/null +++ b/uploader/templates/publications/base.html @@ -0,0 +1,12 @@ +{%extends "base.html"%} + +{%block lvl1_breadcrumbs%} +<li {%if activelink=="publications"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('publications.index')}}">Publications</a> +</li> +{%block lvl2_breadcrumbs%}{%endblock%} +{%endblock%} diff --git a/uploader/templates/publications/create-publication.html b/uploader/templates/publications/create-publication.html new file mode 100644 index 0000000..3f828a9 --- /dev/null +++ b/uploader/templates/publications/create-publication.html @@ -0,0 +1,191 @@ +{%extends "publications/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} + +{%block title%}View Publication{%endblock%} + +{%block pagetitle%}View Publication{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <form id="frm-create-publication" + method="POST" + action="{{url_for('publications.create_publication', **request.args)}}" + class="form-horizontal"> + + <div class="row mb-3"> + <label for="txt-pubmed-id" class="col-sm-2 col-form-label"> + PubMed ID</label> + <div class="col-sm-10"> + <div class="input-group"> + <input type="text" + id="txt-pubmed-id" + name="pubmed-id" + class="form-control"/> + <div class="input-group-text"> + <button class="btn btn-outline-primary" + id="btn-search-pubmed-id">search</button> + </div> + </div> + <span id="search-pubmed-id-error" + class="form-text text-muted text-danger visually-hidden"> + </span> + <span class="form-text text-muted">This is the publication's ID on + <a href="https://pubmed.ncbi.nlm.nih.gov/" + title="Link to NCBI's PubMed service">NCBI's Pubmed Service</a> + </span> + </div> + </div> + + <div class="row mb-3"> + <label for="txt-publication-title" class="col-sm-2 col-form-label"> + Title</label> + <div class="col-sm-10"> + <input type="text" + id="txt-publication-title" + name="publication-title" + class="form-control" /> + <span class="form-text text-muted">Provide the publication's title here.</span> + </div> + </div> + + <div class="row mb-3"> + <label for="txt-publication-authors" class="col-sm-2 col-form-label"> + Authors</label> + <div class="col-sm-10"> + <input type="text" + id="txt-publication-authors" + name="publication-authors" + required="required" + class="form-control" /> + <span class="form-text text-muted"> + A publication <strong>MUST</strong> have an author. You <em>must</em> + provide a value for the authors field. + </span> + </div> + </div> + + <div class="row mb-3"> + <label for="txt-publication-journal" class="col-sm-2 col-form-label"> + Journal</label> + <div class="col-sm-10"> + <input type="text" + id="txt-publication-journal" + name="publication-journal" + class="form-control" /> + <span class="form-text text-muted">Provide the name journal where the + publication was done, here.</span> + </div> + </div> + + <div class="row mb-3"> + <label for="select-publication-month" + class="col-sm-2 col-form-label"> + Month</label> + <div class="col-sm-4"> + <select class="form-control" + id="select-publication-month" + name="publication-month"> + <option value="">Select a month</option> + <option value="january">January</option> + <option value="february">February</option> + <option value="march">March</option> + <option value="april">April</option> + <option value="may">May</option> + <option value="june">June</option> + <option value="july">July</option> + <option value="august">August</option> + <option value="september">September</option> + <option value="october">October</option> + <option value="november">November</option> + <option value="december">December</option> + </select> + <span class="form-text text-muted">Month of publication</span> + </div> + + <label for="txt-publication-year" + class="col-sm-2 col-form-label"> + Year</label> + <div class="col-sm-4"> + <input type="number" + id="txt-publication-year" + name="publication-year" + class="form-control" + min="1960" /> + <span class="form-text text-muted">Year of publication</span> + </div> + </div> + + <div class="row mb-3"> + <label for="txt-publication-volume" + class="col-sm-2 col-form-label"> + Volume</label> + <div class="col-sm-4"> + <input type="text" + id="txt-publication-volume" + name="publication-volume" + class="form-control"> + <span class="form-text text-muted">Journal volume</span> + </div> + + <label for="txt-publication-pages" + class="col-sm-2 col-form-label"> + Pages</label> + <div class="col-sm-4"> + <input type="text" + id="txt-publication-pages" + name="publication-pages" + class="form-control" /> + <span class="form-text text-muted">Journal pages for the publication</span> + </div> + </div> + + <div class="row mb-3"> + <label for="txt-abstract" class="col-sm-2 col-form-label">Abstract</label> + <div class="col-sm-10"> + <textarea id="txt-publication-abstract" + name="publication-abstract" + class="form-control" + rows="7"></textarea> + </div> + </div> + + <div class="row mb-3"> + <div class="col-sm-2"></div> + <div class="col-sm-8"> + <input type="submit" class="btn btn-primary" value="Add" /> + <input type="reset" class="btn btn-danger" /> + </div> + </div> + +</form> +</div> + +{%endblock%} + + +{%block javascript%} +<script type="text/javascript" src="/static/js/pubmed.js"></script> +<script type="text/javascript"> + $(function() { + $("#btn-search-pubmed-id").on("click", (event) => { + event.preventDefault(); + var search_button = event.target; + var pubmed_id = $("#txt-pubmed-id").val().trim(); + remove_class($("#txt-pubmed-id").parent(), "has-error"); + if(pubmed_id == "") { + add_class($("#txt-pubmed-id").parent(), "has-error"); + return false; + } + + search_button.disabled = true; + // Fetch publication details + fetch_publication_details(pubmed_id, + [() => {search_button.disabled = false;}]); + return false; + }); + }); +</script> +{%endblock%} diff --git a/uploader/templates/publications/index.html b/uploader/templates/publications/index.html new file mode 100644 index 0000000..f846d54 --- /dev/null +++ b/uploader/templates/publications/index.html @@ -0,0 +1,92 @@ +{%extends "publications/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} + +{%block title%}Publications{%endblock%} + +{%block pagetitle%}Publications{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row" style="padding-bottom: 1em;"> + <a href="{{url_for('publications.create_publication')}}" + class="btn btn-primary"> + add new publication</a> +</div> + +<div class="row"> + <table id="tbl-list-publications" class="table compact stripe"> + <thead> + <tr> + <th>#</th> + <th>PubMed ID</th> + <th>Title</th> + <th>Authors</th> + </tr> + </thead> + + <tbody></tbody> + </table> +</div> +{%endblock%} + + +{%block javascript%} +<script type="text/javascript"> + $(function() { + var publicationsDataTable = buildDataTable( + "#tbl-list-publications", + [], + [ + {data: "index"}, + { + data: (pub) => { + if(pub.PubMed_ID) { + return `<a href="https://pubmed.ncbi.nlm.nih.gov/` + + `${pub.PubMed_ID}/" target="_blank" ` + + `title="Link to publication on NCBI.">` + + `${pub.PubMed_ID}</a>`; + } + return ""; + } + }, + { + data: (pub) => { + var title = "⸻"; + if(pub.Title) { + title = pub.Title + } + return `<a href="/publications/view/${pub.Id}" ` + + `target="_blank" ` + + `title="Link to view publication details">` + + `${title}</a>`; + } + }, + { + data: (pub) => { + authors = pub.Authors.split(",").map( + (item) => {return item.trim();}); + if(authors.length > 1) { + return authors[0] + ", et. al."; + } + return authors[0]; + } + } + ], + { + ajax: { + url: "/publications/list", + dataSrc: "publications" + }, + scrollY: 700, + paging: false, + deferRender: true, + layout: { + topStart: "info", + topEnd: "search" + } + }); + }); +</script> +{%endblock%} diff --git a/uploader/templates/publications/view-publication.html b/uploader/templates/publications/view-publication.html new file mode 100644 index 0000000..388547a --- /dev/null +++ b/uploader/templates/publications/view-publication.html @@ -0,0 +1,78 @@ +{%extends "publications/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} + +{%block title%}View Publication{%endblock%} + +{%block pagetitle%}View Publication{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <table class="table"> + <tr> + <th>PubMed</th> + <td> + {%if publication.PubMed_ID%} + <a href="https://pubmed.ncbi.nlm.nih.gov/{{publication.PubMed_ID}}/" + target="_blank">{{publication.PubMed_ID}}</a> + {%else%} + — + {%endif%} + </td> + </tr> + <tr> + <th>Title</th> + <td>{{publication.Title or "—"}}</td> + </tr> + <tr> + <th>Authors</th> + <td>{{publication.Authors or "—"}}</td> + </tr> + <tr> + <th>Journal</th> + <td>{{publication.Journal or "—"}}</td> + </tr> + <tr> + <th>Published</th> + <td>{{publication.Month or ""}} {{publication.Year or "—"}}</td> + </tr> + <tr> + <th>Volume</th> + <td>{{publication.Volume or "—"}}</td> + </tr> + <tr> + <th>Pages</th> + <td>{{publication.Pages or "—"}}</td> + </tr> + <tr> + <th>Abstract</th> + <td> + {%for line in (publication.Abstract or "—").replace("\r\n", "<br />").replace("\n", "<br />").split("<br />")%} + <p>{{line}}</p> + {%endfor%} + </td> + </tr> + </table> +</div> + +<div class="row"> + <form id="frm-edit-delete-publication" method="POST" action="#"> + <input type="hidden" name="publication_id" value="{{publication.Id}}" /> + <div class="form-group"> + <input type="submit" value="edit" class="btn btn-primary not-implemented" /> + {%if linked_phenotypes | length == 0%} + <input type="submit" value="delete" class="btn btn-danger not-implemented" /> + {%endif%} + </div> + </form> +</div> +{%endblock%} + + +{%block javascript%} +<script type="text/javascript"> + $(function() {}); +</script> +{%endblock%} diff --git a/uploader/templates/rqtl2/create-geno-dataset-success.html b/uploader/templates/rqtl2/create-geno-dataset-success.html deleted file mode 100644 index bb6d63d..0000000 --- a/uploader/templates/rqtl2/create-geno-dataset-success.html +++ /dev/null @@ -1,55 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Upload R/qtl2 Bundle{%endblock%} - -{%block contents%} -<h2 class="heading">Select Genotypes Dataset</h2> - -<div class="explainer"> - <p>You successfully created the genotype dataset with the following - information. - <dl> - <dt>ID</dt> - <dd>{{geno_dataset.id}}</dd> - - <dt>Name</dt> - <dd>{{geno_dataset.name}}</dd> - - <dt>Full Name</dt> - <dd>{{geno_dataset.fname}}</dd> - - <dt>Short Name</dt> - <dd>{{geno_dataset.sname}}</dd> - - <dt>Created On</dt> - <dd>{{geno_dataset.today}}</dd> - - <dt>Public?</dt> - <dd>{%if geno_dataset.public == 0%}No{%else%}Yes{%endif%}</dd> - </dl> - </p> -</div> - -<div class="row"> - <form id="frm-upload-rqtl2-bundle" - action="{{url_for('expression-data.rqtl2.select_dataset_info', - species_id=species.SpeciesId, - population_id=population.InbredSetId)}}" - method="POST" - enctype="multipart/form-data"> - <legend class="heading">select from existing genotype datasets</legend> - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - <input type="hidden" name="population_id" - value="{{population.InbredSetId}}" /> - <input type="hidden" name="rqtl2_bundle_file" - value="{{rqtl2_bundle_file}}" /> - <input type="hidden" name="geno-dataset-id" - value="{{geno_dataset.id}}" /> - - <button type="submit" class="btn btn-primary">continue</button> - </form> -</div> - -{%endblock%} diff --git a/uploader/templates/rqtl2/create-probe-dataset-success.html b/uploader/templates/rqtl2/create-probe-dataset-success.html deleted file mode 100644 index 03b75c7..0000000 --- a/uploader/templates/rqtl2/create-probe-dataset-success.html +++ /dev/null @@ -1,59 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Upload R/qtl2 Bundle{%endblock%} - -{%block contents%} -<h2 class="heading">Create ProbeSet Dataset</h2> - -<div class="row"> - <p>You successfully created the ProbeSet dataset with the following - information. - <dl> - <dt>Averaging Method</dt> - <dd>{{avgmethod.Name}}</dd> - - <dt>ID</dt> - <dd>{{dataset.datasetid}}</dd> - - <dt>Name</dt> - <dd>{{dataset.name2}}</dd> - - <dt>Full Name</dt> - <dd>{{dataset.fname}}</dd> - - <dt>Short Name</dt> - <dd>{{dataset.sname}}</dd> - - <dt>Created On</dt> - <dd>{{dataset.today}}</dd> - - <dt>DataScale</dt> - <dd>{{dataset.datascale}}</dd> - </dl> - </p> -</div> - -<div class="row"> - <form id="frm-upload-rqtl2-bundle" - action="{{url_for('expression-data.rqtl2.select_dataset_info', - species_id=species.SpeciesId, - population_id=population.InbredSetId)}}" - method="POST" - enctype="multipart/form-data"> - <legend class="heading">Create ProbeSet dataset</legend> - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - <input type="hidden" name="population_id" - value="{{population.InbredSetId}}" /> - <input type="hidden" name="rqtl2_bundle_file" value="{{rqtl2_bundle_file}}" /> - <input type="hidden" name="geno-dataset-id" value="{{geno_dataset.Id}}" /> - <input type="hidden" name="tissueid" value="{{tissue.Id}}" /> - <input type="hidden" name="probe-study-id" value="{{study.Id}}" /> - <input type="hidden" name="probe-dataset-id" value="{{dataset.datasetid}}" /> - - <button type="submit" class="btn btn-primary">continue</button> - </form> -</div> - -{%endblock%} diff --git a/uploader/templates/rqtl2/create-probe-study-success.html b/uploader/templates/rqtl2/create-probe-study-success.html deleted file mode 100644 index e293f6f..0000000 --- a/uploader/templates/rqtl2/create-probe-study-success.html +++ /dev/null @@ -1,49 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Upload R/qtl2 Bundle{%endblock%} - -{%block contents%} -<h2 class="heading">Create ProbeSet Study</h2> - -<div class="row"> - <p>You successfully created the ProbeSet study with the following - information. - <dl> - <dt>ID</dt> - <dd>{{study.id}}</dd> - - <dt>Name</dt> - <dd>{{study.name}}</dd> - - <dt>Full Name</dt> - <dd>{{study.fname}}</dd> - - <dt>Short Name</dt> - <dd>{{study.sname}}</dd> - - <dt>Created On</dt> - <dd>{{study.today}}</dd> - </dl> - </p> - - <form id="frm-upload-rqtl2-bundle" - action="{{url_for('expression-data.rqtl2.select_dataset_info', - species_id=species.SpeciesId, - population_id=population.InbredSetId)}}" - method="POST" - enctype="multipart/form-data"> - <legend class="heading">Create ProbeSet study</legend> - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - <input type="hidden" name="population_id" - value="{{population.InbredSetId}}" /> - <input type="hidden" name="rqtl2_bundle_file" value="{{rqtl2_bundle_file}}" /> - <input type="hidden" name="geno-dataset-id" value="{{geno_dataset.Id}}" /> - <input type="hidden" name="probe-study-id" value="{{study.studyid}}" /> - - <button type="submit" class="btn btn-primary">continue</button> - </form> -</div> - -{%endblock%} diff --git a/uploader/templates/rqtl2/index.html b/uploader/templates/rqtl2/index.html deleted file mode 100644 index 8ce13bf..0000000 --- a/uploader/templates/rqtl2/index.html +++ /dev/null @@ -1,36 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Data Upload{%endblock%} - -{%block contents%} -<h1 class="heading">R/qtl2 data upload</h1> - -<h2>R/qtl2 Upload</h2> - -<form method="POST" action="{{url_for('expression-data.rqtl2.select_species')}}" - id="frm-rqtl2-upload"> - <legend class="heading">upload R/qtl2 bundle</legend> - {{flash_messages("error-rqtl2")}} - - <div class="form-group"> - <label for="select:species" class="form-label">Species</label> - <select id="select:species" - name="species_id" - required="required" - class="form-control"> - <option value="">Select species</option> - {%for spec in species%} - <option value="{{spec.SpeciesId}}">{{spec.MenuName}}</option> - {%endfor%} - </select> - <small class="form-text text-muted"> - Data that you upload to the system should belong to a know species. - Here you can select the species that you wish to upload data for. - </small> - </div> - - <button type="submit" class="btn btn-primary" />submit</button> -</form> - -{%endblock%} diff --git a/uploader/templates/rqtl2/select-geno-dataset.html b/uploader/templates/rqtl2/select-geno-dataset.html deleted file mode 100644 index 1db51e0..0000000 --- a/uploader/templates/rqtl2/select-geno-dataset.html +++ /dev/null @@ -1,144 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Upload R/qtl2 Bundle{%endblock%} - -{%block contents%} -<h2 class="heading">Select Genotypes Dataset</h2> - -<div class="row"> - <p>Your R/qtl2 files bundle contains a "geno" specification. You will - therefore need to select from one of the existing Genotype datasets or - create a new one.</p> - <p>This is the dataset where your data will be organised under.</p> -</div> - -<div class="row"> - <form id="frm-upload-rqtl2-bundle" - action="{{url_for('expression-data.rqtl2.select_geno_dataset', - species_id=species.SpeciesId, - population_id=population.InbredSetId)}}" - method="POST" - enctype="multipart/form-data"> - <legend class="heading">select from existing genotype datasets</legend> - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - <input type="hidden" name="population_id" - value="{{population.InbredSetId}}" /> - <input type="hidden" name="rqtl2_bundle_file" - value="{{rqtl2_bundle_file}}" /> - - {{flash_messages("error-rqtl2-select-geno-dataset")}} - - <div class="form-group"> - <legend>Datasets</legend> - <label for="select:geno-datasets" class="form-label">Dataset</label> - <select id="select:geno-datasets" - name="geno-dataset-id" - required="required" - {%if datasets | length == 0%} - disabled="disabled" - {%endif%} - class="form-control" - aria-describedby="help-geno-dataset-select-dataset"> - <option value="">Select dataset</option> - {%for dset in datasets%} - <option value="{{dset['Id']}}">{{dset["Name"]}} ({{dset["FullName"]}})</option> - {%endfor%} - </select> - <span id="help-geno-dataset-select-dataset" class="form-text text-muted"> - Select from the existing genotype datasets for species - {{species.SpeciesName}} ({{species.FullName}}). - </span> - </div> - - <button type="submit" class="btn btn-primary">select dataset</button> - </form> -</div> - -<div class="row"> - <p style="color:#FE3535; padding-left:20em; font-weight:bolder;">OR</p> -</div> - -<div class="row"> - <form id="frm-upload-rqtl2-bundle" - action="{{url_for('expression-data.rqtl2.create_geno_dataset', - species_id=species.SpeciesId, - population_id=population.InbredSetId)}}" - method="POST" - enctype="multipart/form-data"> - <legend class="heading">create a new genotype dataset</legend> - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - <input type="hidden" name="population_id" - value="{{population.InbredSetId}}" /> - <input type="hidden" name="rqtl2_bundle_file" - value="{{rqtl2_bundle_file}}" /> - - {{flash_messages("error-rqtl2-create-geno-dataset")}} - - <div class="form-group"> - <label for="txt:dataset-name" class="form-label">Name</label> - <input type="text" - id="txt:dataset-name" - name="dataset-name" - maxlength="100" - required="required" - class="form-control" - aria-describedby="help-geno-dataset-name" /> - <span id="help-geno-dataset-name" class="form-text text-muted"> - Provide the new name for the genotype dataset, e.g. "BXDGeno" - </span> - </div> - - <div class="form-group"> - <label for="txt:dataset-fullname" class="form-label">Full Name</label> - <input type="text" - id="txt:dataset-fullname" - name="dataset-fullname" - required="required" - maxlength="100" - class="form-control" - aria-describedby="help-geno-dataset-fullname" /> - - <span id="help-geno-dataset-fullname" class="form-text text-muted"> - Provide a longer name that better describes the genotype dataset, e.g. - "BXD Genotypes" - </span> - </div> - - <div class="form-group"> - <label for="txt:dataset-shortname" class="form-label">Short Name</label> - <input type="text" - id="txt:dataset-shortname" - name="dataset-shortname" - maxlength="100" - class="form-control" - aria-describedby="help-geno-dataset-shortname" /> - - <span id="help-geno-dataset-shortname" class="form-text text-muted"> - Provide a short name for the genotype dataset. This is optional. If not - provided, we'll default to the same value as the "Name" above. - </span> - </div> - - <div class="form-group"> - <input type="checkbox" - id="chk:dataset-public" - name="dataset-public" - checked="checked" - class="form-check" - aria-describedby="help-geno-datasent-public" /> - <label for="chk:dataset-public" class="form-check-label">Public?</label> - - <span id="help-geno-dataset-public" class="form-text text-muted"> - Specify whether the dataset will be available publicly. Check to make the - dataset publicly available and uncheck to limit who can access the dataset. - </span> - </div> - - <button type="submit" class="btn btn-primary">create new dataset</button> - </form> -</div> - -{%endblock%} diff --git a/uploader/templates/rqtl2/select-population.html b/uploader/templates/rqtl2/select-population.html deleted file mode 100644 index 7d27303..0000000 --- a/uploader/templates/rqtl2/select-population.html +++ /dev/null @@ -1,136 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Select Grouping/Population{%endblock%} - -{%block contents%} -<h1 class="heading">Select grouping/population</h1> - -<div class="explainer"> - <p>The data is organised in a hierarchical form, beginning with - <em>species</em> at the very top. Under <em>species</em> the data is - organised by <em>population</em>, sometimes referred to as <em>grouping</em>. - (In some really old documents/systems, you might see this referred to as - <em>InbredSet</em>.)</p> - <p>In this section, you get to define what population your data is to be - organised by.</p> -</div> - -<form method="POST" - action="{{url_for('expression-data.rqtl2.select_population', species_id=species.SpeciesId)}}"> - <legend class="heading">select grouping/population</legend> - {{flash_messages("error-select-population")}} - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - - <div class="form-group"> - <label for="select:inbredset" class="form-label">population</label> - <select id="select:inbredset" - name="inbredset_id" - required="required" - class="form-control"> - <option value="">Select a grouping/population</option> - {%for pop in populations%} - <option value="{{pop.InbredSetId}}"> - {{pop.InbredSetName}} ({{pop.FullName}})</option> - {%endfor%} - </select> - <span class="form-text text-muted">If you are adding data to an already existing - population, simply pick the population from this drop-down selector. If - you cannot find your population from this list, try the form below to - create a new one..</span> - </div> - - <button type="submit" class="btn btn-primary" />select population</button> -</form> - -<p style="color:#FE3535; padding-left:20em; font-weight:bolder;">OR</p> - -<form method="POST" - action="{{url_for('expression-data.rqtl2.create_population', species_id=species.SpeciesId)}}"> - <legend class="heading">create new grouping/population</legend> - {{flash_messages("error-create-population")}} - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - - <div class="form-group"> - <legend class="heading">mandatory</legend> - - <div class="form-group"> - <label for="txt:inbredset-name" class="form-label">name</label> - <input id="txt:inbredset-name" - name="inbredset_name" - type="text" - required="required" - maxlength="30" - placeholder="Enter grouping/population name" - class="form-control" /> - <span class="form-text text-muted">This is a short name that identifies the - population. Useful for menus, and quick scanning.</span> - </div> - - <div class="form-group"> - <label for="txt:" class="form-label">full name</label> - <input id="txt:inbredset-fullname" - name="inbredset_fullname" - type="text" - required="required" - maxlength="100" - placeholder="Enter the grouping/population's full name" - class="form-control" /> - <span class="form-text text-muted">This can be the same as the name above, or can - be longer. Useful for documentation, and human communication.</span> - </div> - </div> - - <div class="form-group"> - <legend class="heading">optional</legend> - - <div class="form-group"> - <label for="num:public" class="form-label">public?</label> - <select id="num:public" - name="public" - class="form-control"> - <option value="0">0 - Only accessible to authorised users</option> - <option value="1">1 - Publicly accessible to all users</option> - <option value="2" selected> - 2 - Publicly accessible to all users</option> - </select> - <span class="form-text text-muted">This determines whether the - population/grouping will appear on the menus for users.</span> - </div> - - <div class="form-group"> - <label for="txt:inbredset-family" class="form-label">family</label> - <input id="txt:inbredset-family" - name="inbredset_family" - type="text" - placeholder="I am not sure what this is about." - class="form-control" /> - <span class="form-text text-muted">I do not currently know what this is about. - This is a failure on my part to figure out what this is and provide a - useful description. Please feel free to remind me.</span> - </div> - - <div class="form-group"> - <label for="txtarea:" class="form-label">Description</label> - <textarea id="txtarea:description" - name="description" - rows="5" - placeholder="Enter a description of this grouping/population" - class="form-control"></textarea> - <span class="form-text text-muted"> - A long-form description of what the population consists of. Useful for - humans.</span> - </div> - </div> - - <button type="submit" class="btn btn-primary" /> - create grouping/population</button> -</form> - -{%endblock%} - - -{%block javascript%} -{%endblock%} diff --git a/uploader/templates/samples/index.html b/uploader/templates/samples/index.html index 7c88c01..ee98734 100644 --- a/uploader/templates/samples/index.html +++ b/uploader/templates/samples/index.html @@ -11,11 +11,13 @@ {{flash_all_messages()}} <div class="row"> - <p>Here, you can upload the samples/individuals that were used in your - experiments.</p> - <p>Since the samples are linked to specific species and populations, we will - need to first select them in the next few steps.</p> + <p>GeneNetwork has a selection of different species of organisms to choose from. Within those species, there are the populations of interest for a variety of experiments, from which you, the researcher, picked your samples (or individuals or cases) from. Here you can provide some basic details about your samples.</p> + <p>To start off, we will need to know what species and population your samples belong to. Please provide that information in the next sections.</p> {{select_species_form(url_for("species.populations.samples.index"), species)}} </div> {%endblock%} + +{%block javascript%} +<script type="text/javascript" src="/static/js/species.js"></script> +{%endblock%} diff --git a/uploader/templates/samples/list-samples.html b/uploader/templates/samples/list-samples.html index 8f1bf16..185e784 100644 --- a/uploader/templates/samples/list-samples.html +++ b/uploader/templates/samples/list-samples.html @@ -24,45 +24,56 @@ <div class="row"> <p> - Samples for population "{{population.FullName}}" from the + You selected the population "{{population.FullName}}" from the "{{species.FullName}}" species. </p> +</div> + +{%if samples | length > 0%} +<div class="row"> + <p> + This population already has <strong>{{total_samples}}</strong> + samples/individuals entered. You can explore the list of samples in this + population in the table below. + </p> +</div> + +<div class="row"> + <div class="col-md-2"> + {%if offset > 0:%} + <a href="{{url_for('species.populations.samples.list_samples', + species_id=species.SpeciesId, + population_id=population.Id, + from=offset-count, + count=count)}}"> + <span class="glyphicon glyphicon-backward"></span> + Previous + </a> + {%endif%} + </div> - {%if samples | length > 0%} - <div class="row"> - <div class="col-md-2"> - {%if offset > 0:%} - <a href="{{url_for('species.populations.samples.list_samples', - species_id=species.SpeciesId, - population_id=population.Id, - from=offset-count, - count=count)}}"> - <span class="glyphicon glyphicon-backward"></span> - Previous - </a> - {%endif%} - </div> - - <div class="col-md-8" style="text-align: center;"> - Samples {{offset}} — {{offset+(count if offset + count < total_samples else total_samples - offset)}} / {{total_samples}} - </div> - - <div class="col-md-2"> - {%if offset + count < total_samples:%} - <a href="{{url_for('species.populations.samples.list_samples', - species_id=species.SpeciesId, - population_id=population.Id, - from=offset+count, - count=count)}}"> - Next - <span class="glyphicon glyphicon-forward"></span> - </a> - {%endif%} - </div> + <div class="col-md-8" style="text-align: center;"> + Samples {{offset}} — {{offset+(count if offset + count < total_samples else total_samples - offset)}} / {{total_samples}} + </div> + + <div class="col-md-2"> + {%if offset + count < total_samples:%} + <a href="{{url_for('species.populations.samples.list_samples', + species_id=species.SpeciesId, + population_id=population.Id, + from=offset+count, + count=count)}}"> + Next + <span class="glyphicon glyphicon-forward"></span> + </a> + {%endif%} </div> +</div> +<div class="row"> <table class="table"> <thead> <tr> + <th></th> <th>Name</th> <th>Auxilliary Name</th> <th>Symbol</th> @@ -73,6 +84,7 @@ <tbody> {%for sample in samples%} <tr> + <td>{{sample.sequence_number}}</td> <td>{{sample.Name}}</td> <td>{{sample.Name2}}</td> <td>{{sample.Symbol or "-"}}</td> @@ -90,10 +102,14 @@ delete all samples </a> </p> - {%else%} - <p class="text-danger"> - <span class="glyphicon glyphicon-exclamation-sign"></span> - There are no samples for this population at this time. +</div> + +{%else%} + +<div class="row"> + <p> + There are no samples entered for this population. Do please go ahead and add + the samples for this population by clicking on the button below. </p> <p> @@ -106,8 +122,9 @@ add samples </a> </p> - {%endif%} </div> +{%endif%} + {%endblock%} {%block sidebarcontents%} diff --git a/uploader/templates/samples/select-population.html b/uploader/templates/samples/select-population.html index 8e22ac1..1cc7573 100644 --- a/uploader/templates/samples/select-population.html +++ b/uploader/templates/samples/select-population.html @@ -12,23 +12,15 @@ {{flash_all_messages()}} <div class="row"> - <p>Select the population to use with your samples:</p> {{select_population_form( - url_for("species.populations.samples.select_population", species_id=species.SpeciesId), - populations)}} -</div> -<div class="row"> - <p><strong>Cannot find your population in the list?</strong></p> - - <p>If you cannot find the population you want in the drop-down above, you can - instead, - <a href="{{url_for('species.populations.create_population', - species_id=species.SpeciesId)}}" - title="Create a new population for species '{{species.FullName}},"> - create a new population</a>. + url_for("species.populations.samples.select_population", species_id=species.SpeciesId), species, populations)}} </div> {%endblock%} {%block sidebarcontents%} {{display_species_card(species)}} {%endblock%} + +{%block javascript%} +<script type="text/javascript" src="/static/js/populations.js"></script> +{%endblock%} diff --git a/uploader/templates/samples/upload-failure.html b/uploader/templates/samples/upload-failure.html index 458ab55..2cf8053 100644 --- a/uploader/templates/samples/upload-failure.html +++ b/uploader/templates/samples/upload-failure.html @@ -15,7 +15,7 @@ <h3>Debugging Information</h3> <ul> - <li><strong>job id</strong>: {{job.job_id}}</li> + <li><strong>job id</strong>: {{job.jobid}}</li> <li><strong>status</strong>: {{job.status}}</li> <li><strong>job type</strong>: {{job["job-type"]}}</li> </ul> diff --git a/uploader/templates/samples/upload-samples.html b/uploader/templates/samples/upload-samples.html index b101b2e..25d3290 100644 --- a/uploader/templates/samples/upload-samples.html +++ b/uploader/templates/samples/upload-samples.html @@ -23,7 +23,12 @@ {{flash_all_messages()}} <div class="row"> - <p>You can now upload a character-separated value (CSV) file that contains + <p> + You can now upload the samples for the "{{population.FullName}}" population + from the "{{species.FullName}}" species here. + </p> + <p> + Upload a <strong>character-separated value (CSV)</strong> file that contains details about your samples. The CSV file should have the following fields: <dl> <dt>Name</dt> diff --git a/uploader/templates/select_species.html b/uploader/templates/select_species.html deleted file mode 100644 index 1642401..0000000 --- a/uploader/templates/select_species.html +++ /dev/null @@ -1,92 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} -{%from "upload_progress_indicator.html" import upload_progress_indicator%} - -{%block title%}expression data: select species{%endblock%} - -{%block contents%} -{{upload_progress_indicator()}} - -<h2 class="heading">expression data: select species</h2> - -<div class="row"> - <form action="{{url_for('expression-data.index.upload_file')}}" - method="POST" - enctype="multipart/form-data" - id="frm-upload-expression-data"> - <legend class="heading">upload expression data</legend> - {{flash_messages("error-expr-data")}} - - <div class="form-group"> - <label for="select_species01" class="form-label">Species</label> - <select id="select_species01" - name="speciesid" - required="required" - class="form-control"> - <option value="">Select species</option> - {%for aspecies in species%} - <option value="{{aspecies.SpeciesId}}">{{aspecies.MenuName}}</option> - {%endfor%} - </select> - </div> - - <div class="form-group"> - <legend class="heading">file type</legend> - - <div class="form-check"> - <input type="radio" name="filetype" value="average" id="filetype_average" - required="required" class="form-check-input" /> - <label for="filetype_average" class="form-check-label">average</label> - </div> - - <div class="form-check"> - <input type="radio" name="filetype" value="standard-error" - id="filetype_standard_error" required="required" - class="form-check-input" /> - <label for="filetype_standard_error" class="form-check-label"> - standard error - </label> - </div> - </div> - - <div class="form-group"> - <span id="no-file-error" class="alert-danger" style="display: none;"> - No file selected - </span> - <label for="file_upload" class="form-label">select file</label> - <input type="file" name="qc_text_file" id="file_upload" - accept="text/plain, text/tab-separated-values, application/zip" - class="form-control"/> - </div> - - <button type="submit" - class="btn btn-primary" - data-toggle="modal" - data-target="#upload-progress-indicator">upload file</button> - </form> -</div> -{%endblock%} - - -{%block javascript%} -<script type="text/javascript" src="static/js/upload_progress.js"></script> -<script type="text/javascript"> - function setup_formdata(form) { - var formdata = new FormData(); - formdata.append( - "speciesid", - form.querySelector("#select_species01").value) - formdata.append( - "qc_text_file", - form.querySelector("input[type='file']").files[0]); - formdata.append( - "filetype", - selected_filetype( - Array.from(form.querySelectorAll("input[type='radio']")))); - return formdata; - } - - setup_upload_handlers( - "frm-upload-expression-data", make_data_uploader(setup_formdata)); -</script> -{%endblock%} diff --git a/uploader/templates/species/base.html b/uploader/templates/species/base.html index 04391db..f64f72b 100644 --- a/uploader/templates/species/base.html +++ b/uploader/templates/species/base.html @@ -6,7 +6,12 @@ {%else%} class="breadcrumb-item" {%endif%}> + {%if species is mapping%} + <a href="{{url_for('species.view_species', species_id=species.SpeciesId)}}"> + {{species.Name}}</a> + {%else%} <a href="{{url_for('species.list_species')}}">Species</a> + {%endif%} </li> {%block lvl2_breadcrumbs%}{%endblock%} {%endblock%} diff --git a/uploader/templates/species/create-species.html b/uploader/templates/species/create-species.html index 0d0bedf..138dbaa 100644 --- a/uploader/templates/species/create-species.html +++ b/uploader/templates/species/create-species.html @@ -19,72 +19,88 @@ <div class="row"> <form id="frm-create-species" method="POST" - action="{{url_for('species.create_species')}}"> + action="{{url_for('species.create_species', return_to=return_to)}}" + class="form-horizontal"> <legend>Create Species</legend> {{flash_all_messages()}} + <input type="hidden" name="return_to" value="{{return_to}}"> + <div class="form-group"> - <label for="txt-taxonomy-id" class="form-label"> + <label for="txt-taxonomy-id" class="control-label col-sm-2"> Taxonomy ID</label> - <div class="input-group"> - <input id="txt-taxonomy-id" - name="species_taxonomy_id" - type="text" - class="form-control" /> - <span class="input-group-btn"> - <button id="btn-search-taxonid" class="btn btn-info">Search</button> - </span> + <div class="col-sm-10"> + <div class="input-group"> + <input id="txt-taxonomy-id" + name="species_taxonomy_id" + type="text" + class="form-control" /> + <span class="input-group-btn"> + <button id="btn-search-taxonid" class="btn btn-info">Search</button> + </span> + </div> + <small class="form-text text-small text-muted"> + Use + <a href="https://www.ncbi.nlm.nih.gov/Taxonomy/taxonomyhome.html/" + title="NCBI's Taxonomy Browser homepage" + target="_blank"> + NCBI's Taxonomy Browser homepage</a> to search for the species you + want. If the species exists on NCBI, they will have a Taxonomy ID. Copy + that Taxonomy ID to this field, and click "Search" to auto-fill the + details.<br /> + This field is optional.</small> </div> - <small class="form-text text-small text-muted">Provide the taxonomy ID for - your species that can be used to link to external sites like NCBI. Enter - the taxonomy ID and click "Search" to auto-fill the form with data. - <br /> - While it is recommended to provide a value for this field, doing so is - optional. - </small> </div> <div class="form-group"> - <label for="txt-species-name" class="form-label">Common Name</label> - <input id="txt-species-name" - name="common_name" - type="text" - class="form-control" - required="required" /> - <small class="form-text text-muted">Provide the common, possibly - non-scientific name for the species here, e.g. Human, Mouse, etc.</small> + <label for="txt-species-name" class="control-label col-sm-2">Common Name</label> + <div class="col-sm-10"> + <input id="txt-species-name" + name="common_name" + type="text" + class="form-control" + required="required" /> + <small class="form-text text-muted">This is the day-to-day term used by + laymen, e.g. Mouse (instead of Mus musculus), round worm (instead of + Ascaris lumbricoides), etc.<br /> + For species without this, just enter the scientific name. + </small> + </div> </div> <div class="form-group"> - <label for="txt-species-scientific" class="form-label"> + <label for="txt-species-scientific" class="control-label col-sm-2"> Scientific Name</label> - <input id="txt-species-scientific" - name="scientific_name" - type="text" - class="form-control" - required="required" /> - <small class="form-text text-muted">Provide the scientific name for the - species you are creating, e.g. Homo sapiens, Mus musculus, etc.</small> + <div class="col-sm-10"> + <input id="txt-species-scientific" + name="scientific_name" + type="text" + class="form-control" + required="required" /> + <small class="form-text text-muted">This is the scientific name for the + species e.g. Homo sapiens, Mus musculus, etc.</small> + </div> </div> <div class="form-group"> - <label for="select-species-family" class="form-label">Family</label> - <select id="select-species-family" - name="species_family" - required="required" - class="form-control"> - <option value="">Please select a grouping</option> - {%for family in families%} - <option value="{{family}}">{{family}}</option> - {%endfor%} - </select> - <small class="form-text text-muted"> - This is a generic grouping for the species that determines under which - grouping the species appears in the GeneNetwork menus</small> + <label for="select-species-family" class="control-label col-sm-2">Family</label> + <div class="col-sm-10"> + <select id="select-species-family" + name="species_family" + required="required" + class="form-control"> + <option value="ungrouped">I do not know what to pick</option> + {%for family in families%} + <option value="{{family}}">{{family}}</option> + {%endfor%} + </select> + <small class="form-text text-muted"> + This is a rough grouping of the species.</small> + </div> </div> - <div class="form-group"> + <div class="col-sm-offset-2 col-sm-10"> <input type="submit" value="create new species" class="btn btn-primary" /> @@ -113,7 +129,7 @@ } msg = ( "Request to '${uri}' failed with message '${textStatus}'. " - + "Please try again later, or fill the details manually."); + + "Please try again later, or fill the details manually."); alert(msg); console.error(msg, data, textStatus); return false; diff --git a/uploader/templates/species/edit-species.html b/uploader/templates/species/edit-species.html index 6827751..5a26455 100644 --- a/uploader/templates/species/edit-species.html +++ b/uploader/templates/species/edit-species.html @@ -107,6 +107,7 @@ id="txt-species-familyorderid" name="species_familyorderid" value="{{species.FamilyOrderId}}" + required="required" class="form-control" /> <small class="form-text text-muted"> This is a number that determines the order of the "Family" groupings @@ -121,7 +122,7 @@ <input type="number" id="txt-species-orderid" name="species_orderid" - value="{{species.OrderId or max_order_id}}" + value="{{species.OrderId or (max_order_id + 5)}}" class="form-control" /> <small class="form-text text-muted"> This integer value determines the order of the species in relation to diff --git a/uploader/templates/species/list-species.html b/uploader/templates/species/list-species.html index 573bcee..64084b0 100644 --- a/uploader/templates/species/list-species.html +++ b/uploader/templates/species/list-species.html @@ -29,29 +29,35 @@ <caption>Available Species</caption> <thead> <tr> - <th>Common Name</th> - <th>Scientific Name</th> - <th>TaxonId</th> - <th>Use</th> + <th></td> + <th title="A common, layman's name for the species.">Common Name</th> + <th title="The scientific name for the species">Organism Name</th> + <th title="An identifier for the species in the NCBI taxonomy database"> + Taxonomy ID + </th> + <th title="A generic grouping used internally by GeneNetwork for organising species."> + Family + </th> </tr> </thead> <tbody> {%for species in allspecies%} <tr> + <td>{{species["sequence_number"]}}</td> <td>{{species["SpeciesName"]}}</td> - <td>{{species["FullName"]}}</td> - <td> - <a href="https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?id={{species['TaxonomyId']}}" - title="View species details on NCBI" - target="_blank">{{species["TaxonomyId"]}}</a> - </td> <td> <a href="{{url_for('species.view_species', species_id=species['SpeciesId'])}}" - title=""> - {{species["SpeciesName"]}} ({{species["FullName"]}}) + title="View details in GeneNetwork on {{species['FullName']}}"> + {{species["FullName"]}} </a> </td> + <td> + <a href="https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?id={{species['TaxonomyId']}}" + title="View species details on NCBI" + target="_blank">{{species["TaxonomyId"]}}</a> + </td> + <td>{{species.Family}}</td> </tr> {%else%} <tr> diff --git a/uploader/templates/species/macro-display-species-card.html b/uploader/templates/species/macro-display-species-card.html index 857c0f0..166c7b9 100644 --- a/uploader/templates/species/macro-display-species-card.html +++ b/uploader/templates/species/macro-display-species-card.html @@ -3,13 +3,19 @@ <div class="card-body"> <h5 class="card-title">Species</h5> <div class="card-text"> - <dl> - <dt>Common Name</dt> - <dd>{{species.SpeciesName}}</dd> + <table class="table"> + <tbody> + <tr> + <td>Common Name</td> + <td>{{species.SpeciesName}}</td> + </tr> - <dt>Scientific Name</dt> - <dd>{{species.FullName}}</dd> - </dl> + <tr> + <td>Scientific Name</td> + <td>{{species.FullName}}</td> + </tr> + </tbody> + </table> </div> </div> </div> diff --git a/uploader/templates/species/macro-select-species.html b/uploader/templates/species/macro-select-species.html index 3dbfc95..3714ae4 100644 --- a/uploader/templates/species/macro-select-species.html +++ b/uploader/templates/species/macro-select-species.html @@ -1,38 +1,59 @@ +{%from "macro-step-indicator.html" import step_indicator%} + {%macro select_species_form(form_action, species)%} -{%if species | length > 0%} -<form method="GET" action="{{form_action}}"> - <legend>Select Species</legend> - - <div class="form-group"> - <label for="select-species" class="form-label">Select Species</label> - <select id="select-species" - name="species_id" - class="form-control" - required="required"> - <option value="">Select Species</option> - {%for group in species%} - {{group}} - <optgroup {%if group[0][1] is not none%} - label="{{group[0][1].capitalize()}}" - {%else%} - label="Undefined" - {%endif%}> - {%for aspecies in group[1]%} - <option value="{{aspecies.SpeciesId}}">{{aspecies.MenuName}}</option> - {%endfor%} - </optgroup> - {%endfor%} - </select> +<form method="GET" action="{{form_action}}" class="form-horizontal"> + + <h2>{{step_indicator("1")}} What species do you want to work with?</h2> + + {%if species | length != 0%} + + <p class="form-text">Search for, and select the species from the table below + and click "Continue"</p> + + <div class="radio"> + <label for="rdo-cant-find-species" + style="font-weight: 1;"> + <input id="rdo-cant-find-species" type="radio" name="species_id" + value="CREATE-SPECIES" /> + I could not find the species I want (create it). + </label> </div> - <div class="form-group"> - <input type="submit" value="Select" class="btn btn-primary" /> + <div class="col-sm-offset-10 col-sm-2"> + <input type="submit" + class="btn btn-primary" + value="continue" /> </div> + + <div style="margin-top:3em;"> + <table id="tbl-select-species" class="table compact stripe" + data-species-list='{{species | tojson}}'> + <div class=""> + <thead> + <tr> + <th></th> + <th>Species Name</th> + </tr> + </thead> + + <tbody></tbody> + </table> + </div> + + {%else%} + + <label class="control-label" for="rdo-cant-find-species"> + <input id="rdo-cant-find-species" type="radio" name="species_id" + value="CREATE-SPECIES" /> + There are no species to select from. Create the first one.</label> + + <div class="col-sm-offset-10 col-sm-2"> + <input type="submit" + class="btn btn-primary col-sm-offset-1" + value="continue" /> + </div> + + {%endif%} + </form> -{%else%} -<p class="text-danger"> - <span class="glyphicon glyphicon-exclamation-mark"></span> - We could not find species to select from! -</p> -{%endif%} {%endmacro%} diff --git a/uploader/templates/species/view-species.html b/uploader/templates/species/view-species.html index 6942168..2d02f7e 100644 --- a/uploader/templates/species/view-species.html +++ b/uploader/templates/species/view-species.html @@ -40,16 +40,17 @@ <ol> <li> - <a href="#" - title="Upload genotypes for {{species.FullName}}">Upload Genotypes</a> - </li> - <li> <a href="{{url_for('species.populations.list_species_populations', species_id=species.SpeciesId)}}" title="Create/Edit populations for {{species.FullName}}"> Manage populations</a> </li> - <li><a href="#" title="">any other action, perhaps …</a></li> + <li> + <a href="{{url_for('species.platforms.list_platforms', + species_id=species.SpeciesId)}}" + title="Create/Edit sequencing platforms for {{species.FullName}}"> + Manage sequencing platforms</a> + </li> </ol> diff --git a/uploader/ui.py b/uploader/ui.py index 4115b02..1994056 100644 --- a/uploader/ui.py +++ b/uploader/ui.py @@ -8,6 +8,7 @@ def make_template_renderer(default): template, **{ **kwargs, + "activemenu": default, "activelink": kwargs.get("activelink", default) }) return render_template |