diff options
112 files changed, 4547 insertions, 1502 deletions
diff --git a/docs/dev/background_jobs.md b/docs/dev/background_jobs.md new file mode 100644 index 0000000..1a41636 --- /dev/null +++ b/docs/dev/background_jobs.md @@ -0,0 +1,62 @@ +# Background Jobs + +We run background jobs for long-running processes, e.g. quality-assurance checks +across multiple huge files, inserting huge data to databases, etc. The system +needs to keep track of the progress of these jobs and communicate the state to +the user whenever the user requests. + +This details some thoughts on how to handle these jobs, especially in failure +conditions. + +We currently use Redis[^redis] to keep track of the state of the background +processes. + +Every background job started will have a Redis[^redis] key with the prefix `gn-uploader:jobs` + +## Users + +Currently (2024-10-23T13:29UTC-05:00), we do not track the user that started the job. Moving forward, we will track this information. + +We could have the keys be something like, `gn-uploader:jobs:<user-id>:<job-id>`. + +Another option is track any particular users jobs with a key of the form +`gn-uploader:users:<user-id>:jobs` and in that case, have the job keys take the +form `gn-uploader:jobs:<job-id>`. I (@fredmanglis) favour this option over +having the user's ID in the jobs keys directly, since it provides a way to +interact with **ALL** the jobs without indirecting through each specific user. +This is a useful ability to have, especially for system administrative tasks. + +## Multiprocessing Within Jobs + +Some jobs, e.g. quality-assurance jobs, can run multiple threads/processes +themselves. This brings up a problem because Redis[^redis] does not allow +parallel access to a key, especially for writing. + +We also do not want to create bottlenecks by writing to the same key from +multiple threads/processes. + +The design I have currently come up with, that might work is as follows: + +- At any point just before where multiple threads/processes are started, a list + of new keys, each of which will collect the output from a single thread, will + be built. +- These keys are recorded in the parent's redis key data +- The threads/processes are started and do whatever they need, pushing their + outputs to the appropriate keys within redis. + +The new keys for the children threads/processe could build on the theme + + +## Fetching Jobs Status + +Different jobs could have different ways of requirements for handling/processing +their outputs, and those of any children they might spawn. The system will need +to provide a way to pass in the correct function/code to process the outputs at +the point where the job status is requested. + +This implies that we need to track the type of job in order to be able to select +the correct code for processing such output. + +## Links + +- [^redis]: https://redis.io/ diff --git a/docs/dev/quality_assurance_on_csv_files.md b/docs/dev/quality_assurance_on_csv_files.md new file mode 100644 index 0000000..02d63c9 --- /dev/null +++ b/docs/dev/quality_assurance_on_csv_files.md @@ -0,0 +1,52 @@ +# Quality Assurance/Control on CSV Files + +## Abbreviations + +- CSV files: Character-separated-values files — these are data files structured in a table-like format, with a specific character chosen as the column/field separator. The comma (,) is the most common field separator used by most csv files. It is, however, possible to encounter files with other characters separating the values. + +## General Pattern + +A general pattern has emerged when performing quality assurance on the data in +CSV files — the pseudocode below shows the general pattern: + +```python +def qc_function(filepath, …): + open(filepath, …) + + headers = read_first_line(…) + perform_qc_on_headings(headers, …) + + for each subsequent line in file: + perform_qc_on_first_column(line, …) + + for each subsequent field in line: + perform_qc_on_field(field, …) +``` + +We want to list the errors found in each file, so it makes sense for the `perform_qc_on*` functions in the pseudocode above to return the list of errors found for each file. + +The actual quality assurance done on the headers, first column of data rows, and the fields can differ from one type of file to the next, but the structure remains relatively unchanged. + +This implies we could make use of a higher-order function that contains the general structure with the actual qc steps passed in as functions that are called in the higher-order structuring function. This gives something like: + +```python +def qc_function(filepath, headers_qc, first_column_qc, data_qc, …): + for line in file: + if line is a comment line: + skip line and continue iteration + if line is first non-comment line: + line is the header line + call headers_qc on fields in this line + if line is not first non-comment line: + line is data line + call first_column_qc on first field of line + call data_qc on each of the subsequent fields of the line + + collect and return errors +``` + +## Improvements + +- Read the file in a separate generator function +- Parallelize QC if many files are present +- Add logging/output for user update (how do we do this correctly?) diff --git a/r_qtl/r_qtl2.py b/r_qtl/r_qtl2.py index 9da4081..c6307f5 100644 --- a/r_qtl/r_qtl2.py +++ b/r_qtl/r_qtl2.py @@ -18,6 +18,10 @@ FILE_TYPES = ( "geno", "founder_geno", "pheno", "covar", "phenocovar", "gmap", "pmap", "phenose") +__CONTROL_FILE_ERROR_MESSAGE__ = ( + "The zipped bundle that was provided does not contain a valid control file " + "in either JSON or YAML format.") + def __special_file__(filename): """ @@ -72,6 +76,8 @@ def transpose_csv( def __read_by_line__(_path): with open(_path, "r", encoding="utf8") as infile: for line in infile: + if line.startswith("#"): + continue yield line transposed_data= (f"{linejoinerfn(items)}\n" for items in zip(*( @@ -112,7 +118,7 @@ def __control_data_from_zipfile__(zfile: ZipFile) -> dict: or filename.endswith(".json")))) num_files = len(files) if num_files == 0: - raise InvalidFormat("Expected a json or yaml control file.") + raise InvalidFormat(__CONTROL_FILE_ERROR_MESSAGE__) if num_files > 1: raise InvalidFormat("Found more than one possible control file.") @@ -129,7 +135,6 @@ def __control_data_from_zipfile__(zfile: ZipFile) -> dict: else yaml.safe_load(zfile.read(files[0]))) } - def __control_data_from_dirpath__(dirpath: Path): """Load control data from a given directory path.""" files = tuple(path for path in dirpath.iterdir() @@ -137,7 +142,7 @@ def __control_data_from_dirpath__(dirpath: Path): and (path.suffix in (".yaml", ".json")))) num_files = len(files) if num_files == 0: - raise InvalidFormat("Expected a json or yaml control file.") + raise InvalidFormat(__CONTROL_FILE_ERROR_MESSAGE__) if num_files > 1: raise InvalidFormat("Found more than one possible control file.") @@ -200,8 +205,8 @@ def control_data(control_src: Union[Path, ZipFile]) -> dict: if control_src.is_dir(): return __cleanup__(__control_data_from_dirpath__(control_src)) raise InvalidFormat( - "Expects either a zipfile.ZipFile object or a pathlib.Path object " - "pointing to a directory containing the R/qtl2 bundle.") + "Expects either a zipped bundle of files or a path-like object " + "pointing to the zipped R/qtl2 bundle.") def replace_na_strings(cdata, val): @@ -549,3 +554,21 @@ def load_samples(zipfilepath: Union[str, Path], pass return tuple(samples) + + + +def read_text_file(filepath: Union[str, Path]) -> Iterator[str]: + """Read the raw text from a text file.""" + with open(filepath, "r", encoding="utf8") as _file: + for line in _file: + yield line + + +def read_csv_file(filepath: Union[str, Path], + separator: str = ",", + comment_char: str = "#") -> Iterator[tuple[str, ...]]: + """Read a file as a csv file.""" + for line in read_text_file(filepath): + if line.startswith(comment_char): + continue + yield tuple(field.strip() for field in line.split(separator)) diff --git a/r_qtl/r_qtl2_qc.py b/r_qtl/r_qtl2_qc.py index 7b26b50..2d9e9a8 100644 --- a/r_qtl/r_qtl2_qc.py +++ b/r_qtl/r_qtl2_qc.py @@ -95,7 +95,7 @@ def missing_files(bundlesrc: Union[Path, ZipFile]) -> tuple[tuple[str, str], ... "pointing to a directory containing the R/qtl2 bundle.") -def validate_bundle(zfile: ZipFile): +def validate_bundle(zfile: Union[Path, ZipFile]): """Ensure the R/qtl2 bundle is valid.""" missing = missing_files(zfile) if len(missing) > 0: diff --git a/scripts/cli_parser.py b/scripts/cli_parser.py index 308ee4b..d42ae66 100644 --- a/scripts/cli_parser.py +++ b/scripts/cli_parser.py @@ -19,6 +19,12 @@ def init_cli_parser(program: str, description: Optional[str] = None) -> Argument type=int, default=86400, help="How long to keep any redis keys around.") + parser.add_argument( + "--loglevel", + type=str, + default="INFO", + choices=["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"], + help="The severity of events to track with the logger.") return parser def add_global_data_arguments(parser: ArgumentParser) -> ArgumentParser: diff --git a/scripts/process_rqtl2_bundle.py b/scripts/process_rqtl2_bundle.py index 20cfd3b..ade9862 100644 --- a/scripts/process_rqtl2_bundle.py +++ b/scripts/process_rqtl2_bundle.py @@ -2,6 +2,7 @@ import sys import uuid import json +import argparse import traceback from typing import Any from pathlib import Path @@ -94,10 +95,11 @@ def process_bundle(dbconn: mdb.Connection, logger.info("Processing geno files.") genoexit = install_genotypes( dbconn, - meta["speciesid"], - meta["populationid"], - meta["geno-dataset-id"], - Path(meta["rqtl2-bundle-file"]), + argparse.Namespace( + speciesid=meta["speciesid"], + populationid=meta["populationid"], + datasetid=meta["geno-dataset-id"], + rqtl2bundle=Path(meta["rqtl2-bundle-file"])), logger) if genoexit != 0: raise Exception("Processing 'geno' file failed.") @@ -109,10 +111,11 @@ def process_bundle(dbconn: mdb.Connection, if has_pheno_file(thejob): phenoexit = install_pheno_files( dbconn, - meta["speciesid"], - meta["platformid"], - meta["probe-dataset-id"], - Path(meta["rqtl2-bundle-file"]), + argparse.Namespace( + speciesid=meta["speciesid"], + platformid=meta["platformid"], + dataset_id=meta["probe-dataset-id"], + rqtl2bundle=Path(meta["rqtl2-bundle-file"])), logger) if phenoexit != 0: raise Exception("Processing 'pheno' file failed.") diff --git a/scripts/qc_on_rqtl2_bundle2.py b/scripts/qc_on_rqtl2_bundle2.py new file mode 100644 index 0000000..7e5d253 --- /dev/null +++ b/scripts/qc_on_rqtl2_bundle2.py @@ -0,0 +1,346 @@ +"""Run Quality Control checks on R/qtl2 bundle.""" +import os +import sys +import json +from time import sleep +from pathlib import Path +from zipfile import ZipFile +from argparse import Namespace +from datetime import timedelta +import multiprocessing as mproc +from functools import reduce, partial +from logging import Logger, getLogger, StreamHandler +from typing import Union, Sequence, Callable, Iterator + +import MySQLdb as mdb +from redis import Redis + +from quality_control.errors import InvalidValue +from quality_control.checks import decimal_points_error + +from uploader import jobs +from uploader.db_utils import database_connection +from uploader.check_connections import check_db, check_redis + +from r_qtl import r_qtl2 as rqtl2 +from r_qtl import r_qtl2_qc as rqc +from r_qtl import exceptions as rqe +from r_qtl import fileerrors as rqfe + +from scripts.process_rqtl2_bundle import parse_job +from scripts.redis_logger import setup_redis_logger +from scripts.cli_parser import init_cli_parser, add_global_data_arguments +from scripts.rqtl2.bundleutils import build_line_joiner, build_line_splitter + + +def check_for_missing_files( + rconn: Redis, fqjobid: str, extractpath: Path, logger: Logger) -> bool: + """Check that all files listed in the control file do actually exist.""" + logger.info("Checking for missing files.") + missing = rqc.missing_files(extractpath) + # add_to_errors(rconn, fqjobid, "errors-generic", tuple( + # rqfe.MissingFile( + # mfile[0], mfile[1], ( + # f"File '{mfile[1]}' is listed in the control file under " + # f"the '{mfile[0]}' key, but it does not actually exist in " + # "the bundle.")) + # for mfile in missing)) + if len(missing) > 0: + logger.error(f"Missing files in the bundle!") + return True + return False + + +def open_file(file_: Path) -> Iterator: + """Open file and return one line at a time.""" + with open(file_, "r", encoding="utf8") as infile: + for line in infile: + yield line + + +def check_markers( + filename: str, + row: tuple[str, ...], + save_error: lambda val: val +) -> tuple[rqfe.InvalidValue]: + """Check that the markers are okay""" + errors = tuple() + counts = {} + for marker in row: + counts = {**counts, marker: counts.get(marker, 0) + 1} + if marker is None or marker == "": + errors = errors + (save_error(rqfe.InvalidValue( + filename, + "markers" + "-", + marker, + "A marker MUST be a valid value.")),) + + return errors + tuple( + save_error(rqfe.InvalidValue( + filename, + "markers", + key, + f"Marker '{key}' was repeated {value} times")) + for key,value in counts.items() if value > 1) + + +def check_geno_line( + filename: str, + headers: tuple[str, ...], + row: tuple[Union[str, None]], + cdata: dict, + save_error: lambda val: val +) -> tuple[rqfe.InvalidValue]: + """Check that the geno line is correct.""" + errors = tuple() + # Verify that line has same number of columns as headers + if len(headers) != len(row): + errors = errors + (save_error(rqfe.InvalidValue( + filename, + headers[0], + row[0], + row[0], + "Every line MUST have the same number of columns.")),) + + # First column is the individuals/cases/samples + if not bool(row[0]): + errors = errors + (save_error(rqfe.InvalidValue( + filename, + headers[0], + row[0], + row[0], + "The sample/case MUST be a valid value.")),) + + def __process_value__(val): + if val in cdata["na.strings"]: + return None + if val in cdata["alleles"]: + return cdata["genotypes"][val] + + genocode = cdata.get("genotypes", {}) + for coltitle, cellvalue in zip(headers[1:],row[1:]): + if ( + bool(genocode) and + cellvalue is not None and + cellvalue not in genocode.keys() + ): + errors = errors + (save_error(rqfe.InvalidValue( + filename, row[0], coltitle, cellvalue, + f"Value '{cellvalue}' is invalid. Expected one of " + f"'{', '.join(genocode.keys())}'.")),) + + return errors + + +def push_file_error_to_redis(rconn: Redis, key: str, error: InvalidValue) -> InvalidValue: + """Push the file error to redis a json string + + Parameters + ---------- + rconn: Connection to redis + key: The name of the list where we push the errors + error: The file error to save + + Returns + ------- + Returns the file error it saved + """ + if bool(error): + rconn.rpush(key, json.dumps(error._asdict())) + return error + + +def file_errors_and_details( + redisargs: dict[str, str], + file_: Path, + filetype: str, + cdata: dict, + linesplitterfn: Callable, + linejoinerfn: Callable, + headercheckers: tuple[Callable, ...], + bodycheckers: tuple[Callable, ...] +) -> dict: + """Compute errors, and other file metadata.""" + errors = tuple() + if cdata[f"{filetype}_transposed"]: + rqtl2.transpose_csv_with_rename(file_, linesplitterfn, linejoinerfn) + + with Redis.from_url(redisargs["redisuri"], decode_responses=True) as rconn: + save_error_fn = partial(push_file_error_to_redis, + rconn, + error_list_name(filetype, file_.name)) + for lineno, line in enumerate(open_file(file_), start=1): + row = linesplitterfn(line) + if lineno == 1: + headers = tuple(row) + errors = errors + reduce( + lambda errs, fnct: errs + fnct( + file_.name, row[1:], save_error_fn), + headercheckers, + tuple()) + continue + + errors = errors + reduce( + lambda errs, fnct: errs + fnct( + file_.name, headers, row, cdata, save_error_fn), + bodycheckers, + tuple()) + + filedetails = { + "filename": file_.name, + "filesize": os.stat(file_).st_size, + "linecount": lineno + } + rconn.hset(redisargs["fqjobid"], + f"file-details:{filetype}:{file_.name}", + json.dumps(filedetails)) + return {**filedetails, "errors": errors} + + +def error_list_name(filetype: str, filename: str): + """Compute the name of the list where the errors will be pushed. + + Parameters + ---------- + filetype: The type of file. One of `r_qtl.r_qtl2.FILE_TYPES` + filename: The name of the file. + """ + return f"errors:{filetype}:{filename}" + + +def check_for_geno_errors( + redisargs: dict[str, str], + extractdir: Path, + cdata: dict, + linesplitterfn: Callable[[str], tuple[Union[str, None]]], + linejoinerfn: Callable[[tuple[Union[str, None], ...]], str], + logger: Logger +) -> bool: + """Check for errors in genotype files.""" + if "geno" in cdata or "founder_geno" in cdata: + genofiles = tuple( + extractdir.joinpath(fname) for fname in cdata.get("geno", [])) + fgenofiles = tuple( + extractdir.joinpath(fname) for fname in cdata.get("founder_geno", [])) + allgenofiles = genofiles + fgenofiles + with Redis.from_url(redisargs["redisuri"], decode_responses=True) as rconn: + error_list_names = [ + error_list_name("geno", file_.name) for file_ in allgenofiles] + for list_name in error_list_names: + rconn.delete(list_name) + rconn.hset( + redisargs["fqjobid"], + "geno-errors-lists", + json.dumps(error_list_names)) + processes = [ + mproc.Process(target=file_errors_and_details, + args=( + redisargs, + file_, + ftype, + cdata, + linesplitterfn, + linejoinerfn, + (check_markers,), + (check_geno_line,)) + ) + for ftype, file_ in ( + tuple(("geno", file_) for file_ in genofiles) + + tuple(("founder_geno", file_) for file_ in fgenofiles)) + ] + for process in processes: + process.start() + # Set expiry for any created error lists + for key in error_list_names: + rconn.expire(name=key, + time=timedelta(seconds=redisargs["redisexpiry"])) + + # TOD0: Add the errors to redis + if any(rconn.llen(errlst) > 0 for errlst in error_list_names): + logger.error("At least one of the 'geno' files has (an) error(s).") + return True + logger.info("No error(s) found in any of the 'geno' files.") + + else: + logger.info("No 'geno' files to check.") + + return False + + +# def check_for_pheno_errors(...): +# """Check for errors in phenotype files.""" +# pass + + +# def check_for_phenose_errors(...): +# """Check for errors in phenotype, standard-error files.""" +# pass + + +# def check_for_phenocovar_errors(...): +# """Check for errors in phenotype-covariates files.""" +# pass + + +def run_qc(rconn: Redis, args: Namespace, fqjobid: str, logger: Logger) -> int: + """Run quality control checks on R/qtl2 bundles.""" + thejob = parse_job(rconn, args.redisprefix, args.jobid) + print(f"THE JOB =================> {thejob}") + jobmeta = thejob["job-metadata"] + inpath = Path(jobmeta["rqtl2-bundle-file"]) + extractdir = inpath.parent.joinpath(f"{inpath.name}__extraction_dir") + with ZipFile(inpath, "r") as zfile: + rqtl2.extract(zfile, extractdir) + + ### BEGIN: The quality control checks ### + cdata = rqtl2.control_data(extractdir) + splitter = build_line_splitter(cdata) + joiner = build_line_joiner(cdata) + + redisargs = { + "fqjobid": fqjobid, + "redisuri": args.redisuri, + "redisexpiry": args.redisexpiry + } + check_for_missing_files(rconn, fqjobid, extractdir, logger) + # check_for_pheno_errors(...) + check_for_geno_errors(redisargs, extractdir, cdata, splitter, joiner, logger) + # check_for_phenose_errors(...) + # check_for_phenocovar_errors(...) + ### END: The quality control checks ### + + def __fetch_errors__(rkey: str) -> tuple: + return tuple(json.loads(rconn.hget(fqjobid, rkey) or "[]")) + + return (1 if any(( + bool(__fetch_errors__(key)) + for key in + ("errors-geno", "errors-pheno", "errors-phenos", "errors-phenocovar"))) + else 0) + + +if __name__ == "__main__": + def main(): + """Enter R/qtl2 bundle QC runner.""" + args = add_global_data_arguments(init_cli_parser( + "qc-on-rqtl2-bundle", "Run QC on R/qtl2 bundle.")).parse_args() + check_redis(args.redisuri) + check_db(args.databaseuri) + + logger = getLogger("qc-on-rqtl2-bundle") + logger.addHandler(StreamHandler(stream=sys.stderr)) + logger.setLevel("DEBUG") + + fqjobid = jobs.job_key(args.redisprefix, args.jobid) + with Redis.from_url(args.redisuri, decode_responses=True) as rconn: + logger.addHandler(setup_redis_logger( + rconn, fqjobid, f"{fqjobid}:log-messages", + args.redisexpiry)) + + exitcode = run_qc(rconn, args, fqjobid, logger) + rconn.hset( + jobs.job_key(args.redisprefix, args.jobid), "exitcode", exitcode) + return exitcode + + sys.exit(main()) diff --git a/scripts/redis_logger.py b/scripts/redis_logger.py index 2ae682b..d3fde5f 100644 --- a/scripts/redis_logger.py +++ b/scripts/redis_logger.py @@ -1,5 +1,6 @@ """Utilities to log to redis for our worker scripts.""" import logging +from typing import Optional from redis import Redis @@ -26,6 +27,26 @@ class RedisLogger(logging.Handler): self.redisconnection.rpush(self.messageslistname, self.format(record)) self.redisconnection.expire(self.messageslistname, self.expiry) +class RedisMessageListHandler(logging.Handler): + """Send messages to specified redis list.""" + def __init__(self, + rconn: Redis, + fullyqualifiedkey: str, + loglevel: int = logging.NOTSET, + expiry: Optional[int] = 86400): + super().__init__(loglevel) + self.redisconnection = rconn + self.fullyqualifiedkey = fullyqualifiedkey + self.expiry = expiry + + def emit(self, record): + """Log out to specified `fullyqualifiedkey`.""" + self.redisconnection.rpush(self.fullyqualifiedkey, self.format(record)) + if bool(self.expiry): + self.redisconnection.expire(self.fullyqualifiedkey, self.expiry) + else: + self.redisconnection.persist(self.fullyqualifiedkey) + def setup_redis_logger(rconn: Redis, fullyqualifiedjobid: str, job_messagelist: str, diff --git a/scripts/rqtl2/bundleutils.py b/scripts/rqtl2/bundleutils.py new file mode 100644 index 0000000..17faa7c --- /dev/null +++ b/scripts/rqtl2/bundleutils.py @@ -0,0 +1,44 @@ +"""Common utilities to operate in R/qtl2 bundles.""" +from typing import Union, Callable + +def build_line_splitter(cdata: dict) -> Callable[[str], tuple[Union[str, None], ...]]: + """Build and return a function to use to split data in the files. + + Parameters + ---------- + cdata: A dict holding the control information included with the R/qtl2 + bundle. + + Returns + ------- + A function that takes a string and return a tuple of strings. + """ + separator = cdata["sep"] + na_strings = cdata["na.strings"] + def __splitter__(line: str) -> tuple[Union[str, None], ...]: + return tuple( + item if item not in na_strings else None + for item in + (field.strip() for field in line.strip().split(separator))) + return __splitter__ + + +def build_line_joiner(cdata: dict) -> Callable[[tuple[Union[str, None], ...]], str]: + """Build and return a function to use to split data in the files. + + Parameters + ---------- + cdata: A dict holding the control information included with the R/qtl2 + bundle. + + Returns + ------- + A function that takes a string and return a tuple of strings. + """ + separator = cdata["sep"] + na_strings = cdata["na.strings"] + def __joiner__(row: tuple[Union[str, None], ...]) -> str: + return separator.join( + (na_strings[0] if item is None else item) + for item in row) + return __joiner__ diff --git a/scripts/rqtl2/cli_parser.py b/scripts/rqtl2/cli_parser.py index bcc7a4f..9bb60a3 100644 --- a/scripts/rqtl2/cli_parser.py +++ b/scripts/rqtl2/cli_parser.py @@ -2,12 +2,22 @@ from pathlib import Path from argparse import ArgumentParser -def add_common_arguments(parser: ArgumentParser) -> ArgumentParser: - """Add common arguments to the CLI parser.""" - parser.add_argument("datasetid", - type=int, - help="The dataset to which the data belongs.") +def add_bundle_argument(parser: ArgumentParser) -> ArgumentParser: + """Add the `rqtl2bundle` argument.""" parser.add_argument("rqtl2bundle", type=Path, help="Path to R/qtl2 bundle zip file.") return parser + + +def add_datasetid_argument(parser: ArgumentParser) -> ArgumentParser: + """Add the `datasetid` argument.""" + parser.add_argument("datasetid", + type=int, + help="The dataset to which the data belongs.") + return parser + + +def add_common_arguments(parser: ArgumentParser) -> ArgumentParser: + """Add common arguments to the CLI parser.""" + return add_bundle_argument(add_datasetid_argument(parser)) diff --git a/scripts/rqtl2/entry.py b/scripts/rqtl2/entry.py index b7fb68e..bc4cd9f 100644 --- a/scripts/rqtl2/entry.py +++ b/scripts/rqtl2/entry.py @@ -1,5 +1,5 @@ """Build common script-entry structure.""" -from logging import Logger +import logging from typing import Callable from argparse import Namespace @@ -12,12 +12,19 @@ from uploader.check_connections import check_db, check_redis from scripts.redis_logger import setup_redis_logger -def build_main(args: Namespace, - run_fn: Callable[[Connection, Namespace], int], - logger: Logger, - loglevel: str = "INFO") -> Callable[[],int]: +def build_main( + args: Namespace, + run_fn: Callable[[Connection, Namespace, logging.Logger], int], + loggername: str +) -> Callable[[],int]: """Build a function to be used as an entry-point for scripts.""" def main(): + logging.basicConfig( + format=( + "%(asctime)s - %(levelname)s %(name)s: " + "(%(pathname)s: %(lineno)d) %(message)s"), + level=args.loglevel) + logger = logging.getLogger(loggername) check_db(args.databaseuri) check_redis(args.redisuri) if not args.rqtl2bundle.exists(): @@ -26,13 +33,12 @@ def build_main(args: Namespace, with (Redis.from_url(args.redisuri, decode_responses=True) as rconn, database_connection(args.databaseuri) as dbconn): - fqjobid = jobs.job_key(jobs.jobsnamespace(), args.jobid) + fqjobid = jobs.job_key(args.redisprefix, args.jobid) logger.addHandler(setup_redis_logger( rconn, fqjobid, f"{fqjobid}:log-messages", args.redisexpiry)) - logger.setLevel(loglevel) - return run_fn(dbconn, args) + return run_fn(dbconn, args, logger) return main diff --git a/scripts/rqtl2/install_genotypes.py b/scripts/rqtl2/install_genotypes.py index 6b89142..20a19da 100644 --- a/scripts/rqtl2/install_genotypes.py +++ b/scripts/rqtl2/install_genotypes.py @@ -1,11 +1,11 @@ """Load genotypes from R/qtl2 bundle into the database.""" import sys +import argparse import traceback -from pathlib import Path from zipfile import ZipFile from functools import reduce from typing import Iterator, Optional -from logging import Logger, getLogger, StreamHandler +from logging import Logger, getLogger import MySQLdb as mdb from MySQLdb.cursors import DictCursor @@ -19,6 +19,8 @@ from scripts.rqtl2.entry import build_main from scripts.rqtl2.cli_parser import add_common_arguments from scripts.cli_parser import init_cli_parser, add_global_data_arguments +__MODULE__ = "scripts.rqtl2.install_genotypes" + def insert_markers( dbconn: mdb.Connection, speciesid: int, @@ -185,13 +187,12 @@ def cross_reference_genotypes( def install_genotypes(#pylint: disable=[too-many-arguments, too-many-locals] dbconn: mdb.Connection, - speciesid: int, - populationid: int, - datasetid: int, - rqtl2bundle: Path, + args: argparse.Namespace, logger: Logger = getLogger(__name__) ) -> int: """Load any existing genotypes into the database.""" + (speciesid, populationid, datasetid, rqtl2bundle) = ( + args.speciesid, args.populationid, args.datasetid, args.rqtl2bundle) count = 0 with ZipFile(str(rqtl2bundle.absolute()), "r") as zfile: try: @@ -253,15 +254,5 @@ if __name__ == "__main__": return parser.parse_args() - thelogger = getLogger("install_genotypes") - thelogger.addHandler(StreamHandler(stream=sys.stderr)) - main = build_main( - cli_args(), - lambda dbconn, args: install_genotypes(dbconn, - args.speciesid, - args.populationid, - args.datasetid, - args.rqtl2bundle), - thelogger, - "INFO") + main = build_main(cli_args(), install_genotypes, __MODULE__) sys.exit(main()) diff --git a/scripts/rqtl2/install_phenos.py b/scripts/rqtl2/install_phenos.py index b5cab8e..a6e9fb2 100644 --- a/scripts/rqtl2/install_phenos.py +++ b/scripts/rqtl2/install_phenos.py @@ -1,10 +1,10 @@ """Load pheno from R/qtl2 bundle into the database.""" import sys +import argparse import traceback -from pathlib import Path from zipfile import ZipFile from functools import reduce -from logging import Logger, getLogger, StreamHandler +from logging import Logger, getLogger import MySQLdb as mdb from MySQLdb.cursors import DictCursor @@ -18,6 +18,8 @@ from r_qtl import r_qtl2_qc as rqc from functional_tools import take +__MODULE__ = "scripts.rqtl2.install_phenos" + def insert_probesets(dbconn: mdb.Connection, platformid: int, phenos: tuple[str, ...]) -> int: @@ -95,12 +97,11 @@ def cross_reference_probeset_data(dbconn: mdb.Connection, def install_pheno_files(#pylint: disable=[too-many-arguments, too-many-locals] dbconn: mdb.Connection, - speciesid: int, - platformid: int, - datasetid: int, - rqtl2bundle: Path, + args: argparse.Namespace, logger: Logger = getLogger()) -> int: """Load data in `pheno` files and other related files into the database.""" + (speciesid, platformid, datasetid, rqtl2bundle) = ( + args.speciesid, args.platformid, args.datasetid, args.rqtl2bundle) with ZipFile(str(rqtl2bundle), "r") as zfile: try: rqc.validate_bundle(zfile) @@ -155,16 +156,5 @@ if __name__ == "__main__": return parser.parse_args() - thelogger = getLogger("install_phenos") - thelogger.addHandler(StreamHandler(stream=sys.stderr)) - main = build_main( - cli_args(), - lambda dbconn, args: install_pheno_files(dbconn, - args.speciesid, - args.platformid, - args.datasetid, - args.rqtl2bundle, - thelogger), - thelogger, - "DEBUG") + main = build_main(cli_args(), install_pheno_files, __MODULE__) sys.exit(main()) diff --git a/scripts/rqtl2/phenotypes_qc.py b/scripts/rqtl2/phenotypes_qc.py new file mode 100644 index 0000000..83828e4 --- /dev/null +++ b/scripts/rqtl2/phenotypes_qc.py @@ -0,0 +1,468 @@ +"""Run quality control on phenotypes-specific files in the bundle.""" +import sys +import uuid +import shutil +import logging +import tempfile +import contextlib +from pathlib import Path +from logging import Logger +from zipfile import ZipFile +from argparse import Namespace +import multiprocessing as mproc +from functools import reduce, partial +from typing import Union, Iterator, Callable, Optional, Sequence + +import MySQLdb as mdb +from redis import Redis + +from r_qtl import r_qtl2 as rqtl2 +from r_qtl import r_qtl2_qc as rqc +from r_qtl import exceptions as rqe +from r_qtl.fileerrors import InvalidValue + +from functional_tools import chain + +from quality_control.checks import decimal_places_pattern + +from uploader.files import sha256_digest_over_file +from uploader.samples.models import samples_by_species_and_population + +from scripts.rqtl2.entry import build_main +from scripts.redis_logger import RedisMessageListHandler +from scripts.rqtl2.cli_parser import add_bundle_argument +from scripts.cli_parser import init_cli_parser, add_global_data_arguments +from scripts.rqtl2.bundleutils import build_line_joiner, build_line_splitter + +__MODULE__ = "scripts.rqtl2.phenotypes_qc" + +def validate(phenobundle: Path, logger: Logger) -> dict: + """Check that the bundle is generally valid""" + try: + rqc.validate_bundle(phenobundle) + except rqe.RQTLError as rqtlerr: + # logger.error("Bundle file validation failed!", exc_info=True) + return { + "skip": True, + "logger": logger, + "phenobundle": phenobundle, + "errors": (" ".join(rqtlerr.args),) + } + return { + "errors": tuple(), + "skip": False, + "phenobundle": phenobundle, + "logger": logger + } + + +def check_for_mandatory_pheno_keys( + phenobundle: Path, + logger: Logger, + **kwargs +) -> dict: + """Check that the mandatory keys exist for phenotypes.""" + if kwargs.get("skip", False): + return { + **kwargs, + "logger": logger, + "phenobundle": phenobundle + } + + _mandatory_keys = ("pheno", "phenocovar") + _cdata = rqtl2.read_control_file(phenobundle) + _errors = kwargs.get("errors", tuple()) + tuple( + f"Expected '{key}' file(s) are not declared in the bundle." + for key in _mandatory_keys if key not in _cdata.keys()) + return { + **kwargs, + "logger": logger, + "phenobundle": phenobundle, + "errors": _errors, + "skip": len(_errors) > 0 + } + + +def check_for_averages_files( + phenobundle: Path, + logger: Logger, + **kwargs +) -> dict: + """Check that averages files appear together""" + if kwargs.get("skip", False): + return { + **kwargs, + "logger": logger, + "phenobundle": phenobundle + } + + _together = (("phenose", "phenonum"), ("phenonum", "phenose")) + _cdata = rqtl2.read_control_file(phenobundle) + _errors = kwargs.get("errors", tuple()) + tuple( + f"'{first}' is defined in the control file but there is no " + f"corresponding '{second}'" + for first, second in _together + if ((first in _cdata.keys()) and (second not in _cdata.keys()))) + return { + **kwargs, + "logger": logger, + "phenobundle": phenobundle, + "errors": _errors, + "skip": len(_errors) > 0 + } + + +def extract_bundle( + bundle: Path, workdir: Path, jobid: uuid.UUID +) -> tuple[Path, tuple[Path, ...]]: + """Extract the bundle.""" + with ZipFile(bundle) as zfile: + extractiondir = workdir.joinpath( + f"{str(jobid)}-{sha256_digest_over_file(bundle)}-{bundle.name}") + return extractiondir, rqtl2.extract(zfile, extractiondir) + + +def undo_transpose(filetype: str, cdata: dict, extractiondir): + """Undo transposition of all files of type `filetype` in thebundle.""" + if len(cdata.get(filetype, [])) > 0 and cdata.get(f"{filetype}_transposed", False): + files = (extractiondir.joinpath(_file) for _file in cdata[filetype]) + for _file in files: + rqtl2.transpose_csv_with_rename( + _file, + build_line_splitter(cdata), + build_line_joiner(cdata)) + + +@contextlib.contextmanager +def redis_logger( + redisuri: str, loggername: str, filename: str, fqkey: str +) -> Iterator[logging.Logger]: + """Build a Redis message-list logger.""" + rconn = Redis.from_url(redisuri, decode_responses=True) + logger = logging.getLogger(loggername) + logger.propagate = False + handler = RedisMessageListHandler( + rconn, + fullyqualifiedkey(fqkey, filename))#type: ignore[arg-type] + handler.setFormatter(logging.getLogger().handlers[0].formatter) + logger.addHandler(handler) + try: + yield logger + finally: + rconn.close() + + +def qc_phenocovar_file( + filepath: Path, + redisuri, + fqkey: str, + separator: str, + comment_char: str): + """Check that `phenocovar` files are structured correctly.""" + with redis_logger( + redisuri, + f"{__MODULE__}.qc_phenocovar_file", + filepath.name, + fqkey) as logger: + logger.info("Running QC on file: %s", filepath.name) + _csvfile = rqtl2.read_csv_file(filepath, separator, comment_char) + _headings = tuple(heading.lower() for heading in next(_csvfile)) + _errors: tuple[InvalidValue, ...] = tuple() + for heading in ("description", "units"): + if heading not in _headings: + _errors = (InvalidValue( + filepath.name, + "header row", + "-", + "-", + (f"File {filepath.name} is missing the {heading} heading " + "in the header line.")),) + + def collect_errors(errors_and_linecount, line): + _errs, _lc = errors_and_linecount + logger.info("Testing record '%s'", line[0]) + if len(line) != len(_headings): + _errs = _errs + (InvalidValue( + filepath.name, + line[0], + "-", + "-", + (f"Record {_lc} in file {filepath.name} has a different " + "number of columns than the number of headings")),) + _line = dict(zip(_headings, line)) + if not bool(_line["description"]): + _errs = _errs + ( + InvalidValue(filepath.name, + _line[_headings[0]], + "description", + _line["description"], + "The description is not provided!"),) + + return _errs, _lc+1 + + return { + filepath.name: dict(zip( + ("errors", "linecount"), + reduce(collect_errors, _csvfile, (_errors, 1)))) + } + + +def merge_dicts(*dicts): + """Merge multiple dicts into a single one.""" + return reduce(lambda merged, dct: {**merged, **dct}, dicts, {}) + + +def decimal_points_error(# pylint: disable=[too-many-arguments] + filename: str, + rowtitle: str, + coltitle: str, + cellvalue: str, + message: str, + decimal_places: int = 1 +) -> Optional[InvalidValue]: + """Returns an error if the value does not meet the checks.""" + if not bool(decimal_places_pattern(decimal_places).match(cellvalue)): + return InvalidValue(filename, rowtitle, coltitle, cellvalue, message) + return None + + +def integer_error( + filename: str, + rowtitle: str, + coltitle: str, + cellvalue: str, + message: str +) -> Optional[InvalidValue]: + """Returns an error if the value does not meet the checks.""" + try: + value = int(cellvalue) + if value <= 0: + raise ValueError("Must be a non-zero, positive number.") + return None + except ValueError as _verr: + return InvalidValue(filename, rowtitle, coltitle, cellvalue, message) + + +def qc_pheno_file(# pylint: disable=[too-many-arguments] + filepath: Path, + redisuri: str, + fqkey: str, + samples: tuple[str, ...], + phenonames: tuple[str, ...], + separator: str, + comment_char: str, + na_strings: Sequence[str], + error_fn: Callable = decimal_points_error +): + """Run QC/QA on a `pheno` file.""" + with redis_logger( + redisuri, + f"{__MODULE__}.qc_pheno_file", + filepath.name, + fqkey) as logger: + logger.info("Running QC on file: %s", filepath.name) + _csvfile = rqtl2.read_csv_file(filepath, separator, comment_char) + _headings: tuple[str, ...] = tuple( + heading.lower() for heading in next(_csvfile)) + _errors: tuple[InvalidValue, ...] = tuple() + + _absent = tuple(pheno for pheno in _headings[1:] if pheno not in phenonames) + if len(_absent) > 0: + _errors = _errors + (InvalidValue( + filepath.name, + "header row", + "-", + ", ".join(_absent), + (f"The phenotype names ({', '.join(samples)}) do not exist in any " + "of the provided phenocovar files.")),) + + def collect_errors(errors_and_linecount, line): + _errs, _lc = errors_and_linecount + if line[0] not in samples: + _errs = _errs + (InvalidValue( + filepath.name, + line[0], + _headings[0], + line[0], + (f"The sample named '{line[0]}' does not exist in the database. " + "You will need to upload that first.")),) + + for field, value in zip(_headings[1:], line[1:]): + if value in na_strings: + continue + _err = error_fn( + filepath.name, + line[0], + field, + value) + _errs = _errs + ((_err,) if bool(_err) else tuple()) + + return _errs, _lc+1 + + return { + filepath.name: dict(zip( + ("errors", "linecount"), + reduce(collect_errors, _csvfile, (_errors, 1)))) + } + + +def phenotype_names(filepath: Path, + separator: str, + comment_char: str) -> tuple[str, ...]: + """Read phenotype names from `phenocovar` file.""" + return reduce(lambda tpl, line: tpl + (line[0],),#type: ignore[arg-type, return-value] + rqtl2.read_csv_file(filepath, separator, comment_char), + tuple())[1:] + +def fullyqualifiedkey( + prefix: str, + rest: Optional[str] = None +) -> Union[Callable[[str], str], str]: + """Compute fully qualified Redis key.""" + if not bool(rest): + return lambda _rest: f"{prefix}:{_rest}" + return f"{prefix}:{rest}" + +def run_qc(# pylint: disable=[too-many-locals] + dbconn: mdb.Connection, + args: Namespace, + logger: Logger +) -> int: + """Run quality control checks on the bundle.""" + logger.debug("Beginning the quality assuarance checks.") + results = check_for_averages_files( + **check_for_mandatory_pheno_keys( + **validate(args.rqtl2bundle, logger))) + errors = results.get("errors", tuple()) + if len(errors) > 0: + logger.error("We found the following errors:\n%s", + "\n".join(f" - {error}" for error in errors)) + return 1 + # Run QC on actual values + # Steps: + # - Extract file to specific directory + extractiondir, *_bundlefiles = extract_bundle( + args.rqtl2bundle, args.workingdir, args.jobid) + + # - For every pheno, phenocovar, phenose, phenonum file, undo + # transposition where relevant + cdata = rqtl2.control_data(extractiondir) + with mproc.Pool(mproc.cpu_count() - 1) as pool: + pool.starmap( + undo_transpose, + ((ftype, cdata, extractiondir) + for ftype in ("pheno", "phenocovar", "phenose", "phenonum"))) + + # - Fetch samples/individuals from database. + logger.debug("Fetching samples/individuals from the database.") + samples = tuple(#type: ignore[var-annotated] + item for item in set(reduce( + lambda acc, item: acc + ( + item["Name"], item["Name2"], item["Symbol"], item["Alias"]), + samples_by_species_and_population( + dbconn, args.speciesid, args.populationid), + tuple())) + if bool(item)) + + # - Check that `description` and `units` is present in phenocovar for + # all phenotypes + with mproc.Pool(mproc.cpu_count() - 1) as pool: + logger.debug("Check for errors in 'phenocovar' file(s).") + _phenocovar_qc_res = merge_dicts(*pool.starmap(qc_phenocovar_file, tuple( + (extractiondir.joinpath(_file), + args.redisuri, + chain( + "phenocovar", + fullyqualifiedkey(args.jobid), + fullyqualifiedkey(args.redisprefix)), + cdata["sep"], + cdata["comment.char"]) + for _file in cdata.get("phenocovar", [])))) + + # - Check all samples in pheno files exist in database + # - Check all phenotypes in pheno files exist in phenocovar files + # - Check all numeric values in pheno files + phenonames = tuple(set( + name for names in pool.starmap(phenotype_names, tuple( + (extractiondir.joinpath(_file), cdata["sep"], cdata["comment.char"]) + for _file in cdata.get("phenocovar", []))) + for name in names)) + + dec_err_fn = partial(decimal_points_error, message=( + "Expected a non-negative number with at least one decimal " + "place.")) + + logger.debug("Check for errors in 'pheno' file(s).") + _pheno_qc_res = merge_dicts(*pool.starmap(qc_pheno_file, tuple(( + extractiondir.joinpath(_file), + args.redisuri, + chain( + "pheno", + fullyqualifiedkey(args.jobid), + fullyqualifiedkey(args.redisprefix)), + samples, + phenonames, + cdata["sep"], + cdata["comment.char"], + cdata["na.strings"], + dec_err_fn + ) for _file in cdata.get("pheno", [])))) + + # - Check the 3 checks above for phenose and phenonum values too + # qc_phenose_files(…) + # qc_phenonum_files(…) + logger.debug("Check for errors in 'phenose' file(s).") + _phenose_qc_res = merge_dicts(*pool.starmap(qc_pheno_file, tuple(( + extractiondir.joinpath(_file), + args.redisuri, + chain( + "phenose", + fullyqualifiedkey(args.jobid), + fullyqualifiedkey(args.redisprefix)), + samples, + phenonames, + cdata["sep"], + cdata["comment.char"], + cdata["na.strings"], + dec_err_fn + ) for _file in cdata.get("phenose", [])))) + + logger.debug("Check for errors in 'phenonum' file(s).") + _phenonum_qc_res = merge_dicts(*pool.starmap(qc_pheno_file, tuple(( + extractiondir.joinpath(_file), + args.redisuri, + chain( + "phenonum", + fullyqualifiedkey(args.jobid), + fullyqualifiedkey(args.redisprefix)), + samples, + phenonames, + cdata["sep"], + cdata["comment.char"], + cdata["na.strings"], + partial(integer_error, message=( + "Expected a non-negative, non-zero integer value.")) + ) for _file in cdata.get("phenonum", [])))) + + # - Delete all extracted files + shutil.rmtree(extractiondir) + raise NotImplementedError("WIP!") + + +if __name__ == "__main__": + def cli_args(): + """Process command-line arguments for `install_phenos`""" + parser = add_bundle_argument(add_global_data_arguments(init_cli_parser( + program="PhenotypesQC", + description=( + "Perform Quality Control checks on a phenotypes bundle file")))) + parser.add_argument( + "--workingdir", + default=f"{tempfile.gettempdir()}/phenotypes_qc", + help=("The directory where this script will put its intermediate " + "files."), + type=Path) + return parser.parse_args() + + main = build_main(cli_args(), run_qc, __MODULE__) + sys.exit(main()) diff --git a/tests/qc_app/__init__.py b/tests/uploader/__init__.py index e69de29..e69de29 100644 --- a/tests/qc_app/__init__.py +++ b/tests/uploader/__init__.py diff --git a/tests/qc_app/test_entry.py b/tests/uploader/test_entry.py index 0c614a5..0c614a5 100644 --- a/tests/qc_app/test_entry.py +++ b/tests/uploader/test_entry.py diff --git a/tests/qc_app/test_expression_data_pages.py b/tests/uploader/test_expression_data_pages.py index c2f7de1..c2f7de1 100644 --- a/tests/qc_app/test_expression_data_pages.py +++ b/tests/uploader/test_expression_data_pages.py diff --git a/tests/uploader/test_files.py b/tests/uploader/test_files.py new file mode 100644 index 0000000..cb22fff --- /dev/null +++ b/tests/uploader/test_files.py @@ -0,0 +1,17 @@ +"""Tests functions in the `uploader.files` module.""" +from pathlib import Path + +import pytest + +from uploader.files import sha256_digest_over_file + +@pytest.mark.unit_test +@pytest.mark.parametrize( + "filepath,expectedhash", + ((Path("tests/test_data/average.tsv.zip"), + "a371c654c095c030edad468e1c3d6b176ea8adfbcd91a322afd37779044478d9"), + (Path("tests/test_data/standarderror.tsv"), + "a08332e0b06391d50eecb722f69d85fbdf374a2d77713ee879d3fd6c60419d55"))) +def test_sha256_digest_over_file(filepath: Path, expectedhash: str): + """Test the `sha256_digest_over_file` function.""" + assert sha256_digest_over_file(filepath) == expectedhash diff --git a/tests/qc_app/test_parse.py b/tests/uploader/test_parse.py index 076c47c..076c47c 100644 --- a/tests/qc_app/test_parse.py +++ b/tests/uploader/test_parse.py diff --git a/tests/qc_app/test_progress_indication.py b/tests/uploader/test_progress_indication.py index 14a1050..14a1050 100644 --- a/tests/qc_app/test_progress_indication.py +++ b/tests/uploader/test_progress_indication.py diff --git a/tests/qc_app/test_results_page.py b/tests/uploader/test_results_page.py index 8c8379f..8c8379f 100644 --- a/tests/qc_app/test_results_page.py +++ b/tests/uploader/test_results_page.py diff --git a/tests/qc_app/test_uploads_with_zip_files.py b/tests/uploader/test_uploads_with_zip_files.py index 1506cfa..1506cfa 100644 --- a/tests/qc_app/test_uploads_with_zip_files.py +++ b/tests/uploader/test_uploads_with_zip_files.py diff --git a/uploader/__init__.py b/uploader/__init__.py index 1af159b..9fdb383 100644 --- a/uploader/__init__.py +++ b/uploader/__init__.py @@ -12,7 +12,6 @@ from uploader.oauth2.client import user_logged_in, authserver_authorise_uri from . import session from .base_routes import base from .species import speciesbp -from .dbinsert import dbinsertbp from .oauth2.views import oauth2 from .expression_data import exprdatabp from .errors import register_error_handlers @@ -85,8 +84,6 @@ def create_app(): app.register_blueprint(base, url_prefix="/") app.register_blueprint(oauth2, url_prefix="/oauth2") app.register_blueprint(speciesbp, url_prefix="/species") - app.register_blueprint(dbinsertbp, url_prefix="/dbinsert") - app.register_blueprint(exprdatabp, url_prefix="/expression-data") register_error_handlers(app) return app diff --git a/uploader/base_routes.py b/uploader/base_routes.py index a20b7ff..742a254 100644 --- a/uploader/base_routes.py +++ b/uploader/base_routes.py @@ -2,15 +2,15 @@ import os from urllib.parse import urljoin -from flask import ( - Blueprint, - render_template, - current_app as app, - send_from_directory) +from flask import (Blueprint, + current_app as app, + send_from_directory) +from uploader.ui import make_template_renderer from uploader.oauth2.client import user_logged_in base = Blueprint("base", __name__) +render_template = make_template_renderer("home") @base.route("/favicon.ico", methods=["GET"]) diff --git a/uploader/datautils.py b/uploader/datautils.py index 2ee079d..46a55c4 100644 --- a/uploader/datautils.py +++ b/uploader/datautils.py @@ -1,7 +1,7 @@ """Generic data utilities: Rename module.""" import math -from typing import Sequence from functools import reduce +from typing import Union, Sequence def enumerate_sequence(seq: Sequence[dict], start:int = 1) -> Sequence[dict]: """Enumerate sequence beginning at 1""" @@ -28,7 +28,7 @@ def order_by_family(items: tuple[dict, ...], key=lambda item: item[0][0]) -def safe_int(val: str) -> int: +def safe_int(val: Union[str, int, float]) -> int: """ Convert val into an integer: if val cannot be converted, return a zero. """ diff --git a/uploader/db/platforms.py b/uploader/db/platforms.py deleted file mode 100644 index cb527a7..0000000 --- a/uploader/db/platforms.py +++ /dev/null @@ -1,25 +0,0 @@ -"""Handle db interactions for platforms.""" -from typing import Optional - -import MySQLdb as mdb -from MySQLdb.cursors import DictCursor - -def platforms_by_species( - conn: mdb.Connection, speciesid: int) -> tuple[dict, ...]: - """Retrieve platforms by the species""" - with conn.cursor(cursorclass=DictCursor) as cursor: - cursor.execute("SELECT * FROM GeneChip WHERE SpeciesId=%s " - "ORDER BY GeneChipName ASC", - (speciesid,)) - return tuple(dict(row) for row in cursor.fetchall()) - -def platform_by_id(conn: mdb.Connection, platformid: int) -> Optional[dict]: - """Retrieve a platform by its ID""" - with conn.cursor(cursorclass=DictCursor) as cursor: - cursor.execute("SELECT * FROM GeneChip WHERE Id=%s", - (platformid,)) - result = cursor.fetchone() - if bool(result): - return dict(result) - - return None diff --git a/uploader/default_settings.py b/uploader/default_settings.py index 26fe665..1acb247 100644 --- a/uploader/default_settings.py +++ b/uploader/default_settings.py @@ -2,15 +2,12 @@ The default configuration file. The values here should be overridden in the actual configuration file used for the production and staging systems. """ - -import os - -LOG_LEVEL = os.getenv("LOG_LEVEL", "WARNING") +LOG_LEVEL = "WARNING" SECRET_KEY = b"<Please! Please! Please! Change This!>" UPLOAD_FOLDER = "/tmp/qc_app_files" REDIS_URL = "redis://" JOBS_TTL_SECONDS = 1209600 # 14 days -GNQC_REDIS_PREFIX="GNQC" +GNQC_REDIS_PREFIX="gn-uploader" SQL_URI = "" GN2_SERVER_URL = "https://genenetwork.org/" diff --git a/uploader/expression_data/__init__.py b/uploader/expression_data/__init__.py index 206a764..fc8bd41 100644 --- a/uploader/expression_data/__init__.py +++ b/uploader/expression_data/__init__.py @@ -1,11 +1,2 @@ """Package handling upload of files.""" -from flask import Blueprint - -from .rqtl2 import rqtl2 -from .index import indexbp -from .parse import parsebp - -exprdatabp = Blueprint("expression-data", __name__) -exprdatabp.register_blueprint(indexbp, url_prefix="/") -exprdatabp.register_blueprint(rqtl2, url_prefix="/rqtl2") -exprdatabp.register_blueprint(parsebp, url_prefix="/parse") +from .views import exprdatabp diff --git a/uploader/dbinsert.py b/uploader/expression_data/dbinsert.py index 2116031..32ca359 100644 --- a/uploader/dbinsert.py +++ b/uploader/expression_data/dbinsert.py @@ -11,12 +11,12 @@ from flask import ( flash, request, url_for, Blueprint, redirect, render_template, current_app as app) +from uploader import jobs from uploader.authorisation import require_login from uploader.population.models import populations_by_species +from uploader.species.models import all_species, species_by_id +from uploader.platforms.models import platform_by_species_and_id from uploader.db_utils import with_db_connection, database_connection -from uploader.species.models import species_by_id, all_species as species - -from . import jobs dbinsertbp = Blueprint("dbinsert", __name__) @@ -49,14 +49,6 @@ def genechips(): return {} -def platform_by_id(genechipid:int) -> Union[dict, None]: - "Retrieve the gene platform by id" - with database_connection(app.config["SQL_URI"]) as conn: - with conn.cursor(cursorclass=DictCursor) as cursor: - cursor.execute( - "SELECT * FROM GeneChip WHERE GeneChipId=%s", - (genechipid,)) - return cursor.fetchone() def studies_by_species_and_platform(speciesid:int, genechipid:int) -> tuple: "Retrieve the studies by the related species and gene platform" @@ -108,7 +100,7 @@ def select_platform(): return render_template( "select_platform.html", filename=filename, filetype=job["filetype"], totallines=int(job["currentline"]), - default_species=default_species, species=species(conn), + default_species=default_species, species=all_species(conn), genechips=gchips[default_species], genechips_data=json.dumps(gchips)) return render_error(f"File '{filename}' no longer exists.") @@ -327,37 +319,38 @@ def selected_keys(original: dict, keys: tuple) -> dict: @require_login def final_confirmation(): "Preview the data before triggering entry into the database" - form = request.form - try: - assert form.get("filename"), "filename" - assert form.get("filetype"), "filetype" - assert form.get("species"), "species" - assert form.get("genechipid"), "platform" - assert form.get("studyid"), "study" - assert form.get("datasetid"), "dataset" - - speciesid = form["species"] - genechipid = form["genechipid"] - studyid = form["studyid"] - datasetid=form["datasetid"] - return render_template( - "final_confirmation.html", filename=form["filename"], - filetype=form["filetype"], totallines=form["totallines"], - species=speciesid, genechipid=genechipid, studyid=studyid, - datasetid=datasetid, the_species=selected_keys( - with_db_connection(lambda conn: species_by_id(conn, speciesid)), - ("SpeciesName", "Name", "MenuName")), - platform=selected_keys( - platform_by_id(genechipid), - ("GeneChipName", "Name", "GeoPlatform", "Title", "GO_tree_value")), - study=selected_keys( - study_by_id(studyid), ("Name", "FullName", "ShortName")), - dataset=selected_keys( - dataset_by_id(datasetid), - ("AvgMethodName", "Name", "Name2", "FullName", "ShortName", - "DataScale"))) - except AssertionError as aserr: - return render_error(f"Missing data: {aserr.args[0]}") + with database_connection(app.config["SQL_URI"]) as conn: + form = request.form + try: + assert form.get("filename"), "filename" + assert form.get("filetype"), "filetype" + assert form.get("species"), "species" + assert form.get("genechipid"), "platform" + assert form.get("studyid"), "study" + assert form.get("datasetid"), "dataset" + + speciesid = form["species"] + genechipid = form["genechipid"] + studyid = form["studyid"] + datasetid=form["datasetid"] + return render_template( + "final_confirmation.html", filename=form["filename"], + filetype=form["filetype"], totallines=form["totallines"], + species=speciesid, genechipid=genechipid, studyid=studyid, + datasetid=datasetid, the_species=selected_keys( + with_db_connection(lambda conn: species_by_id(conn, speciesid)), + ("SpeciesName", "Name", "MenuName")), + platform=selected_keys( + platform_by_species_and_id(conn, speciesid, genechipid), + ("GeneChipName", "Name", "GeoPlatform", "Title", "GO_tree_value")), + study=selected_keys( + study_by_id(studyid), ("Name", "FullName", "ShortName")), + dataset=selected_keys( + dataset_by_id(datasetid), + ("AvgMethodName", "Name", "Name2", "FullName", "ShortName", + "DataScale"))) + except AssertionError as aserr: + return render_error(f"Missing data: {aserr.args[0]}") @dbinsertbp.route("/insert-data", methods=["POST"]) @require_login diff --git a/uploader/expression_data/index.py b/uploader/expression_data/index.py deleted file mode 100644 index db23136..0000000 --- a/uploader/expression_data/index.py +++ /dev/null @@ -1,125 +0,0 @@ -"""Entry-point module""" -import os -import mimetypes -from typing import Tuple -from zipfile import ZipFile, is_zipfile - -from werkzeug.utils import secure_filename -from flask import ( - flash, - request, - url_for, - redirect, - Blueprint, - render_template, - current_app as app) - -from uploader.species.models import all_species as species -from uploader.authorisation import require_login -from uploader.db_utils import with_db_connection - -indexbp = Blueprint("index", __name__) - - -def errors(rqst) -> Tuple[str, ...]: - """Return a tuple of the errors found in the request `rqst`. If no error is - found, then an empty tuple is returned.""" - def __filetype_error__(): - return ( - ("Invalid file type provided.",) - if rqst.form.get("filetype") not in ("average", "standard-error") - else tuple()) - - def __file_missing_error__(): - return ( - ("No file was uploaded.",) - if ("qc_text_file" not in rqst.files or - rqst.files["qc_text_file"].filename == "") - else tuple()) - - def __file_mimetype_error__(): - text_file = rqst.files["qc_text_file"] - return ( - ( - ("Invalid file! Expected a tab-separated-values file, or a zip " - "file of the a tab-separated-values file."),) - if text_file.mimetype not in ( - "text/plain", "text/tab-separated-values", - "application/zip") - else tuple()) - - return ( - __filetype_error__() + - (__file_missing_error__() or __file_mimetype_error__())) - -def zip_file_errors(filepath, upload_dir) -> Tuple[str, ...]: - """Check the uploaded zip file for errors.""" - zfile_errors: Tuple[str, ...] = tuple() - if is_zipfile(filepath): - with ZipFile(filepath, "r") as zfile: - infolist = zfile.infolist() - if len(infolist) != 1: - zfile_errors = zfile_errors + ( - ("Expected exactly one (1) member file within the uploaded zip " - f"file. Got {len(infolist)} member files."),) - if len(infolist) == 1 and infolist[0].is_dir(): - zfile_errors = zfile_errors + ( - ("Expected a member text file in the uploaded zip file. Got a " - "directory/folder."),) - - if len(infolist) == 1 and not infolist[0].is_dir(): - zfile.extract(infolist[0], path=upload_dir) - mime = mimetypes.guess_type(f"{upload_dir}/{infolist[0].filename}") - if mime[0] != "text/tab-separated-values": - zfile_errors = zfile_errors + ( - ("Expected the member text file in the uploaded zip file to" - " be a tab-separated file."),) - - return zfile_errors - - -@indexbp.route("/", methods=["GET"]) -@require_login -def index(): - """Display the expression data index page.""" - return render_template("expression-data/index.html") - - -@indexbp.route("/upload", methods=["GET", "POST"]) -@require_login -def upload_file(): - """Enables uploading the files""" - if request.method == "GET": - return render_template( - "select_species.html", species=with_db_connection(species)) - - upload_dir = app.config["UPLOAD_FOLDER"] - request_errors = errors(request) - if request_errors: - for error in request_errors: - flash(error, "alert-danger error-expr-data") - return redirect(url_for("expression-data.index.upload_file")) - - filename = secure_filename(request.files["qc_text_file"].filename) - if not os.path.exists(upload_dir): - os.mkdir(upload_dir) - - filepath = os.path.join(upload_dir, filename) - request.files["qc_text_file"].save(os.path.join(upload_dir, filename)) - - zip_errors = zip_file_errors(filepath, upload_dir) - if zip_errors: - for error in zip_errors: - flash(error, "alert-danger error-expr-data") - return redirect(url_for("expression-data.index.upload_file")) - - return redirect(url_for("expression-data.parse.parse", - speciesid=request.form["speciesid"], - filename=filename, - filetype=request.form["filetype"])) - -@indexbp.route("/data-review", methods=["GET"]) -@require_login -def data_review(): - """Provide some help on data expectations to the user.""" - return render_template("data_review.html") diff --git a/uploader/expression_data/parse.py b/uploader/expression_data/parse.py deleted file mode 100644 index fc1c3f0..0000000 --- a/uploader/expression_data/parse.py +++ /dev/null @@ -1,178 +0,0 @@ -"""File parsing module""" -import os - -import jsonpickle -from redis import Redis -from flask import flash, request, url_for, redirect, Blueprint, render_template -from flask import current_app as app - -from quality_control.errors import InvalidValue, DuplicateHeading - -from uploader import jobs -from uploader.dbinsert import species_by_id -from uploader.db_utils import with_db_connection -from uploader.authorisation import require_login - -parsebp = Blueprint("parse", __name__) - -def isinvalidvalue(item): - """Check whether item is of type InvalidValue""" - return isinstance(item, InvalidValue) - -def isduplicateheading(item): - """Check whether item is of type DuplicateHeading""" - return isinstance(item, DuplicateHeading) - -@parsebp.route("/parse", methods=["GET"]) -@require_login -def parse(): - """Trigger file parsing""" - errors = False - speciesid = request.args.get("speciesid") - filename = request.args.get("filename") - filetype = request.args.get("filetype") - if speciesid is None: - flash("No species selected", "alert-error error-expr-data") - errors = True - else: - try: - speciesid = int(speciesid) - species = with_db_connection( - lambda con: species_by_id(con, speciesid)) - if not bool(species): - flash("No such species.", "alert-error error-expr-data") - errors = True - except ValueError: - flash("Invalid speciesid provided. Expected an integer.", - "alert-error error-expr-data") - errors = True - - if filename is None: - flash("No file provided", "alert-error error-expr-data") - errors = True - - if filetype is None: - flash("No filetype provided", "alert-error error-expr-data") - errors = True - - if filetype not in ("average", "standard-error"): - flash("Invalid filetype provided", "alert-error error-expr-data") - errors = True - - if filename: - filepath = os.path.join(app.config["UPLOAD_FOLDER"], filename) - if not os.path.exists(filepath): - flash("Selected file does not exist (any longer)", - "alert-error error-expr-data") - errors = True - - if errors: - return redirect(url_for("expression-data.index.upload_file")) - - redisurl = app.config["REDIS_URL"] - with Redis.from_url(redisurl, decode_responses=True) as rconn: - job = jobs.launch_job( - jobs.build_file_verification_job( - rconn, app.config["SQL_URI"], redisurl, - speciesid, filepath, filetype, - app.config["JOBS_TTL_SECONDS"]), - redisurl, - f"{app.config['UPLOAD_FOLDER']}/job_errors") - - return redirect(url_for("expression-data.parse.parse_status", job_id=job["jobid"])) - -@parsebp.route("/status/<job_id>", methods=["GET"]) -def parse_status(job_id: str): - "Retrieve the status of the job" - with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: - try: - job = jobs.job(rconn, jobs.jobsnamespace(), job_id) - except jobs.JobNotFound as _exc: - return render_template("no_such_job.html", job_id=job_id), 400 - - error_filename = jobs.error_filename( - job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors") - if os.path.exists(error_filename): - stat = os.stat(error_filename) - if stat.st_size > 0: - return redirect(url_for("parse.fail", job_id=job_id)) - - job_id = job["jobid"] - progress = float(job["percent"]) - status = job["status"] - filename = job.get("filename", "uploaded file") - errors = jsonpickle.decode( - job.get("errors", jsonpickle.encode(tuple()))) - if status in ("success", "aborted"): - return redirect(url_for("expression-data.parse.results", job_id=job_id)) - - if status == "parse-error": - return redirect(url_for("parse.fail", job_id=job_id)) - - app.jinja_env.globals.update( - isinvalidvalue=isinvalidvalue, - isduplicateheading=isduplicateheading) - return render_template( - "job_progress.html", - job_id = job_id, - job_status = status, - progress = progress, - message = job.get("message", ""), - job_name = f"Parsing '{filename}'", - errors=errors) - -@parsebp.route("/results/<job_id>", methods=["GET"]) -def results(job_id: str): - """Show results of parsing...""" - with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: - job = jobs.job(rconn, jobs.jobsnamespace(), job_id) - - if job: - filename = job["filename"] - errors = jsonpickle.decode(job.get("errors", jsonpickle.encode(tuple()))) - app.jinja_env.globals.update( - isinvalidvalue=isinvalidvalue, - isduplicateheading=isduplicateheading) - return render_template( - "parse_results.html", - errors=errors, - job_name = f"Parsing '{filename}'", - user_aborted = job.get("user_aborted"), - job_id=job["jobid"]) - - return render_template("no_such_job.html", job_id=job_id) - -@parsebp.route("/fail/<job_id>", methods=["GET"]) -def fail(job_id: str): - """Handle parsing failure""" - with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: - job = jobs.job(rconn, jobs.jobsnamespace(), job_id) - - if job: - error_filename = jobs.error_filename( - job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors") - if os.path.exists(error_filename): - stat = os.stat(error_filename) - if stat.st_size > 0: - return render_template( - "worker_failure.html", job_id=job_id) - - return render_template("parse_failure.html", job=job) - - return render_template("no_such_job.html", job_id=job_id) - -@parsebp.route("/abort", methods=["POST"]) -@require_login -def abort(): - """Handle user request to abort file processing""" - job_id = request.form["job_id"] - - with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: - job = jobs.job(rconn, jobs.jobsnamespace(), job_id) - - if job: - rconn.hset(name=jobs.job_key(jobs.jobsnamespace(), job_id), - key="user_aborted", - value=int(True)) - - return redirect(url_for("expression-data.parse.parse_status", job_id=job_id)) diff --git a/uploader/expression_data/views.py b/uploader/expression_data/views.py new file mode 100644 index 0000000..bbe6538 --- /dev/null +++ b/uploader/expression_data/views.py @@ -0,0 +1,384 @@ +"""Views for expression data""" +import os +import uuid +import mimetypes +from typing import Tuple +from zipfile import ZipFile, is_zipfile + +import jsonpickle +from redis import Redis +from werkzeug.utils import secure_filename +from flask import (flash, + request, + url_for, + redirect, + Blueprint, + current_app as app) + +from quality_control.errors import InvalidValue, DuplicateHeading + +from uploader import jobs +from uploader.datautils import order_by_family +from uploader.ui import make_template_renderer +from uploader.authorisation import require_login +from uploader.species.models import all_species, species_by_id +from uploader.db_utils import with_db_connection, database_connection +from uploader.population.models import (populations_by_species, + population_by_species_and_id) + +exprdatabp = Blueprint("expression-data", __name__) +render_template = make_template_renderer("expression-data") + +def isinvalidvalue(item): + """Check whether item is of type InvalidValue""" + return isinstance(item, InvalidValue) + + +def isduplicateheading(item): + """Check whether item is of type DuplicateHeading""" + return isinstance(item, DuplicateHeading) + + +def errors(rqst) -> Tuple[str, ...]: + """Return a tuple of the errors found in the request `rqst`. If no error is + found, then an empty tuple is returned.""" + def __filetype_error__(): + return ( + ("Invalid file type provided.",) + if rqst.form.get("filetype") not in ("average", "standard-error") + else tuple()) + + def __file_missing_error__(): + return ( + ("No file was uploaded.",) + if ("qc_text_file" not in rqst.files or + rqst.files["qc_text_file"].filename == "") + else tuple()) + + def __file_mimetype_error__(): + text_file = rqst.files["qc_text_file"] + return ( + ( + ("Invalid file! Expected a tab-separated-values file, or a zip " + "file of the a tab-separated-values file."),) + if text_file.mimetype not in ( + "text/plain", "text/tab-separated-values", + "application/zip") + else tuple()) + + return ( + __filetype_error__() + + (__file_missing_error__() or __file_mimetype_error__())) + + +def zip_file_errors(filepath, upload_dir) -> Tuple[str, ...]: + """Check the uploaded zip file for errors.""" + zfile_errors: Tuple[str, ...] = tuple() + if is_zipfile(filepath): + with ZipFile(filepath, "r") as zfile: + infolist = zfile.infolist() + if len(infolist) != 1: + zfile_errors = zfile_errors + ( + ("Expected exactly one (1) member file within the uploaded zip " + f"file. Got {len(infolist)} member files."),) + if len(infolist) == 1 and infolist[0].is_dir(): + zfile_errors = zfile_errors + ( + ("Expected a member text file in the uploaded zip file. Got a " + "directory/folder."),) + + if len(infolist) == 1 and not infolist[0].is_dir(): + zfile.extract(infolist[0], path=upload_dir) + mime = mimetypes.guess_type(f"{upload_dir}/{infolist[0].filename}") + if mime[0] != "text/tab-separated-values": + zfile_errors = zfile_errors + ( + ("Expected the member text file in the uploaded zip file to" + " be a tab-separated file."),) + + return zfile_errors + + +@exprdatabp.route("populations/expression-data", methods=["GET"]) +@require_login +def index(): + """Display the expression data index page.""" + with database_connection(app.config["SQL_URI"]) as conn: + if not bool(request.args.get("species_id")): + return render_template("expression-data/index.html", + species=order_by_family(all_species(conn)), + activelink="expression-data") + species = species_by_id(conn, request.args.get("species_id")) + if not bool(species): + flash("Could not find species selected!", "alert-danger") + return redirect(url_for("species.populations.expression-data.index")) + return redirect(url_for( + "species.populations.expression-data.select_population", + species_id=species["SpeciesId"])) + + +@exprdatabp.route("<int:species_id>/populations/expression-data/select-population", + methods=["GET"]) +@require_login +def select_population(species_id: int): + """Select the expression data's population.""" + with database_connection(app.config["SQL_URI"]) as conn: + species = species_by_id(conn, species_id) + if not bool(species): + flash("No such species!", "alert-danger") + return redirect(url_for("species.populations.expression-data.index")) + + if not bool(request.args.get("population_id")): + return render_template("expression-data/select-population.html", + species=species, + populations=order_by_family( + populations_by_species(conn, species_id), + order_key="FamilyOrder"), + activelink="expression-data") + + population = population_by_species_and_id( + conn, species_id, request.args.get("population_id")) + if not bool(population): + flash("No such population!", "alert-danger") + return redirect(url_for( + "species.populations.expression-data.select_population", + species_id=species_id)) + + return redirect(url_for("species.populations.expression-data.upload_file", + species_id=species_id, + population_id=population["Id"])) + + +@exprdatabp.route("<int:species_id>/populations/<int:population_id>/" + "expression-data/upload", + methods=["GET", "POST"]) +@require_login +def upload_file(species_id: int, population_id: int): + """Enables uploading the files""" + with database_connection(app.config["SQL_URI"]) as conn: + species = species_by_id(conn, species_id) + population = population_by_species_and_id(conn, species_id, population_id) + if request.method == "GET": + return render_template("expression-data/select-file.html", + species=species, + population=population) + + upload_dir = app.config["UPLOAD_FOLDER"] + request_errors = errors(request) + if request_errors: + for error in request_errors: + flash(error, "alert-danger error-expr-data") + return redirect(url_for("species.populations.expression-data.upload_file")) + + filename = secure_filename( + request.files["qc_text_file"].filename)# type: ignore[arg-type] + if not os.path.exists(upload_dir): + os.mkdir(upload_dir) + + filepath = os.path.join(upload_dir, filename) + request.files["qc_text_file"].save(os.path.join(upload_dir, filename)) + + zip_errors = zip_file_errors(filepath, upload_dir) + if zip_errors: + for error in zip_errors: + flash(error, "alert-danger error-expr-data") + return redirect(url_for("species.populations.expression-data.index.upload_file")) + + return redirect(url_for("species.populations.expression-data.parse_file", + species_id=species_id, + population_id=population_id, + filename=filename, + filetype=request.form["filetype"])) + + +@exprdatabp.route("/data-review", methods=["GET"]) +@require_login +def data_review(): + """Provide some help on data expectations to the user.""" + return render_template("expression-data/data-review.html") + + +@exprdatabp.route( + "<int:species_id>/populations/<int:population_id>/expression-data/parse", + methods=["GET"]) +@require_login +def parse_file(species_id: int, population_id: int): + """Trigger file parsing""" + _errors = False + filename = request.args.get("filename") + filetype = request.args.get("filetype") + + species = with_db_connection(lambda con: species_by_id(con, species_id)) + if not bool(species): + flash("No such species.", "alert-danger") + _errors = True + + if filename is None: + flash("No file provided", "alert-danger") + _errors = True + + if filetype is None: + flash("No filetype provided", "alert-danger") + _errors = True + + if filetype not in ("average", "standard-error"): + flash("Invalid filetype provided", "alert-danger") + _errors = True + + if filename: + filepath = os.path.join(app.config["UPLOAD_FOLDER"], filename) + if not os.path.exists(filepath): + flash("Selected file does not exist (any longer)", "alert-danger") + _errors = True + + if _errors: + return redirect(url_for("species.populations.expression-data.upload_file")) + + redisurl = app.config["REDIS_URL"] + with Redis.from_url(redisurl, decode_responses=True) as rconn: + job = jobs.launch_job( + jobs.build_file_verification_job( + rconn, app.config["SQL_URI"], redisurl, + species_id, filepath, filetype,# type: ignore[arg-type] + app.config["JOBS_TTL_SECONDS"]), + redisurl, + f"{app.config['UPLOAD_FOLDER']}/job_errors") + + return redirect(url_for("species.populations.expression-data.parse_status", + species_id=species_id, + population_id=population_id, + job_id=job["jobid"])) + + +@exprdatabp.route( + "<int:species_id>/populations/<int:population_id>/expression-data/parse/" + "status/<uuid:job_id>", + methods=["GET"]) +@require_login +def parse_status(species_id: int, population_id: int, job_id: str): + "Retrieve the status of the job" + with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: + try: + job = jobs.job(rconn, jobs.jobsnamespace(), job_id) + except jobs.JobNotFound as _exc: + return render_template("no_such_job.html", job_id=job_id), 400 + + error_filename = jobs.error_filename( + job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors") + if os.path.exists(error_filename): + stat = os.stat(error_filename) + if stat.st_size > 0: + return redirect(url_for("parse.fail", job_id=job_id)) + + job_id = job["jobid"] + progress = float(job["percent"]) + status = job["status"] + filename = job.get("filename", "uploaded file") + _errors = jsonpickle.decode( + job.get("errors", jsonpickle.encode(tuple()))) + if status in ("success", "aborted"): + return redirect(url_for("species.populations.expression-data.results", + species_id=species_id, + population_id=population_id, + job_id=job_id)) + + if status == "parse-error": + return redirect(url_for("species.populations.expression-data.fail", job_id=job_id)) + + app.jinja_env.globals.update( + isinvalidvalue=isinvalidvalue, + isduplicateheading=isduplicateheading) + return render_template( + "expression-data/job-progress.html", + job_id = job_id, + job_status = status, + progress = progress, + message = job.get("message", ""), + job_name = f"Parsing '{filename}'", + errors=_errors, + species=with_db_connection( + lambda conn: species_by_id(conn, species_id)), + population=with_db_connection( + lambda conn: population_by_species_and_id( + conn, species_id, population_id))) + + +@exprdatabp.route( + "<int:species_id>/populations/<int:population_id>/expression-data/parse/" + "<uuid:job_id>/results", + methods=["GET"]) +@require_login +def results(species_id: int, population_id: int, job_id: uuid.UUID): + """Show results of parsing...""" + with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: + job = jobs.job(rconn, jobs.jobsnamespace(), job_id) + + if job: + filename = job["filename"] + _errors = jsonpickle.decode(job.get("errors", jsonpickle.encode(tuple()))) + app.jinja_env.globals.update( + isinvalidvalue=isinvalidvalue, + isduplicateheading=isduplicateheading) + return render_template( + "expression-data/parse-results.html", + errors=_errors, + job_name = f"Parsing '{filename}'", + user_aborted = job.get("user_aborted"), + job_id=job["jobid"], + species=with_db_connection( + lambda conn: species_by_id(conn, species_id)), + population=with_db_connection( + lambda conn: population_by_species_and_id( + conn, species_id, population_id))) + + return render_template("expression-data/no-such-job.html", job_id=job_id) + + +@exprdatabp.route( + "<int:species_id>/populations/<int:population_id>/expression-data/parse/" + "<uuid:job_id>/fail", + methods=["GET"]) +@require_login +def fail(species_id: int, population_id: int, job_id: str): + """Handle parsing failure""" + with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: + job = jobs.job(rconn, jobs.jobsnamespace(), job_id) + + if job: + error_filename = jobs.error_filename( + job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors") + if os.path.exists(error_filename): + stat = os.stat(error_filename) + if stat.st_size > 0: + return render_template( + "worker_failure.html", job_id=job_id) + + return render_template("parse_failure.html", job=job) + + return render_template("expression-data/no-such-job.html", + **with_db_connection(lambda conn: { + "species_id": species_by_id(conn, species_id), + "population_id": population_by_species_and_id( + conn, species_id, population_id)}), + job_id=job_id) + + +@exprdatabp.route( + "<int:species_id>/populations/<int:population_id>/expression-data/parse/" + "abort", + methods=["POST"]) +@require_login +def abort(species_id: int, population_id: int): + """Handle user request to abort file processing""" + job_id = request.form["job_id"] + + with Redis.from_url(app.config["REDIS_URL"], decode_responses=True) as rconn: + job = jobs.job(rconn, jobs.jobsnamespace(), job_id) + + if job: + rconn.hset(name=jobs.job_key(jobs.jobsnamespace(), job_id), + key="user_aborted", + value=int(True)) + + return redirect(url_for("species.populations.expression-data.parse_status", + species_id=species_id, + population_id=population_id, + job_id=job_id)) diff --git a/uploader/files.py b/uploader/files.py index b163612..d37a53e 100644 --- a/uploader/files.py +++ b/uploader/files.py @@ -1,7 +1,9 @@ """Utilities to deal with uploaded files.""" import hashlib from pathlib import Path +from typing import Iterator from datetime import datetime + from flask import current_app from werkzeug.utils import secure_filename @@ -21,6 +23,27 @@ def save_file(fileobj: FileStorage, upload_dir: Path) -> Path: fileobj.save(filepath) return filepath + def fullpath(filename: str): """Get a file's full path. This makes use of `flask.current_app`.""" return Path(current_app.config["UPLOAD_FOLDER"], filename).absolute() + + +def chunked_binary_read(filepath: Path, chunksize: int = 2048) -> Iterator: + """Read a file in binary mode in chunks.""" + with open(filepath, "rb") as inputfile: + while True: + data = inputfile.read(chunksize) + if data != b"": + yield data + continue + break + + +def sha256_digest_over_file(filepath: Path) -> str: + """Compute the sha256 digest over a file's contents.""" + filehash = hashlib.sha256() + for chunk in chunked_binary_read(filepath): + filehash.update(chunk) + + return filehash.hexdigest() diff --git a/uploader/genotypes/models.py b/uploader/genotypes/models.py index 53c5fb8..44c98b1 100644 --- a/uploader/genotypes/models.py +++ b/uploader/genotypes/models.py @@ -1,8 +1,9 @@ """Functions for handling genotypes.""" from typing import Optional +from datetime import datetime import MySQLdb as mdb -from MySQLdb.cursors import DictCursor +from MySQLdb.cursors import Cursor, DictCursor from uploader.db_utils import debug_query @@ -32,10 +33,69 @@ def genotype_markers( ) -> tuple[dict, ...]: """Retrieve markers from the database.""" _query = "SELECT * FROM Geno WHERE SpeciesId=%s" - if bool(limit) and limit > 0: + if bool(limit) and limit > 0:# type: ignore[operator] _query = _query + f" LIMIT {limit} OFFSET {offset}" with conn.cursor(cursorclass=DictCursor) as cursor: cursor.execute(_query, (species_id,)) debug_query(cursor) return tuple(dict(row) for row in cursor.fetchall()) + + +def genotype_dataset( + conn: mdb.Connection, + species_id: int, + population_id: int, + dataset_id: Optional[int] = None +) -> Optional[dict]: + """Retrieve genotype datasets from the database. + + Apparently, you should only ever have one genotype dataset for a population. + """ + _query = ( + "SELECT gf.* FROM Species AS s INNER JOIN InbredSet AS iset " + "ON s.Id=iset.SpeciesId INNER JOIN GenoFreeze AS gf " + "ON iset.Id=gf.InbredSetId " + "WHERE s.Id=%s AND iset.Id=%s") + _params = (species_id, population_id) + if bool(dataset_id): + _query = _query + " AND gf.Id=%s" + _params = _params + (dataset_id,)# type: ignore[assignment] + + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_query, _params) + debug_query(cursor) + result = cursor.fetchone() + if bool(result): + return dict(result) + return None + + +def save_new_dataset( + cursor: Cursor, + population_id: int, + name: str, + fullname: str, + shortname: str +) -> dict: + """Save a new genotype dataset into the database.""" + params = { + "InbredSetId": population_id, + "Name": name, + "FullName": fullname, + "ShortName": shortname, + "CreateTime": datetime.now().date().isoformat(), + "public": 2, + "confidentiality": 0, + "AuthorisedUsers": None + } + cursor.execute( + "INSERT INTO GenoFreeze(" + "Name, FullName, ShortName, CreateTime, public, InbredSetId, " + "confidentiality, AuthorisedUsers" + ") VALUES (" + "%(Name)s, %(FullName)s, %(ShortName)s, %(CreateTime)s, %(public)s, " + "%(InbredSetId)s, %(confidentiality)s, %(AuthorisedUsers)s" + ")", + params) + return {**params, "Id": cursor.lastrowid} diff --git a/uploader/genotypes/views.py b/uploader/genotypes/views.py index 2ff9965..0821eca 100644 --- a/uploader/genotypes/views.py +++ b/uploader/genotypes/views.py @@ -1,4 +1,5 @@ """Views for the genotypes.""" +from MySQLdb.cursors import DictCursor from flask import (flash, request, url_for, @@ -7,18 +8,25 @@ from flask import (flash, render_template, current_app as app) +from uploader.ui import make_template_renderer +from uploader.oauth2.client import oauth2_post from uploader.authorisation import require_login from uploader.db_utils import database_connection from uploader.species.models import all_species, species_by_id +from uploader.monadic_requests import make_either_error_handler +from uploader.request_checks import with_species, with_population from uploader.datautils import safe_int, order_by_family, enumerate_sequence from uploader.population.models import (populations_by_species, population_by_species_and_id) from .models import (genotype_markers, + genotype_dataset, + save_new_dataset, genotype_markers_count, genocode_by_population) genotypesbp = Blueprint("genotypes", __name__) +render_template = make_template_renderer("genotypes") @genotypesbp.route("populations/genotypes", methods=["GET"]) @require_login @@ -41,14 +49,10 @@ def index(): @genotypesbp.route("/<int:species_id>/populations/genotypes/select-population", methods=["GET"]) @require_login -def select_population(species_id: int): +@with_species(redirect_uri="species.populations.genotypes.index") +def select_population(species: dict, species_id: int): """Select the population under which the genotypes go.""" with database_connection(app.config["SQL_URI"]) as conn: - species = species_by_id(conn, species_id) - if not bool(species): - flash("Invalid species provided!", "alert-danger") - return redirect(url_for("species.populations.genotypes.index")) - if not bool(request.args.get("population_id")): return render_template("genotypes/select-population.html", species=species, @@ -74,54 +78,127 @@ def select_population(species_id: int): "/<int:species_id>/populations/<int:population_id>/genotypes", methods=["GET"]) @require_login -def list_genotypes(species_id: int, population_id: int): +@with_population(species_redirect_uri="species.populations.genotypes.index", + redirect_uri="species.populations.genotypes.select_population") +def list_genotypes(species: dict, population: dict, **kwargs):# pylint: disable=[unused-argument] """List genotype details for species and population.""" with database_connection(app.config["SQL_URI"]) as conn: - species = species_by_id(conn, species_id) - if not bool(species): - flash("Invalid species provided!", "alert-danger") - return redirect(url_for("species.populations.genotypes.index")) - - population = population_by_species_and_id( - conn, species_id, population_id) - if not bool(population): - flash("Invalid population selected!", "alert-danger") - return redirect(url_for( - "species.populations.genotypes.select_population", - species_id=species_id)) - return render_template("genotypes/list-genotypes.html", species=species, population=population, genocode=genocode_by_population( - conn, population_id), + conn, population["Id"]), total_markers=genotype_markers_count( - conn, species_id), + conn, species["SpeciesId"]), + dataset=genotype_dataset(conn, + species["SpeciesId"], + population["Id"]), activelink="list-genotypes") @genotypesbp.route("/<int:species_id>/genotypes/list-markers", methods=["GET"]) @require_login -def list_markers(species_id: int): +@with_species(redirect_uri="species.populations.genotypes.index") +def list_markers(species: dict, **kwargs):# pylint: disable=[unused-argument] """List a species' genetic markers.""" with database_connection(app.config["SQL_URI"]) as conn: - species = species_by_id(conn, species_id) - if not bool(species): - flash("Invalid species provided!", "alert-danger") - return redirect(url_for("species.populations.genotypes.index")) - - start_from = safe_int(request.args.get("start_from") or 0) - if start_from < 0: - start_from = 0 + start_from = max(safe_int(request.args.get("start_from") or 0), 0) count = safe_int(request.args.get("count") or 20) - markers = enumerate_sequence( - genotype_markers(conn, species_id, offset=start_from, limit=count), - start=start_from+1) return render_template("genotypes/list-markers.html", species=species, total_markers=genotype_markers_count( - conn, species_id), + conn, species["SpeciesId"]), start_from=start_from, count=count, - markers=markers, + markers=enumerate_sequence( + genotype_markers(conn, + species["SpeciesId"], + offset=start_from, + limit=count), + start=start_from+1), activelink="list-markers") + +@genotypesbp.route( + "/<int:species_id>/populations/<int:population_id>/genotypes/datasets/" + "<int:dataset_id>/view", + methods=["GET"]) +@require_login +def view_dataset(species_id: int, population_id: int, dataset_id: int): + """View details regarding a specific dataset.""" + with database_connection(app.config["SQL_URI"]) as conn: + species = species_by_id(conn, species_id) + if not bool(species): + flash("Invalid species provided!", "alert-danger") + return redirect(url_for("species.populations.genotypes.index")) + + population = population_by_species_and_id( + conn, species_id, population_id) + if not bool(population): + flash("Invalid population selected!", "alert-danger") + return redirect(url_for( + "species.populations.genotypes.select_population", + species_id=species_id)) + + dataset = genotype_dataset(conn, species_id, population_id, dataset_id) + if not bool(dataset): + flash("Could not find such a dataset!", "alert-danger") + return redirect(url_for( + "species.populations.genotypes.list_genotypes", + species_id=species_id, + population_id=population_id)) + + return render_template("genotypes/view-dataset.html", + species=species, + population=population, + dataset=dataset, + activelink="view-dataset") + + +@genotypesbp.route( + "/<int:species_id>/populations/<int:population_id>/genotypes/datasets/" + "create", + methods=["GET", "POST"]) +@require_login +@with_population(species_redirect_uri="species.populations.genotypes.index", + redirect_uri="species.populations.genotypes.select_population") +def create_dataset(species: dict, population: dict, **kwargs):# pylint: disable=[unused-argument] + """Create a genotype dataset.""" + with (database_connection(app.config["SQL_URI"]) as conn, + conn.cursor(cursorclass=DictCursor) as cursor): + if request.method == "GET": + return render_template("genotypes/create-dataset.html", + species=species, + population=population, + activelink="create-dataset") + + form = request.form + new_dataset = save_new_dataset( + cursor, + population["Id"], + form["geno-dataset-name"], + form["geno-dataset-fullname"], + form["geno-dataset-shortname"]) + + def __success__(_success): + flash("Successfully created genotype dataset.", "alert-success") + return redirect(url_for( + "species.populations.genotypes.list_genotypes", + species_id=species["SpeciesId"], + population_id=population["Id"])) + + return oauth2_post( + "auth/resource/genotypes/create", + json={ + **dict(request.form), + "species_id": species["SpeciesId"], + "population_id": population["Id"], + "dataset_id": new_dataset["Id"], + "dataset_name": form["geno-dataset-name"], + "dataset_fullname": form["geno-dataset-fullname"], + "dataset_shortname": form["geno-dataset-shortname"], + "public": "on" + } + ).either( + make_either_error_handler( + "There was an error creating the genotype dataset."), + __success__) diff --git a/uploader/input_validation.py b/uploader/input_validation.py index 9abe742..627c69e 100644 --- a/uploader/input_validation.py +++ b/uploader/input_validation.py @@ -1,14 +1,19 @@ """Input validation utilities""" +import re +import json +import base64 from typing import Any def is_empty_string(value: str) -> bool: """Check whether as string is empty""" return (isinstance(value, str) and value.strip() == "") + def is_empty_input(value: Any) -> bool: """Check whether user provided an empty value.""" return (value is None or is_empty_string(value)) + def is_integer_input(value: Any) -> bool: """ Check whether user provided a value that can be parsed into an integer. @@ -25,3 +30,42 @@ def is_integer_input(value: Any) -> bool: __is_int__(value, 10) or __is_int__(value, 8) or __is_int__(value, 16)))) + + +def is_valid_representative_name(repr_name: str) -> bool: + """ + Check whether the given representative name is a valid according to our rules. + + Parameters + ---------- + repr_name: a string of characters. + + Checks For + ---------- + * The name MUST start with an alphabet [a-zA-Z] + * The name MUST end with an alphabet [a-zA-Z] or number [0-9] + * The name MUST be composed of alphabets [a-zA-Z], numbers [0-9], + underscores (_) and/or hyphens (-). + + Returns + ------- + Boolean indicating whether or not the name is valid. + """ + pattern = re.compile(r"^[a-zA-Z]+[a-zA-Z0-9_-]*[a-zA-Z0-9]$") + return bool(pattern.match(repr_name)) + + +def encode_errors(errors: tuple[tuple[str, str], ...], form) -> bytes: + """Encode form errors into base64 string.""" + return base64.b64encode( + json.dumps({ + "errors": dict(errors), + "original_formdata": dict(form) + }).encode("utf8")) + + +def decode_errors(errorstr) -> dict[str, dict]: + """Decode errors from base64 string""" + if not bool(errorstr): + return {"errors": {}, "original_formdata": {}} + return json.loads(base64.b64decode(errorstr.encode("utf8")).decode("utf8")) diff --git a/uploader/jobs.py b/uploader/jobs.py index 21889da..4a3fc80 100644 --- a/uploader/jobs.py +++ b/uploader/jobs.py @@ -10,7 +10,7 @@ from typing import Union, Optional from redis import Redis from flask import current_app as app -JOBS_PREFIX = "JOBS" +JOBS_PREFIX = "jobs" class JobNotFound(Exception): """Raised if we try to retrieve a non-existent job.""" diff --git a/uploader/monadic_requests.py b/uploader/monadic_requests.py index aa34951..c492df5 100644 --- a/uploader/monadic_requests.py +++ b/uploader/monadic_requests.py @@ -5,13 +5,12 @@ from typing import Union, Optional, Callable import requests from requests.models import Response from pymonad.either import Left, Right, Either -from flask import ( - flash, - request, - redirect, - render_template, - current_app as app, - escape as flask_escape) +from flask import (flash, + request, + redirect, + render_template, + current_app as app, + escape as flask_escape) # HTML Status codes indicating a successful request. SUCCESS_CODES = (200, 201, 202, 203, 204, 205, 206, 207, 208, 226) @@ -84,3 +83,22 @@ def post(url, data=None, json=None, **kwargs) -> Either: return Left(resp) except requests.exceptions.RequestException as exc: return Left(exc) + + +def make_either_error_handler(msg): + """Make generic error handler for pymonads Either objects.""" + def __fail__(error): + if issubclass(type(error), Exception): + app.logger.debug("\n\n%s (Exception)\n\n", msg, exc_info=True) + raise error + if issubclass(type(error), Response): + try: + _data = error.json() + except Exception as _exc: + raise Exception(error.content) from _exc + raise Exception(_data) + + app.logger.debug("\n\n%s\n\n", msg) + raise Exception(error) + + return __fail__ diff --git a/uploader/oauth2/client.py b/uploader/oauth2/client.py index 70a32ff..e7128de 100644 --- a/uploader/oauth2/client.py +++ b/uploader/oauth2/client.py @@ -191,7 +191,7 @@ def oauth2_get(url, **kwargs) -> Either: return Right(resp.json()) return Left(resp) except Exception as exc:#pylint: disable=[broad-except] - app.logger.error("Error retriving data from auth server: (GET %s)", + app.logger.error("Error retrieving data from auth server: (GET %s)", _uri, exc_info=True) return Left(exc) @@ -223,7 +223,7 @@ def oauth2_post(url, data=None, json=None, **kwargs):#pylint: disable=[redefined return Right(resp.json()) return Left(resp) except Exception as exc:#pylint: disable=[broad-except] - app.logger.error("Error retriving data from auth server: (POST %s)", + app.logger.error("Error retrieving data from auth server: (POST %s)", _uri, exc_info=True) return Left(exc) diff --git a/uploader/phenotypes/__init__.py b/uploader/phenotypes/__init__.py new file mode 100644 index 0000000..c17d32c --- /dev/null +++ b/uploader/phenotypes/__init__.py @@ -0,0 +1,2 @@ +"""Package for handling ('classical') phenotype data""" +from .views import phenotypesbp diff --git a/uploader/phenotypes/models.py b/uploader/phenotypes/models.py new file mode 100644 index 0000000..9324601 --- /dev/null +++ b/uploader/phenotypes/models.py @@ -0,0 +1,232 @@ +"""Database and utility functions for phenotypes.""" +from typing import Optional +from functools import reduce +from datetime import datetime + +import MySQLdb as mdb +from MySQLdb.cursors import Cursor, DictCursor + +from uploader.db_utils import debug_query + +def datasets_by_population( + conn: mdb.Connection, + species_id: int, + population_id: int +) -> tuple[dict, ...]: + """Retrieve all of a population's phenotype studies.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT s.SpeciesId, pf.* FROM Species AS s " + "INNER JOIN InbredSet AS iset ON s.Id=iset.SpeciesId " + "INNER JOIN PublishFreeze AS pf ON iset.Id=pf.InbredSetId " + "WHERE s.Id=%s AND iset.Id=%s;", + (species_id, population_id)) + return tuple(dict(row) for row in cursor.fetchall()) + + +def dataset_by_id(conn: mdb.Connection, + species_id: int, + population_id: int, + dataset_id: int) -> dict: + """Fetch dataset details by identifier""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT s.SpeciesId, pf.* FROM Species AS s " + "INNER JOIN InbredSet AS iset ON s.Id=iset.SpeciesId " + "INNER JOIN PublishFreeze AS pf ON iset.Id=pf.InbredSetId " + "WHERE s.Id=%s AND iset.Id=%s AND pf.Id=%s", + (species_id, population_id, dataset_id)) + return dict(cursor.fetchone()) + + +def phenotypes_count(conn: mdb.Connection, + population_id: int, + dataset_id: int) -> int: + """Count the number of phenotypes in the dataset.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT COUNT(*) AS total_phenos FROM Phenotype AS pheno " + "INNER JOIN PublishXRef AS pxr ON pheno.Id=pxr.PhenotypeId " + "INNER JOIN PublishFreeze AS pf ON pxr.InbredSetId=pf.InbredSetId " + "WHERE pxr.InbredSetId=%s AND pf.Id=%s", + (population_id, dataset_id)) + return int(cursor.fetchone()["total_phenos"]) + + +def dataset_phenotypes(conn: mdb.Connection, + population_id: int, + dataset_id: int, + offset: int = 0, + limit: Optional[int] = None) -> tuple[dict, ...]: + """Fetch the actual phenotypes.""" + _query = ( + "SELECT pheno.*, pxr.Id, ist.InbredSetCode FROM Phenotype AS pheno " + "INNER JOIN PublishXRef AS pxr ON pheno.Id=pxr.PhenotypeId " + "INNER JOIN PublishFreeze AS pf ON pxr.InbredSetId=pf.InbredSetId " + "INNER JOIN InbredSet AS ist ON pf.InbredSetId=ist.Id " + "WHERE pxr.InbredSetId=%s AND pf.Id=%s") + ( + f" LIMIT {limit} OFFSET {offset}" if bool(limit) else "") + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_query, (population_id, dataset_id)) + debug_query(cursor) + return tuple(dict(row) for row in cursor.fetchall()) + + +def __phenotype_se__(cursor: Cursor, + species_id: int, + population_id: int, + dataset_id: int, + xref_id: str) -> dict: + """Fetch standard-error values (if they exist) for a phenotype.""" + _sequery = ( + "SELECT pxr.Id AS xref_id, pxr.DataId, str.Id AS StrainId, pse.error, nst.count " + "FROM Phenotype AS pheno " + "INNER JOIN PublishXRef AS pxr ON pheno.Id=pxr.PhenotypeId " + "INNER JOIN PublishSE AS pse ON pxr.DataId=pse.DataId " + "INNER JOIN NStrain AS nst ON pse.DataId=nst.DataId " + "INNER JOIN Strain AS str ON nst.StrainId=str.Id " + "INNER JOIN StrainXRef AS sxr ON str.Id=sxr.StrainId " + "INNER JOIN PublishFreeze AS pf ON sxr.InbredSetId=pf.InbredSetId " + "INNER JOIN InbredSet AS iset ON pf.InbredSetId=iset.InbredSetId " + "WHERE (str.SpeciesId, pxr.InbredSetId, pf.Id, pxr.Id)=(%s, %s, %s, %s)") + cursor.execute(_sequery, + (species_id, population_id, dataset_id, xref_id)) + return {(row["DataId"], row["StrainId"]): { + "xref_id": row["xref_id"], + "DataId": row["DataId"], + "error": row["error"], + "count": row["count"] + } for row in cursor.fetchall()} + +def __organise_by_phenotype__(pheno, row): + """Organise disparate data rows into phenotype 'objects'.""" + _pheno = pheno.get(row["Id"]) + return { + **pheno, + row["Id"]: { + "Id": row["Id"], + "Pre_publication_description": row["Pre_publication_description"], + "Post_publication_description": row["Post_publication_description"], + "Original_description": row["Original_description"], + "Units": row["Units"], + "Pre_publication_abbreviation": row["Pre_publication_abbreviation"], + "Post_publication_abbreviation": row["Post_publication_abbreviation"], + "xref_id": row["pxr.Id"], + "data": { + **(_pheno["data"] if bool(_pheno) else {}), + (row["DataId"], row["StrainId"]): { + "DataId": row["DataId"], + "mean": row["mean"], + "Locus": row["Locus"], + "LRS": row["LRS"], + "additive": row["additive"], + "Sequence": row["Sequence"], + "comments": row["comments"], + "value": row["value"], + "StrainName": row["Name"], + "StrainName2": row["Name2"], + "StrainSymbol": row["Symbol"], + "StrainAlias": row["Alias"] + } + } + } + } + + +def __merge_pheno_data_and_se__(data, sedata) -> dict: + """Merge phenotype data with the standard errors.""" + return { + key: {**value, **sedata.get(key, {})} + for key, value in data.items() + } + + +def phenotype_by_id( + conn: mdb.Connection, + species_id: int, + population_id: int, + dataset_id: int, + xref_id +) -> Optional[dict]: + """Fetch a specific phenotype.""" + _dataquery = ("SELECT pheno.*, pxr.*, pd.*, str.*, iset.InbredSetCode " + "FROM Phenotype AS pheno " + "INNER JOIN PublishXRef AS pxr ON pheno.Id=pxr.PhenotypeId " + "INNER JOIN PublishData AS pd ON pxr.DataId=pd.Id " + "INNER JOIN Strain AS str ON pd.StrainId=str.Id " + "INNER JOIN StrainXRef AS sxr ON str.Id=sxr.StrainId " + "INNER JOIN PublishFreeze AS pf ON sxr.InbredSetId=pf.InbredSetId " + "INNER JOIN InbredSet AS iset ON pf.InbredSetId=iset.InbredSetId " + "WHERE " + "(str.SpeciesId, pxr.InbredSetId, pf.Id, pxr.Id)=(%s, %s, %s, %s)") + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_dataquery, + (species_id, population_id, dataset_id, xref_id)) + _pheno: dict = reduce(__organise_by_phenotype__, cursor.fetchall(), {}) + if bool(_pheno) and len(_pheno.keys()) == 1: + _pheno = tuple(_pheno.values())[0] + return { + **_pheno, + "data": tuple(__merge_pheno_data_and_se__( + _pheno["data"], + __phenotype_se__(cursor, + species_id, + population_id, + dataset_id, + xref_id)).values()) + } + if bool(_pheno) and len(_pheno.keys()) > 1: + raise Exception( + "We found more than one phenotype with the same identifier!") + + return None + + +def phenotypes_data(conn: mdb.Connection, + population_id: int, + dataset_id: int, + offset: int = 0, + limit: Optional[int] = None) -> tuple[dict, ...]: + """Fetch the data for the phenotypes.""" + # — Phenotype -> PublishXRef -> PublishData -> Strain -> StrainXRef -> PublishFreeze + _query = ("SELECT pheno.*, pxr.*, pd.*, str.*, iset.InbredSetCode " + "FROM Phenotype AS pheno " + "INNER JOIN PublishXRef AS pxr ON pheno.Id=pxr.PhenotypeId " + "INNER JOIN PublishData AS pd ON pxr.DataId=pd.Id " + "INNER JOIN Strain AS str ON pd.StrainId=str.Id " + "INNER JOIN StrainXRef AS sxr ON str.Id=sxr.StrainId " + "INNER JOIN PublishFreeze AS pf ON sxr.InbredSetId=pf.InbredSetId " + "INNER JOIN InbredSet AS iset ON pf.InbredSetId=iset.InbredSetId " + "WHERE pxr.InbredSetId=%s AND pf.Id=%s") + ( + f" LIMIT {limit} OFFSET {offset}" if bool(limit) else "") + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_query, (population_id, dataset_id)) + debug_query(cursor) + return tuple(dict(row) for row in cursor.fetchall()) + + +def save_new_dataset(cursor: Cursor, + population_id: int, + dataset_name: str, + dataset_fullname: str, + dataset_shortname: str) -> dict: + """Create a new phenotype dataset.""" + params = { + "population_id": population_id, + "dataset_name": dataset_name, + "dataset_fullname": dataset_fullname, + "dataset_shortname": dataset_shortname, + "created": datetime.now().date().isoformat(), + "public": 2, + "confidentiality": 0, + "users": None + } + cursor.execute( + "INSERT INTO PublishFreeze(Name, FullName, ShortName, CreateTime, " + "public, InbredSetId, confidentiality, AuthorisedUsers) " + "VALUES(%(dataset_name)s, %(dataset_fullname)s, %(dataset_shortname)s, " + "%(created)s, %(public)s, %(population_id)s, %(confidentiality)s, " + "%(users)s)", + params) + debug_query(cursor) + return {**params, "Id": cursor.lastrowid} diff --git a/uploader/phenotypes/views.py b/uploader/phenotypes/views.py new file mode 100644 index 0000000..02e8078 --- /dev/null +++ b/uploader/phenotypes/views.py @@ -0,0 +1,368 @@ +"""Views handling ('classical') phenotypes.""" +import sys +import uuid +import json +from pathlib import Path +from functools import wraps + +from redis import Redis +from requests.models import Response +from MySQLdb.cursors import DictCursor +from flask import (flash, + request, + url_for, + redirect, + Blueprint, + render_template, + current_app as app) + +# from r_qtl import r_qtl2 as rqtl2 +from r_qtl import r_qtl2_qc as rqc +from r_qtl import exceptions as rqe + +from uploader import jobs +from uploader.files import save_file#, fullpath +from uploader.oauth2.client import oauth2_post +from uploader.authorisation import require_login +from uploader.db_utils import database_connection +from uploader.species.models import all_species, species_by_id +from uploader.monadic_requests import make_either_error_handler +from uploader.request_checks import with_species, with_population +from uploader.datautils import safe_int, order_by_family, enumerate_sequence +from uploader.population.models import (populations_by_species, + population_by_species_and_id) +from uploader.input_validation import (encode_errors, + decode_errors, + is_valid_representative_name) + +from .models import (dataset_by_id, + phenotype_by_id, + phenotypes_count, + save_new_dataset, + dataset_phenotypes, + datasets_by_population) + +phenotypesbp = Blueprint("phenotypes", __name__) + +@phenotypesbp.route("/phenotypes", methods=["GET"]) +@require_login +def index(): + """Direct entry-point for phenotypes data handling.""" + with database_connection(app.config["SQL_URI"]) as conn: + if not bool(request.args.get("species_id")): + return render_template("phenotypes/index.html", + species=order_by_family(all_species(conn)), + activelink="phenotypes") + + species = species_by_id(conn, request.args.get("species_id")) + if not bool(species): + flash("No such species!", "alert-danger") + return redirect(url_for("species.populations.phenotypes.index")) + return redirect(url_for("species.populations.phenotypes.select_population", + species_id=species["SpeciesId"])) + + +@phenotypesbp.route("<int:species_id>/phenotypes/select-population", + methods=["GET"]) +@require_login +@with_species(redirect_uri="species.populations.phenotypes.index") +def select_population(species: dict, **kwargs):# pylint: disable=[unused-argument] + """Select the population for your phenotypes.""" + with database_connection(app.config["SQL_URI"]) as conn: + if not bool(request.args.get("population_id")): + return render_template("phenotypes/select-population.html", + species=species, + populations=order_by_family( + populations_by_species( + conn, species["SpeciesId"]), + order_key="FamilyOrder"), + activelink="phenotypes") + + population = population_by_species_and_id( + conn, species["SpeciesId"], int(request.args["population_id"])) + if not bool(population): + flash("No such population found!", "alert-danger") + return redirect(url_for( + "species.populations.phenotypes.select_population", + species_id=species["SpeciesId"])) + + return redirect(url_for("species.populations.phenotypes.list_datasets", + species_id=species["SpeciesId"], + population_id=population["Id"])) + + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets", + methods=["GET"]) +@require_login +@with_population(species_redirect_uri="species.populations.phenotypes.index", + redirect_uri="species.populations.phenotypes.select_population") +def list_datasets(species: dict, population: dict, **kwargs):# pylint: disable=[unused-argument] + """List available phenotype datasets.""" + with database_connection(app.config["SQL_URI"]) as conn: + return render_template("phenotypes/list-datasets.html", + species=species, + population=population, + datasets=datasets_by_population( + conn, + species["SpeciesId"], + population["Id"]), + activelink="list-datasets") + + +def with_dataset( + species_redirect_uri: str, + population_redirect_uri: str, + redirect_uri: str +): + """Ensure the dataset actually exists.""" + def __decorator__(func): + @wraps(func) + @with_population(species_redirect_uri, population_redirect_uri) + def __with_dataset__(**kwargs): + try: + _spcid = int(kwargs["species_id"]) + _popid = int(kwargs["population_id"]) + _dsetid = int(kwargs.get("dataset_id")) + select_dataset_uri = redirect(url_for( + redirect_uri, species_id=_spcid, population_id=_popid)) + if not bool(_dsetid): + flash("You need to select a valid 'dataset_id' value.", + "alert-danger") + return select_dataset_uri + with database_connection(app.config["SQL_URI"]) as conn: + dataset = dataset_by_id(conn, _spcid, _popid, _dsetid) + if not bool(dataset): + flash("You must select a valid dataset.", + "alert-danger") + return select_dataset_uri + except ValueError as _verr: + app.logger.debug( + "Exception converting 'dataset_id' to integer: %s", + kwargs.get("dataset_id"), + exc_info=True) + flash("Expected 'dataset_id' value to be an integer." + "alert-danger") + return select_dataset_uri + return func(dataset=dataset, **kwargs) + return __with_dataset__ + return __decorator__ + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/view", + methods=["GET"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def view_dataset(# pylint: disable=[unused-argument] + species: dict, population: dict, dataset: dict, **kwargs): + """View a specific dataset""" + with database_connection(app.config["SQL_URI"]) as conn: + dataset = dataset_by_id( + conn, species["SpeciesId"], population["Id"], dataset["Id"]) + if not bool(dataset): + flash("Could not find such a phenotype dataset!", "alert-danger") + return redirect(url_for( + "species.populations.phenotypes.list_datasets", + species_id=species["SpeciesId"], + population_id=population["Id"])) + + start_at = max(safe_int(request.args.get("start_at") or 0), 0) + count = int(request.args.get("count") or 20) + return render_template("phenotypes/view-dataset.html", + species=species, + population=population, + dataset=dataset, + phenotype_count=phenotypes_count( + conn, population["Id"], dataset["Id"]), + phenotypes=enumerate_sequence( + dataset_phenotypes(conn, + population["Id"], + dataset["Id"], + offset=start_at, + limit=count), + start=start_at+1), + start_from=start_at, + count=count, + activelink="view-dataset") + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/phenotype/<xref_id>", + methods=["GET"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def view_phenotype(# pylint: disable=[unused-argument] + species: dict, + population: dict, + dataset: dict, + xref_id: int, + **kwargs +): + """View an individual phenotype from the dataset.""" + def __render__(privileges): + return render_template( + "phenotypes/view-phenotype.html", + species=species, + population=population, + dataset=dataset, + phenotype=phenotype_by_id(conn, + species["SpeciesId"], + population["Id"], + dataset["Id"], + xref_id), + privileges=(privileges + ### For demo! Do not commit this part + + ("group:resource:edit-resource", + "group:resource:delete-resource",) + ### END: For demo! Do not commit this part + ), + activelink="view-phenotype") + + def __fail__(error): + if isinstance(error, Response) and error.json() == "No linked resource!": + return __render__(tuple()) + return make_either_error_handler( + "There was an error fetching the roles and privileges.")(error) + + with database_connection(app.config["SQL_URI"]) as conn: + return oauth2_post( + "/auth/resource/phenotypes/individual/linked-resource", + json={ + "species_id": species["SpeciesId"], + "population_id": population["Id"], + "dataset_id": dataset["Id"], + "xref_id": xref_id + } + ).then( + lambda resource: tuple( + privilege["privilege_id"] for role in resource["roles"] + for privilege in role["privileges"]) + ).then(__render__).either(__fail__, lambda resp: resp) + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets/create", + methods=["GET", "POST"]) +@require_login +@with_population( + species_redirect_uri="species.populations.phenotypes.index", + redirect_uri="species.populations.phenotypes.select_population") +def create_dataset(species: dict, population: dict, **kwargs):# pylint: disable=[unused-argument] + """Create a new phenotype dataset.""" + with (database_connection(app.config["SQL_URI"]) as conn, + conn.cursor(cursorclass=DictCursor) as cursor): + if request.method == "GET": + return render_template("phenotypes/create-dataset.html", + activelink="create-dataset", + species=species, + population=population, + **decode_errors( + request.args.get("error_values", ""))) + + form = request.form + _errors: tuple[tuple[str, str], ...] = tuple() + if not is_valid_representative_name( + (form.get("dataset-name") or "").strip()): + _errors = _errors + (("dataset-name", "Invalid dataset name."),) + + if not bool((form.get("dataset-fullname") or "").strip()): + _errors = _errors + (("dataset-fullname", + "You must provide a value for 'Full Name'."),) + + if bool(_errors) > 0: + return redirect(url_for( + "species.populations.phenotypes.create_dataset", + species_id=species["SpeciesId"], + population_id=population["Id"], + error_values=encode_errors(_errors, form))) + + dataset_shortname = ( + form["dataset-shortname"] or form["dataset-name"]).strip() + _pheno_dataset = save_new_dataset( + cursor, + population["Id"], + form["dataset-name"].strip(), + form["dataset-fullname"].strip(), + dataset_shortname) + return redirect(url_for("species.populations.phenotypes.list_datasets", + species_id=species["SpeciesId"], + population_id=population["Id"])) + + +@phenotypesbp.route( + "<int:species_id>/populations/<int:population_id>/phenotypes/datasets" + "/<int:dataset_id>/add-phenotypes", + methods=["GET", "POST"]) +@require_login +@with_dataset( + species_redirect_uri="species.populations.phenotypes.index", + population_redirect_uri="species.populations.phenotypes.select_population", + redirect_uri="species.populations.phenotypes.list_datasets") +def add_phenotypes(species: dict, population: dict, dataset: dict, **kwargs):# pylint: disable=[unused-argument, too-many-locals] + """Add one or more phenotypes to the dataset.""" + add_phenos_uri = redirect(url_for( + "species.populations.phenotypes.add_phenotypes", + species_id=species["SpeciesId"], + population_id=population["Id"], + dataset_id=dataset["Id"])) + _redisuri = app.config["REDIS_URL"] + _sqluri = app.config["SQL_URI"] + with (Redis.from_url(_redisuri, decode_responses=True) as rconn, + # database_connection(_sqluri) as conn, + # conn.cursor(cursorclass=DictCursor) as cursor + ): + if request.method == "GET": + return render_template("phenotypes/add-phenotypes.html", + species=species, + population=population, + dataset=dataset, + activelink="add-phenotypes") + + try: + ## Handle huge files here... + phenobundle = save_file(request.files["phenotypes-bundle"], + Path(app.config["UPLOAD_FOLDER"])) + rqc.validate_bundle(phenobundle) + except AssertionError as _aerr: + app.logger.debug("File upload error!", exc_info=True) + flash("Expected a zipped bundle of files with phenotypes' " + "information.", + "alert-danger") + return add_phenos_uri + except rqe.RQTLError as rqtlerr: + app.logger.debug("Bundle validation error!", exc_info=True) + flash("R/qtl2 Error: " + " ".join(rqtlerr.args), "alert-danger") + return add_phenos_uri + + _jobid = uuid.uuid4() + _namespace = jobs.jobsnamespace() + _ttl_seconds = app.config["JOBS_TTL_SECONDS"] + _job = jobs.initialise_job( + rconn, + _namespace, + str(_jobid), + [sys.executable, "-m", "scripts.rqtl2.phenotypes_qc", _sqluri, + _redisuri, _namespace, str(_jobid), str(species["SpeciesId"]), + str(population["Id"]), str(dataset["Id"]), "--redisexpiry", + str(_ttl_seconds)], "phenotype_qc", _ttl_seconds, + {"job-metadata": json.dumps({ + "speciesid": species["SpeciesId"], + "populationid": population["Id"], + "datasetid": dataset["Id"], + "bundle": str(phenobundle.absolute())})}) + # jobs.launch_job( + # _job, + # redisuri, + # f"{app.config['UPLOAD_FOLDER']}/job_errors") + + raise NotImplementedError("Please implement this...") diff --git a/uploader/platforms/__init__.py b/uploader/platforms/__init__.py new file mode 100644 index 0000000..8cb89c9 --- /dev/null +++ b/uploader/platforms/__init__.py @@ -0,0 +1,2 @@ +"""Module to handle management of genetic platforms.""" +from .views import platformsbp diff --git a/uploader/platforms/models.py b/uploader/platforms/models.py new file mode 100644 index 0000000..a859371 --- /dev/null +++ b/uploader/platforms/models.py @@ -0,0 +1,95 @@ +"""Handle db interactions for platforms.""" +from typing import Optional + +import MySQLdb as mdb +from MySQLdb.cursors import Cursor, DictCursor + +def platforms_by_species( + conn: mdb.Connection, + speciesid: int, + offset: int = 0, + limit: Optional[int] = None +) -> tuple[dict, ...]: + """Retrieve platforms by the species""" + _query = ("SELECT * FROM GeneChip WHERE SpeciesId=%s " + "ORDER BY GeneChipName ASC") + if bool(limit) and limit > 0:# type: ignore[operator] + _query = f"{_query} LIMIT {limit} OFFSET {offset}" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute(_query, (speciesid,)) + return tuple(dict(row) for row in cursor.fetchall()) + + +def species_platforms_count(conn: mdb.Connection, species_id: int) -> int: + """Get the number of platforms in the database for a particular species.""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute( + "SELECT COUNT(GeneChipName) AS count FROM GeneChip " + "WHERE SpeciesId=%s", + (species_id,)) + return int(cursor.fetchone()["count"]) + + +def platform_by_id(conn: mdb.Connection, platformid: int) -> Optional[dict]: + """Retrieve a platform by its ID""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute("SELECT * FROM GeneChip WHERE Id=%s", + (platformid,)) + result = cursor.fetchone() + if bool(result): + return dict(result) + + return None + + +def platform_by_species_and_id( + conn: mdb.Connection, species_id: int, platformid: int +) -> Optional[dict]: + """Retrieve a platform by its species and ID""" + with conn.cursor(cursorclass=DictCursor) as cursor: + cursor.execute("SELECT * FROM GeneChip WHERE SpeciesId=%s AND Id=%s", + (species_id, platformid)) + result = cursor.fetchone()#pylint: disable=[duplicate-code] + if bool(result): + return dict(result) + + return None + + +def save_new_platform(# pylint: disable=[too-many-arguments] + cursor: Cursor, + species_id: int, + geo_platform: str, + platform_name: str, + platform_shortname: str, + platform_title: str, + go_tree_value: Optional[str] +) -> dict: + """Save a new platform to the database.""" + params = { + "species_id": species_id, + "GeoPlatform": geo_platform, + "GeneChipName": platform_name, + "Name": platform_shortname, + "Title": platform_title, + "GO_tree_value": go_tree_value + } + cursor.execute("SELECT SpeciesId, GeoPlatform FROM GeneChip") + assert (species_id, geo_platform) not in ( + (row["SpeciesId"], row["GeoPlatform"]) for row in cursor.fetchall()) + cursor.execute( + "INSERT INTO " + "GeneChip(SpeciesId, GeneChipName, Name, GeoPlatform, Title, GO_tree_value) " + "VALUES(" + "%(species_id)s, %(GeneChipName)s, %(Name)s, %(GeoPlatform)s, " + "%(Title)s, %(GO_tree_value)s" + ")", + params) + new_id = cursor.lastrowid + cursor.execute("UPDATE GeneChip SET GeneChipId=%s WHERE Id=%s", + (new_id, new_id)) + return { + **params, + "Id": new_id, + "GeneChipId": new_id + } diff --git a/uploader/platforms/views.py b/uploader/platforms/views.py new file mode 100644 index 0000000..2d61b6a --- /dev/null +++ b/uploader/platforms/views.py @@ -0,0 +1,112 @@ +"""The endpoints for the platforms""" +from MySQLdb.cursors import DictCursor +from flask import ( + flash, + request, + url_for, + redirect, + Blueprint, + current_app as app) + +from uploader.ui import make_template_renderer +from uploader.authorisation import require_login +from uploader.db_utils import database_connection +from uploader.species.models import all_species, species_by_id +from uploader.datautils import safe_int, order_by_family, enumerate_sequence + +from .models import (save_new_platform, + platforms_by_species, + species_platforms_count) + +platformsbp = Blueprint("platforms", __name__) +render_template = make_template_renderer("platforms") + +@platformsbp.route("platforms", methods=["GET"]) +@require_login +def index(): + """Entry-point to the platforms feature.""" + with database_connection(app.config["SQL_URI"]) as conn: + if not bool(request.args.get("species_id")): + return render_template( + "platforms/index.html", + species=order_by_family(all_species(conn)), + activelink="platforms") + + species = species_by_id(conn, request.args["species_id"]) + if not bool(species): + flash("No species selected.", "alert-danger") + return redirect(url_for("species.platforms.index")) + + return redirect(url_for("species.platforms.list_platforms", + species_id=species["SpeciesId"])) + + +@platformsbp.route("<int:species_id>/platforms", methods=["GET"]) +@require_login +def list_platforms(species_id: int): + """List all the available genetic sequencing platforms.""" + with database_connection(app.config["SQL_URI"]) as conn: + species = species_by_id(conn, species_id) + if not bool(species): + flash("No species provided.", "alert-danger") + return redirect(url_for("species.platforms.index")) + + start_from = max(safe_int(request.args.get("start_from") or 0), 0) + count = safe_int(request.args.get("count") or 20) + return render_template( + "platforms/list-platforms.html", + species=species, + platforms=enumerate_sequence( + platforms_by_species(conn, + species_id, + offset=start_from, + limit=count), + start=start_from+1), + start_from=start_from, + count=count, + total_platforms=species_platforms_count(conn, species_id), + activelink="list-platforms") + + +@platformsbp.route("<int:species_id>/platforms/create", methods=["GET", "POST"]) +@require_login +def create_platform(species_id: int): + """Create a new genetic sequencing platform.""" + with (database_connection(app.config["SQL_URI"]) as conn, + conn.cursor(cursorclass=DictCursor) as cursor): + species = species_by_id(conn, species_id) + if not bool(species): + flash("No species provided.", "alert-danger") + return redirect(url_for("species.platforms.index")) + + if request.method == "GET": + return render_template( + "platforms/create-platform.html", + species=species, + activelink="create-platform") + + try: + form = request.form + _new_platform = save_new_platform( + cursor, + species_id, + form["geo-platform"], + form["platform-name"], + form["platform-shortname"], + form["platform-title"], + form.get("go-tree-value") or None) + except KeyError as _kerr: + flash(f"Required value for field {_kerr.args[0]} was not provided.", + "alert-danger") + return redirect(url_for("species.platforms.create_platform", + species_id=species_id)) + except AssertionError as _aerr: + flash(f"Platform with GeoPlatform value of '{form['geo-platform']}'" + f" already exists for species '{species['FullName']}'.", + "alert-danger") + return redirect(url_for("species.platforms.create_platform", + species_id=species_id)) + + flash("Platform created successfully", "alert-success") + return redirect(url_for("species.platforms.list_platforms", + species_id=species_id)) diff --git a/uploader/population/models.py b/uploader/population/models.py index c6c77ae..6dcd85e 100644 --- a/uploader/population/models.py +++ b/uploader/population/models.py @@ -46,29 +46,41 @@ def population_genetic_types(conn) -> tuple: def save_population(cursor: mdb.cursors.Cursor, population_details: dict) -> dict: """Save the population details to the db.""" - #TODO: Handle FamilyOrder here + cursor.execute("SELECT DISTINCT(Family), FamilyOrder FROM InbredSet " + "WHERE Family IS NOT NULL AND Family != '' " + "AND FamilyOrder IS NOT NULL " + "ORDER BY FamilyOrder ASC") + _families = { + row["Family"]: int(row["FamilyOrder"]) + for row in cursor.fetchall() + } + params = { + "MenuOrderId": 0, + "InbredSetId": 0, + "public": 2, + **population_details, + "FamilyOrder": _families.get( + population_details["Family"], + max(_families.values())+1) + } cursor.execute( "INSERT INTO InbredSet(" "InbredSetId, InbredSetName, Name, SpeciesId, FullName, " - "public, MappingMethodId, GeneticType, Family, MenuOrderId, " - "InbredSetCode, Description" + "public, MappingMethodId, GeneticType, Family, FamilyOrder," + " MenuOrderId, InbredSetCode, Description" ") " "VALUES (" "%(InbredSetId)s, %(InbredSetName)s, %(Name)s, %(SpeciesId)s, " "%(FullName)s, %(public)s, %(MappingMethodId)s, %(GeneticType)s, " - "%(Family)s, %(MenuOrderId)s, %(InbredSetCode)s, %(Description)s" + "%(Family)s, %(FamilyOrder)s, %(MenuOrderId)s, %(InbredSetCode)s, " + "%(Description)s" ")", - { - "MenuOrderId": 0, - "InbredSetId": 0, - "public": 2, - **population_details - }) + params) new_id = cursor.lastrowid cursor.execute("UPDATE InbredSet SET InbredSetId=%s WHERE Id=%s", (new_id, new_id)) return { - **population_details, + **params, "Id": new_id, "InbredSetId": new_id, "population_id": new_id diff --git a/uploader/expression_data/rqtl2.py b/uploader/population/rqtl2.py index a855699..9968bd6 100644 --- a/uploader/expression_data/rqtl2.py +++ b/uploader/population/rqtl2.py @@ -3,7 +3,6 @@ import sys import json import traceback from pathlib import Path -from datetime import date from uuid import UUID, uuid4 from functools import partial from zipfile import ZipFile, is_zipfile @@ -29,15 +28,14 @@ from r_qtl import r_qtl2 from uploader import jobs from uploader.files import save_file, fullpath -from uploader.dbinsert import species as all_species +from uploader.species.models import all_species from uploader.db_utils import with_db_connection, database_connection from uploader.authorisation import require_login -from uploader.db.platforms import platform_by_id, platforms_by_species +from uploader.platforms.models import platform_by_id, platforms_by_species from uploader.db.averaging import averaging_methods, averaging_method_by_id from uploader.db.tissues import all_tissues, tissue_by_id, create_new_tissue -from uploader.population.models import (save_population, - populations_by_species, +from uploader.population.models import (populations_by_species, population_by_species_and_id) from uploader.species.models import species_by_id from uploader.db.datasets import ( @@ -60,19 +58,21 @@ rqtl2 = Blueprint("rqtl2", __name__) def select_species(): """Select the species.""" if request.method == "GET": - return render_template("rqtl2/index.html", species=with_db_connection(all_species)) + return render_template("expression-data/rqtl2/index.html", + species=with_db_connection(all_species)) species_id = request.form.get("species_id") species = with_db_connection( lambda conn: species_by_id(conn, species_id)) if bool(species): return redirect(url_for( - "expression-data.rqtl2.select_population", species_id=species_id)) + "species.populations.expression-data.rqtl2.select_population", + species_id=species_id)) flash("Invalid species or no species selected!", "alert-error error-rqtl2") return redirect(url_for("expression-data.rqtl2.select_species")) -@rqtl2.route("/upload/species/<int:species_id>/select-population", +@rqtl2.route("<int:species_id>/expression-data/rqtl2/select-population", methods=["GET", "POST"]) @require_login def select_population(species_id: int): @@ -85,7 +85,7 @@ def select_population(species_id: int): if request.method == "GET": return render_template( - "rqtl2/select-population.html", + "expression-data/rqtl2/select-population.html", species=species, populations=populations_by_species(conn, species_id)) @@ -102,44 +102,6 @@ def select_population(species_id: int): population_id=population["InbredSetId"])) -@rqtl2.route("/upload/species/<int:species_id>/create-population", - methods=["POST"]) -@require_login -def create_population(species_id: int): - """Create a new population for the given species.""" - population_page = redirect(url_for("expression-data.rqtl2.select_population", - species_id=species_id)) - with database_connection(app.config["SQL_URI"]) as conn: - species = species_by_id(conn, species_id) - population_name = request.form.get("inbredset_name", "").strip() - population_fullname = request.form.get("inbredset_fullname", "").strip() - if not bool(species): - flash("Invalid species!", "alert-error error-rqtl2") - return redirect(url_for("expression-data.rqtl2.select_species")) - if not bool(population_name): - flash("Invalid Population Name!", "alert-error error-rqtl2") - return population_page - if not bool(population_fullname): - flash("Invalid Population Full Name!", "alert-error error-rqtl2") - return population_page - new_population = save_population(conn, { - "SpeciesId": species["SpeciesId"], - "Name": population_name, - "InbredSetName": population_fullname, - "FullName": population_fullname, - "Family": request.form.get("inbredset_family") or None, - "Description": request.form.get("description") or None - }) - - flash("Population created successfully.", "alert-success") - return redirect( - url_for("expression-data.rqtl2.upload_rqtl2_bundle", - species_id=species_id, - population_id=new_population["population_id"], - pgsrc="create-population"), - code=307) - - class __RequestError__(Exception): #pylint: disable=[invalid-name] """Internal class to avoid pylint's `too-many-return-statements` error.""" @@ -165,9 +127,10 @@ def upload_rqtl2_bundle(species_id: int, population_id: int): if request.method == "GET" or ( request.method == "POST" and bool(request.args.get("pgsrc"))): - return render_template("rqtl2/upload-rqtl2-bundle-step-01.html", - species=species, - population=population) + return render_template( + "expression-data/rqtl2/upload-rqtl2-bundle-step-01.html", + species=species, + population=population) try: app.logger.debug("Files in the form: %s", request.files) @@ -243,7 +206,7 @@ def chunks_directory(uniqueidentifier: str) -> Path: return Path(app.config["UPLOAD_FOLDER"], f"tempdir_{uniqueidentifier}") -@rqtl2.route(("/upload/species/<int:species_id>/population/<int:population_id>" +@rqtl2.route(("<int:species_id>/populations/<int:population_id>/rqtl2/" "/rqtl2-bundle-chunked"), methods=["GET"]) @require_login @@ -288,7 +251,7 @@ def __merge_chunks__(targetfile: Path, chunkpaths: tuple[Path, ...]) -> Path: return targetfile -@rqtl2.route(("/upload/species/<int:species_id>/population/<int:population_id>" +@rqtl2.route(("<int:species_id>/population/<int:population_id>/rqtl2/upload/" "/rqtl2-bundle-chunked"), methods=["POST"]) @require_login @@ -362,24 +325,25 @@ def rqtl2_bundle_qc_status(jobid: UUID): if bool(messagelistname) else []) jobstatus = thejob["status"] if jobstatus == "error": - return render_template("rqtl2/rqtl2-qc-job-error.html", - job=thejob, - errorsgeneric=json.loads( - thejob.get("errors-generic", "[]")), - errorsgeno=json.loads( - thejob.get("errors-geno", "[]")), - errorspheno=json.loads( - thejob.get("errors-pheno", "[]")), - errorsphenose=json.loads( - thejob.get("errors-phenose", "[]")), - errorsphenocovar=json.loads( - thejob.get("errors-phenocovar", "[]")), - messages=logmessages) + return render_template( + "expression-data/rqtl2/rqtl2-qc-job-error.html", + job=thejob, + errorsgeneric=json.loads( + thejob.get("errors-generic", "[]")), + errorsgeno=json.loads( + thejob.get("errors-geno", "[]")), + errorspheno=json.loads( + thejob.get("errors-pheno", "[]")), + errorsphenose=json.loads( + thejob.get("errors-phenose", "[]")), + errorsphenocovar=json.loads( + thejob.get("errors-phenocovar", "[]")), + messages=logmessages) if jobstatus == "success": jobmeta = json.loads(thejob["job-metadata"]) species = species_by_id(dbconn, jobmeta["speciesid"]) return render_template( - "rqtl2/rqtl2-qc-job-results.html", + "expression-data/rqtl2/rqtl2-qc-job-results.html", species=species, population=population_by_species_and_id( dbconn, species["SpeciesId"], jobmeta["populationid"]), @@ -398,14 +362,14 @@ def rqtl2_bundle_qc_status(jobid: UUID): return None return render_template( - "rqtl2/rqtl2-qc-job-status.html", + "expression-data/rqtl2/rqtl2-qc-job-status.html", job=thejob, geno_percent=compute_percentage(thejob, "geno"), pheno_percent=compute_percentage(thejob, "pheno"), phenose_percent=compute_percentage(thejob, "phenose"), messages=logmessages) except jobs.JobNotFound: - return render_template("rqtl2/no-such-job.html", jobid=jobid) + return render_template("expression-data/rqtl2/no-such-job.html", jobid=jobid) def redirect_on_error(flaskroute, **kwargs): @@ -609,76 +573,6 @@ def select_geno_dataset(species_id: int, population_id: int): @rqtl2.route(("/upload/species/<int:species_id>/population/<int:population_id>" - "/rqtl2-bundle/create-geno-dataset"), - methods=["POST"]) -@require_login -def create_geno_dataset(species_id: int, population_id: int): - """Create a new geno dataset.""" - with database_connection(app.config["SQL_URI"]) as conn: - def __thunk__(): - sgeno_page = redirect(url_for("expression-data.rqtl2.select_dataset_info", - species_id=species_id, - population_id=population_id, - pgsrc="error"), - code=307) - errorclasses = "alert-error error-rqtl2 error-rqtl2-create-geno-dataset" - if not bool(request.form.get("dataset-name")): - flash("You must provide the dataset name", errorclasses) - return sgeno_page - if not bool(request.form.get("dataset-fullname")): - flash("You must provide the dataset full name", errorclasses) - return sgeno_page - public = 2 if request.form.get("dataset-public") == "on" else 0 - - with conn.cursor(cursorclass=DictCursor) as cursor: - datasetname = request.form["dataset-name"] - new_dataset = { - "name": datasetname, - "fname": request.form.get("dataset-fullname"), - "sname": request.form.get("dataset-shortname") or datasetname, - "today": date.today().isoformat(), - "pub": public, - "isetid": population_id - } - cursor.execute("SELECT * FROM GenoFreeze WHERE Name=%s", - (datasetname,)) - results = cursor.fetchall() - if bool(results): - flash( - f"A genotype dataset with name '{escape(datasetname)}' " - "already exists.", - errorclasses) - return redirect(url_for("expression-data.rqtl2.select_dataset_info", - species_id=species_id, - population_id=population_id, - pgsrc="error"), - code=307) - cursor.execute( - "INSERT INTO GenoFreeze(" - "Name, FullName, ShortName, CreateTime, public, InbredSetId" - ") " - "VALUES(" - "%(name)s, %(fname)s, %(sname)s, %(today)s, %(pub)s, %(isetid)s" - ")", - new_dataset) - flash("Created dataset successfully.", "alert-success") - return render_template( - "rqtl2/create-geno-dataset-success.html", - species=species_by_id(conn, species_id), - population=population_by_species_and_id( - conn, species_id, population_id), - rqtl2_bundle_file=request.form["rqtl2_bundle_file"], - geno_dataset={**new_dataset, "id": cursor.lastrowid}) - - return with_errors(__thunk__, - partial(check_species, conn=conn), - partial(check_population, conn=conn, species_id=species_id), - partial(check_r_qtl2_bundle, - species_id=species_id, - population_id=population_id)) - - -@rqtl2.route(("/upload/species/<int:species_id>/population/<int:population_id>" "/rqtl2-bundle/select-tissue"), methods=["POST"]) @require_login @@ -739,7 +633,7 @@ def create_tissue(species_id: int, population_id: int): tissue = create_new_tissue(conn, tissuename, tissueshortname) flash("Tissue created successfully!", "alert-success") return render_template( - "rqtl2/create-tissue-success.html", + "expression-data/rqtl2/create-tissue-success.html", species=species_by_id(conn, species_id), population=population_by_species_and_id( conn, species_id, population_id), @@ -869,7 +763,7 @@ def create_probeset_study(species_id: int, population_id: int): errorclasses) return dataset_info_page return render_template( - "rqtl2/create-probe-study-success.html", + "expression-data/rqtl2/create-probe-study-success.html", species=species_by_id(conn, species_id), population=population_by_species_and_id( conn, species_id, population_id), @@ -954,7 +848,7 @@ def create_probeset_dataset(species_id: int, population_id: int):#pylint: disabl errorclasses) return summary_page return render_template( - "rqtl2/create-probe-dataset-success.html", + "expression-data/rqtl2/create-probe-dataset-success.html", species=species_by_id(conn, species_id), population=population_by_species_and_id( conn, species_id, population_id), @@ -1009,7 +903,7 @@ def select_dataset_info(species_id: int, population_id: int): conn,form.get("geno-dataset-id", "").strip()) if "geno" in cdata and not bool(form.get("geno-dataset-id")): return render_template( - "rqtl2/select-geno-dataset.html", + "expression-data/rqtl2/select-geno-dataset.html", species=species, population=population, rqtl2_bundle_file=thefile.name, @@ -1019,7 +913,7 @@ def select_dataset_info(species_id: int, population_id: int): tissue = tissue_by_id(conn, form.get("tissueid", "").strip()) if "pheno" in cdata and not bool(tissue): return render_template( - "rqtl2/select-tissue.html", + "expression-data/rqtl2/select-tissue.html", species=species, population=population, rqtl2_bundle_file=thefile.name, @@ -1033,7 +927,7 @@ def select_dataset_info(species_id: int, population_id: int): conn, form.get("probe-study-id", "").strip()) if "pheno" in cdata and not bool(probeset_study): return render_template( - "rqtl2/select-probeset-study-id.html", + "expression-data/rqtl2/select-probeset-study-id.html", species=species, population=population, rqtl2_bundle_file=thefile.name, @@ -1049,7 +943,7 @@ def select_dataset_info(species_id: int, population_id: int): conn, form.get("probe-dataset-id", "").strip()) if "pheno" in cdata and not bool(probeset_dataset): return render_template( - "rqtl2/select-probeset-dataset.html", + "expression-data/rqtl2/select-probeset-dataset.html", species=species, population=population, rqtl2_bundle_file=thefile.name, @@ -1060,7 +954,7 @@ def select_dataset_info(species_id: int, population_id: int): conn, int(form["probe-study-id"])), avgmethods=averaging_methods(conn)) - return render_template("rqtl2/summary-info.html", + return render_template("expression-data/rqtl2/summary-info.html", species=species, population=population, rqtl2_bundle_file=thefile.name, @@ -1163,13 +1057,19 @@ def rqtl2_processing_status(jobid: UUID): if thejob["status"] == "error": return render_template( - "rqtl2/rqtl2-job-error.html", job=thejob, messages=logmessages) + "expression-data/rqtl2/rqtl2-job-error.html", + job=thejob, + messages=logmessages) if thejob["status"] == "success": - return render_template("rqtl2/rqtl2-job-results.html", - job=thejob, - messages=logmessages) + return render_template( + "expression-data/rqtl2/rqtl2-job-results.html", + job=thejob, + messages=logmessages) return render_template( - "rqtl2/rqtl2-job-status.html", job=thejob, messages=logmessages) + "expression-data/rqtl2/rqtl2-job-status.html", + job=thejob, + messages=logmessages) except jobs.JobNotFound as _exc: - return render_template("rqtl2/no-such-job.html", jobid=jobid) + return render_template("expression-data/rqtl2/no-such-job.html", + jobid=jobid) diff --git a/uploader/population/views.py b/uploader/population/views.py index 39a5762..36201ba 100644 --- a/uploader/population/views.py +++ b/uploader/population/views.py @@ -1,10 +1,7 @@ """Views dealing with populations/inbredsets""" -import re import json import base64 -import traceback -from requests.models import Response from MySQLdb.cursors import DictCursor from flask import (flash, request, @@ -20,6 +17,10 @@ from uploader.authorisation import require_login from uploader.genotypes.views import genotypesbp from uploader.db_utils import database_connection from uploader.datautils import enumerate_sequence +from uploader.phenotypes.views import phenotypesbp +from uploader.expression_data.views import exprdatabp +from uploader.monadic_requests import make_either_error_handler +from uploader.input_validation import is_valid_representative_name from uploader.species.models import (all_species, species_by_id, order_species_by_family) @@ -34,6 +35,8 @@ __active_link__ = "populations" popbp = Blueprint("populations", __name__) popbp.register_blueprint(samplesbp, url_prefix="/") popbp.register_blueprint(genotypesbp, url_prefix="/") +popbp.register_blueprint(phenotypesbp, url_prefix="/") +popbp.register_blueprint(exprdatabp, url_prefix="/") render_template = make_template_renderer("populations") @@ -70,29 +73,6 @@ def list_species_populations(species_id: int): activelink="list-populations") -def valid_population_name(population_name: str) -> bool: - """ - Check whether the given name is a valid population name. - - Parameters - ---------- - population_name: a string of characters. - - Checks For - ---------- - * The name MUST start with an alphabet [a-zA-Z] - * The name MUST end with an alphabet [a-zA-Z] or number [0-9] - * The name MUST be composed of alphabets [a-zA-Z], numbers [0-9], - underscores (_) and/or hyphens (-). - - Returns - ------- - Boolean indicating whether or not the name is valid. - """ - pattern = re.compile(r"^[a-zA-Z]+[a-zA-Z0-9_-]*[a-zA-Z0-9]$") - return bool(pattern.match(population_name)) - - @popbp.route("/<int:species_id>/populations/create", methods=["GET", "POST"]) @require_login def create_population(species_id: int): @@ -136,7 +116,7 @@ def create_population(species_id: int): errors = errors + (("population_name", "You must provide a name for the population!"),) - if not valid_population_name(population_name): + if not is_valid_representative_name(population_name): errors = errors + (( "population_name", "The population name can only contain letters, numbers, " @@ -170,22 +150,6 @@ def create_population(species_id: int): "GeneticType": request.form.get("population_genetic_type") or None }) - def __handle_error__(error): - error_format = ( - "\n\nThere was an error creating the population:\n\t%s\n\n") - if issubclass(type(error), Exception): - app.logger.debug(error_format, traceback.format_exc()) - raise error - if issubclass(type(error), Response): - try: - _data = error.json() - except Exception as _exc: - raise Exception(error.content) from _exc - raise Exception(_data) - - app.logger.debug(error_format, error) - raise Exception(error) - def __flash_success__(_success): flash("Successfully created resource.", "alert-success") return redirect(url_for( @@ -202,7 +166,10 @@ def create_population(species_id: int): "population_id": new_population["Id"], "public": "on" } - ).either(__handle_error__, __flash_success__) + ).either( + make_either_error_handler( + "There was an error creating the population"), + __flash_success__) @popbp.route("/<int:species_id>/populations/<int:population_id>", diff --git a/uploader/request_checks.py b/uploader/request_checks.py new file mode 100644 index 0000000..a24b2f7 --- /dev/null +++ b/uploader/request_checks.py @@ -0,0 +1,75 @@ +"""Functions to perform common checks. + +These are useful for reusability, and hence maintainability of the code. +""" +from functools import wraps + +from flask import flash, url_for, redirect, current_app as app + +from uploader.species.models import species_by_id +from uploader.db_utils import database_connection +from uploader.population.models import population_by_species_and_id + +def with_species(redirect_uri: str): + """Ensure the species actually exists.""" + def __decorator__(function): + @wraps(function) + def __with_species__(**kwargs): + try: + species_id = int(kwargs.get("species_id")) + if not bool(species_id): + flash("Expected species_id value to be present!", + "alert-danger") + return redirect(url_for(redirect_uri)) + with database_connection(app.config["SQL_URI"]) as conn: + species = species_by_id(conn, species_id) + if not bool(species): + flash("Could not find species with that ID", + "alert-danger") + return redirect(url_for(redirect_uri)) + except ValueError as _verr: + app.logger.debug( + "Exception converting value to integer: %s", + kwargs.get("species_id"), + exc_info=True) + flash("Expected an integer for 'species_id' value.", + "alert-danger") + return redirect(url_for(redirect_uri)) + return function(**{**kwargs, "species": species}) + return __with_species__ + return __decorator__ + + +def with_population(species_redirect_uri: str, redirect_uri: str): + """Ensure the population actually exists.""" + def __decorator__(function): + @wraps(function) + @with_species(redirect_uri=species_redirect_uri) + def __with_population__(**kwargs): + try: + species_id = int(kwargs["species_id"]) + population_id = int(kwargs.get("population_id")) + select_population_uri = redirect(url_for( + redirect_uri, species_id=species_id)) + if not bool(population_id): + flash("Expected population_id value to be present!", + "alert-danger") + return select_population_uri + with database_connection(app.config["SQL_URI"]) as conn: + population = population_by_species_and_id( + conn, species_id, population_id) + if not bool(population): + flash("Could not find population with that ID", + "alert-danger") + return select_population_uri + except ValueError as _verr: + app.logger.debug( + "Exception converting value to integer: %s", + kwargs.get("population_id"), + exc_info=True) + flash("Expected an integer for 'population_id' value.", + "alert-danger") + return select_population_uri + return function(**{**kwargs, "population": population}) + return __with_population__ + return __decorator__ diff --git a/uploader/samples/__init__.py b/uploader/samples/__init__.py new file mode 100644 index 0000000..1bd6d2d --- /dev/null +++ b/uploader/samples/__init__.py @@ -0,0 +1 @@ +"""Samples package. Handle samples uploads and editing.""" diff --git a/uploader/samples/views.py b/uploader/samples/views.py index fd3c601..ed79101 100644 --- a/uploader/samples/views.py +++ b/uploader/samples/views.py @@ -3,40 +3,34 @@ import os import sys import uuid from pathlib import Path -from typing import Iterator -import MySQLdb as mdb from redis import Redis -from MySQLdb.cursors import DictCursor -from flask import ( - flash, - request, - url_for, - redirect, - Blueprint, - render_template, - current_app as app) +from flask import (flash, + request, + url_for, + redirect, + Blueprint, + current_app as app) from uploader import jobs from uploader.files import save_file +from uploader.ui import make_template_renderer from uploader.authorisation import require_login +from uploader.request_checks import with_population from uploader.input_validation import is_integer_input -from uploader.datautils import order_by_family, enumerate_sequence -from uploader.db_utils import ( - with_db_connection, - database_connection, - with_redis_connection) +from uploader.datautils import safe_int, order_by_family, enumerate_sequence +from uploader.population.models import population_by_id, populations_by_species +from uploader.db_utils import (with_db_connection, + database_connection, + with_redis_connection) from uploader.species.models import (all_species, species_by_id, order_species_by_family) -from uploader.population.models import(save_population, - population_by_id, - populations_by_species, - population_by_species_and_id) from .models import samples_by_species_and_population samplesbp = Blueprint("samples", __name__) +render_template = make_template_renderer("samples") @samplesbp.route("/samples", methods=["GET"]) @require_login @@ -110,9 +104,7 @@ def list_samples(species_id: int, population_id: int): all_samples = enumerate_sequence(samples_by_species_and_population( conn, species_id, population_id)) total_samples = len(all_samples) - offset = int(request.args.get("from") or 0) - if offset < 0: - offset = 0 + offset = max(safe_int(request.args.get("from") or 0), 0) count = int(request.args.get("count") or 20) return render_template("samples/list-samples.html", species=species, @@ -233,53 +225,41 @@ def upload_samples(species_id: int, population_id: int):#pylint: disable=[too-ma "upload-samples/status/<uuid:job_id>", methods=["GET"]) @require_login -def upload_status(species_id: int, population_id: int, job_id: uuid.UUID): +@with_population(species_redirect_uri="species.populations.samples.index", + redirect_uri="species.populations.samples.select_population") +def upload_status(species: dict, population: dict, job_id: uuid.UUID, **kwargs):# pylint: disable=[unused-argument] """Check on the status of a samples upload job.""" - with database_connection(app.config["SQL_URI"]) as conn: - species = species_by_id(conn, species_id) - if not bool(species): - flash("You must provide a valid species.", "alert-danger") - return redirect(url_for("species.populations.samples.index")) + job = with_redis_connection(lambda rconn: jobs.job( + rconn, jobs.jobsnamespace(), job_id)) + if job: + status = job["status"] + if status == "success": + return render_template("samples/upload-success.html", + job=job, + species=species, + population=population,) - population = population_by_species_and_id( - conn, species_id, population_id) - if not bool(population): - flash("You must provide a valid population.", "alert-danger") + if status == "error": return redirect(url_for( - "species.populations.samples.select_population", - species_id=species_id)) + "species.populations.samples.upload_failure", job_id=job_id)) - job = with_redis_connection(lambda rconn: jobs.job( - rconn, jobs.jobsnamespace(), job_id)) - if job: - status = job["status"] - if status == "success": - return render_template("samples/upload-success.html", - job=job, - species=species, - population=population,) - - if status == "error": + error_filename = Path(jobs.error_filename( + job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors")) + if error_filename.exists(): + stat = os.stat(error_filename) + if stat.st_size > 0: return redirect(url_for( - "species.populations.samples.upload_failure", job_id=job_id)) + "samples.upload_failure", job_id=job_id)) - error_filename = Path(jobs.error_filename( - job_id, f"{app.config['UPLOAD_FOLDER']}/job_errors")) - if error_filename.exists(): - stat = os.stat(error_filename) - if stat.st_size > 0: - return redirect(url_for( - "samples.upload_failure", job_id=job_id)) - - return render_template("samples/upload-progress.html", - species=species, - population=population, - job=job) # maybe also handle this? - - return render_template("no_such_job.html", - job_id=job_id, + return render_template("samples/upload-progress.html", species=species, - population=population), 400 + population=population, + job=job) # maybe also handle this? + + return render_template("no_such_job.html", + job_id=job_id, + species=species, + population=population), 400 @samplesbp.route("/upload/failure/<uuid:job_id>", methods=["GET"]) @require_login diff --git a/uploader/species/views.py b/uploader/species/views.py index f478505..10715a5 100644 --- a/uploader/species/views.py +++ b/uploader/species/views.py @@ -8,6 +8,7 @@ from flask import (flash, current_app as app) from uploader.population import popbp +from uploader.platforms import platformsbp from uploader.ui import make_template_renderer from uploader.db_utils import database_connection from uploader.oauth2.client import oauth2_get, oauth2_post @@ -23,6 +24,7 @@ from .models import (all_species, speciesbp = Blueprint("species", __name__) speciesbp.register_blueprint(popbp, url_prefix="/") +speciesbp.register_blueprint(platformsbp, url_prefix="/") render_template = make_template_renderer("species") diff --git a/uploader/static/css/styles.css b/uploader/static/css/styles.css index 30d5808..f482c1b 100644 --- a/uploader/static/css/styles.css +++ b/uploader/static/css/styles.css @@ -39,6 +39,10 @@ body { border-color: #FFFFFF; vertical-align: middle; margin: 0.2em; + border-style: solid; + border-width: 2px; + border-radius: 0.5em; + text-align: center; } #header .header-nav a { @@ -102,7 +106,7 @@ dd { padding-bottom: 1em; } -input[type="submit"] { +input[type="submit"], .btn { text-transform: capitalize; } @@ -114,3 +118,44 @@ input[type="submit"] { border-color: #AAAAAA; padding: 0.5em; } + +.activemenu { + border-style: solid; + border-radius: 0.5em; + border-color: #AAAAAA; + background-color: #EFEFEF; +} + +.danger { + color: #A94442; + border-color: #DCA7A7; + background-color: #F2DEDE; +} + +.heading { + border-bottom: solid #EEBB88; +} + +.subheading { + padding: 1em 0 0.1em 0.5em; + border-bottom: solid #88BBEE; +} + +form { + margin-top: 0.3em; + background: #E5E5FF; + padding: 0.5em; + border-radius:0.5em; +} + +form .form-control { + background-color: #EAEAFF; +} + +.sidebar-content .card .card-title { + font-size: 1.5em; +} + +.sidebar-content .card-text table tbody td:nth-child(1) { + font-weight: bolder; +} diff --git a/uploader/static/js/misc.js b/uploader/static/js/misc.js new file mode 100644 index 0000000..cf7b39e --- /dev/null +++ b/uploader/static/js/misc.js @@ -0,0 +1,6 @@ +"Miscellaneous functions and event-handlers" + +$(".not-implemented").click((event) => { + event.preventDefault(); + alert("This feature is not implemented yet. Please bear with us."); +}); diff --git a/uploader/templates/base.html b/uploader/templates/base.html index 886f503..019aa39 100644 --- a/uploader/templates/base.html +++ b/uploader/templates/base.html @@ -28,7 +28,7 @@ <span class="header col-lg-9">GeneNetwork Data Quality Control and Upload</span> <nav class="header-nav col-lg-3"> <ul class="nav justify-content-end"> - <li class="btn"> + <li> {%if user_logged_in()%} <a href="{{url_for('oauth2.logout')}}" title="Log out of the system">{{user_email()}} — Log Out</a> @@ -43,23 +43,49 @@ <aside id="nav-sidebar" class="container-fluid"> <ul class="nav flex-column"> - <li><a href="/" >Home</a></li> - <li><a href="{{url_for('species.list_species')}}" - title="View and manage species information.">Species</a></li> - <li><a href="{{url_for('species.populations.index')}}" - title="View and manage species populations.">Populations</a></li> - <li><a href="{{url_for('species.populations.samples.index')}}" - title="Upload population samples.">Samples</a></li> - <li><a href="{{url_for('species.populations.genotypes.index')}}" - title="Upload Genotype data.">Genotype Data</a></li> - <li><a href="{{url_for('expression-data.index.index')}}" - title="Upload expression data.">Expression Data</a></li> - <li><a href="#" - title="Upload phenotype data.">Phenotype Data</a></li> - <li><a href="#" - title="Upload individual data.">Individual Data</a></li> - <li><a href="#" - title="Upload RNA-Seq data.">RNA-Seq Data</a></li> + <li {%if activemenu=="home"%}class="activemenu"{%endif%}> + <a href="/" >Home</a></li> + <li {%if activemenu=="species"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.list_species')}}" + title="View and manage species information.">Species</a></li> + <li {%if activemenu=="platforms"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.platforms.index')}}" + title="View and manage species platforms.">Sequencing Platforms</a></li> + <li {%if activemenu=="populations"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.populations.index')}}" + title="View and manage species populations.">Populations</a></li> + <li {%if activemenu=="samples"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.populations.samples.index')}}" + title="Upload population samples.">Samples</a></li> + <li {%if activemenu=="genotypes"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.populations.genotypes.index')}}" + title="Upload Genotype data.">Genotype Data</a></li> + <!-- + TODO: Maybe include menus here for managing studies and dataset or + maybe have the studies/datasets managed under their respective + sections, e.g. "Publish*" studies/datasets under the "Phenotypes" + section, "ProbeSet*" studies/datasets under the "Expression Data" + sections, etc. + --> + <li {%if activemenu=="phenotypes"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.populations.phenotypes.index')}}" + title="Upload phenotype data.">Phenotype Data</a></li> + <li {%if activemenu=="expression-data"%}class="activemenu"{%endif%}> + <a href="{{url_for('species.populations.expression-data.index')}}" + title="Upload expression data.">Expression Data</a></li> + <li {%if activemenu=="individuals"%}class="activemenu"{%endif%}> + <a href="#" + class="not-implemented" + title="Upload individual data.">Individual Data</a></li> + <li {%if activemenu=="rna-seq"%}class="activemenu"{%endif%}> + <a href="#" + class="not-implemented" + title="Upload RNA-Seq data.">RNA-Seq Data</a></li> + <li {%if activemenu=="async-jobs"%}class="activemenu"{%endif%}> + <a href="#" + class="not-implemented" + title="View and manage the backgroud jobs you have running"> + Background Jobs</a></li> </ul> </aside> @@ -98,6 +124,7 @@ filename='jquery.min.js')}}"></script> <script src="{{url_for('base.bootstrap', filename='js/bootstrap.min.js')}}"></script> + <script type="text/javascript" src="/static/js/misc.js"></script> {%block javascript%}{%endblock%} </body> diff --git a/uploader/templates/expression-data/base.html b/uploader/templates/expression-data/base.html new file mode 100644 index 0000000..d63fd7e --- /dev/null +++ b/uploader/templates/expression-data/base.html @@ -0,0 +1,13 @@ +{%extends "populations/base.html"%} + +{%block lvl3_breadcrumbs%} +<li {%if activelink=="expression-data"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.expression-data.index')}}"> + Expression Data</a> +</li> +{%block lvl4_breadcrumbs%}{%endblock%} +{%endblock%} diff --git a/uploader/templates/data_review.html b/uploader/templates/expression-data/data-review.html index 4e5c586..c985b03 100644 --- a/uploader/templates/data_review.html +++ b/uploader/templates/expression-data/data-review.html @@ -26,7 +26,7 @@ <small class="text-muted"> If you encounter an error saying your sample(s)/case(s) do not exist in the GeneNetwork database, then you will have to use the - <a href="{{url_for('expression-data.samples.select_species')}}" + <a href="{{url_for('species.populations.samples.index')}}" title="Upload samples/cases feature">Upload Samples/Cases</a> option on this system to upload them. </small> @@ -70,8 +70,8 @@ column</li> <li>The values of each field <strong>ARE NOT</strong> quoted.</li> <li>Here is an - <a href="https://gitlab.com/fredmanglis/gnqc_py/-/blob/main/tests/test_data/no_data_errors.tsv"> - example file</a> with a single data row.</li> + <a href="https://gitlab.com/fredmanglis/gnqc_py/-/blob/main/tests/test_data/no_data_errors.tsv" + target="_blank">example file</a> with a single data row.</li> </ul> </li> <li>.txt files: Content has the same format as .tsv file above</li> diff --git a/uploader/templates/expression-data/index.html b/uploader/templates/expression-data/index.html index ed5d8dd..9ba3582 100644 --- a/uploader/templates/expression-data/index.html +++ b/uploader/templates/expression-data/index.html @@ -1,5 +1,6 @@ -{%extends "base.html"%} +{%extends "expression-data/base.html"%} {%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-select-species.html" import select_species_form%} {%block title%}Expression Data{%endblock%} @@ -10,86 +11,23 @@ <a href="{{url_for('base.index')}}">Home</a> </li> <li class="breadcrumb-item active"> - <a href="{{url_for('expression-data.index.index')}}">Expression Data</a> + <a href="{{url_for('species.populations.expression-data.index')}}" + title="Upload expression data."> + Expression Data</a> </li> {%endblock%} {%block contents%} <div class="row"> - {{flash_all_messages()}} - - <h1 class="heading">data upload</h1> - - <div class="explainer"> - <p>Each of the sections below gives you a different option for data expression-data. - Please read the documentation for each section carefully to understand what - each section is about.</p> - </div> -</div> - -<div class="row"> - <h2 class="heading">R/qtl2 Bundles</h2> - - <div class="explainer"> - <p>This feature combines and extends the two upload methods below. Instead of - uploading one item at a time, the R/qtl2 bundle you upload can contain both - the genotypes data (samples/individuals/cases and their data) and the - expression data.</p> - <p>The R/qtl2 bundle, additionally, can contain extra metadata, that neither - of the methods below can handle.</p> - - <a href="{{url_for('expression-data.rqtl2.select_species')}}" - title="Upload a zip bundle of R/qtl2 files"> - <button class="btn btn-primary">upload R/qtl2 bundle</button></a> - </div> -</div> - - -<div class="row"> <h2 class="heading">Expression Data</h2> + {{flash_all_messages()}} - <div class="explainer"> - <p>This feature enables you to upload expression data. It expects the data to - be in <strong>tab-separated values (TSV)</strong> files. The data should be - a simple matrix of <em>phenotype × sample</em>, i.e. The first column is a - list of the <em>phenotypes</em> and the first row is a list of - <em>samples/cases</em>.</p> - - <p>If you haven't done so please go to this page to learn the requirements for - file formats and helpful suggestions to enter your data in a fast and easy - way.</p> - - <ol> - <li><strong>PLEASE REVIEW YOUR DATA.</strong>Make sure your data complies - with our system requirements. ( - <a href="{{url_for('expression-data.index.data_review')}}#data-concerns" - title="Details for the data expectations.">Help</a> - )</li> - <li><strong>UPLOAD YOUR DATA FOR DATA VERIFICATION.</strong> We accept - <strong>.csv</strong>, <strong>.txt</strong> and <strong>.zip</strong> - files (<a href="{{url_for('expression-data.index.data_review')}}#file-types" - title="Details for the data expectations.">Help</a>)</li> - </ol> - </div> - - <a href="{{url_for('expression-data.index.upload_file')}}" - title="Upload your expression data" - class="btn btn-primary">upload expression data</a> + <p>This section allows you to enter the expression data for your experiment. + You will need to select the species that your data concerns below.</p> </div> <div class="row"> - <h2 class="heading">samples/cases</h2> - - <div class="explainer"> - <p>For the expression data above, you need the samples/cases in your file to - already exist in the GeneNetwork database. If there are any samples that do - not already exist the upload of the expression data will fail.</p> - <p>This section gives you the opportunity to upload any missing samples</p> - </div> - - <a href="{{url_for('expression-data.samples.select_species')}}" - title="Upload samples/cases/individuals for your data" - class="btn btn-primary">upload Samples/Cases</a> + {{select_species_form(url_for("species.populations.expression-data.index"), + species)}} </div> - {%endblock%} diff --git a/uploader/templates/job_progress.html b/uploader/templates/expression-data/job-progress.html index 2feaa89..ef264e1 100644 --- a/uploader/templates/job_progress.html +++ b/uploader/templates/expression-data/job-progress.html @@ -1,5 +1,6 @@ {%extends "base.html"%} {%from "errors_display.html" import errors_display%} +{%from "populations/macro-display-population-card.html" import display_population_card%} {%block extrameta%} <meta http-equiv="refresh" content="5"> @@ -11,7 +12,9 @@ <h1 class="heading">{{job_name}}</h2> <div class="row"> - <form action="{{url_for('expression-data.parse.abort')}}" method="POST"> + <form action="{{url_for('species.populations.expression-data.abort', + species_id=species.SpeciesId, + population_id=population.Id)}}" method="POST"> <legend class="heading">Status</legend> <div class="form-group"> <label for="job_status" class="form-label">status:</label> @@ -38,3 +41,7 @@ </div> {%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/no_such_job.html b/uploader/templates/expression-data/no-such-job.html index 874d047..d22c429 100644 --- a/uploader/templates/no_such_job.html +++ b/uploader/templates/expression-data/no-such-job.html @@ -1,7 +1,8 @@ {%extends "base.html"%} {%block extrameta%} -<meta http-equiv="refresh" content="5;url={{url_for('expression-data.index.upload_file')}}"> +<meta http-equiv="refresh" + content="5;url={{url_for('species.populations.expression-data.index.upload_file')}}"> {%endblock%} {%block title%}No Such Job{%endblock%} diff --git a/uploader/templates/parse_failure.html b/uploader/templates/expression-data/parse-failure.html index 31f6be8..31f6be8 100644 --- a/uploader/templates/parse_failure.html +++ b/uploader/templates/expression-data/parse-failure.html diff --git a/uploader/templates/expression-data/parse-results.html b/uploader/templates/expression-data/parse-results.html new file mode 100644 index 0000000..03a23e2 --- /dev/null +++ b/uploader/templates/expression-data/parse-results.html @@ -0,0 +1,39 @@ +{%extends "base.html"%} +{%from "errors_display.html" import errors_display%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Parse Results{%endblock%} + +{%block contents%} + +<div class="row"> + <h2 class="heading">{{job_name}}: parse results</h2> + + {%if user_aborted%} + <span class="alert-warning">Job aborted by the user</span> + {%endif%} + + {{errors_display(errors, "No errors found in the file", "We found the following errors", True)}} + + {%if errors | length == 0 and not user_aborted %} + <form method="post" action="{{url_for('dbinsert.select_platform')}}"> + <input type="hidden" name="job_id" value="{{job_id}}" /> + <input type="submit" value="update database" class="btn btn-primary" /> + </form> + {%endif%} + + {%if errors | length > 0 or user_aborted %} + <br /> + <a href="{{url_for('species.populations.expression-data.upload_file', + species_id=species.SpeciesId, + population_id=population.Id)}}" + title="Back to index page." + class="btn btn-primary">Go back</a> + + {%endif%} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/expression-data/select-file.html b/uploader/templates/expression-data/select-file.html new file mode 100644 index 0000000..4ca461e --- /dev/null +++ b/uploader/templates/expression-data/select-file.html @@ -0,0 +1,115 @@ +{%extends "expression-data/base.html"%} +{%from "flash_messages.html" import flash_messages%} +{%from "upload_progress_indicator.html" import upload_progress_indicator%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Expression Data — Upload Data{%endblock%} + +{%block pagetitle%}Expression Data — Upload Data{%endblock%} + +{%block contents%} +{{upload_progress_indicator()}} + +<div class="row"> + <h2 class="heading">Upload Expression Data</h2> + + <p>This feature enables you to upload expression data. It expects the data to + be in <strong>tab-separated values (TSV)</strong> files. The data should be + a simple matrix of <em>phenotype × sample</em>, i.e. The first column is a + list of the <em>phenotypes</em> and the first row is a list of + <em>samples/cases</em>.</p> + + <p>If you haven't done so please go to this page to learn the requirements for + file formats and helpful suggestions to enter your data in a fast and easy + way.</p> + + <ol> + <li><strong>PLEASE REVIEW YOUR DATA.</strong>Make sure your data complies + with our system requirements. ( + <a href="{{url_for('species.populations.expression-data.data_review')}}#data-concerns" + title="Details for the data expectations.">Help</a> + )</li> + <li><strong>UPLOAD YOUR DATA FOR DATA VERIFICATION.</strong> We accept + <strong>.csv</strong>, <strong>.txt</strong> and <strong>.zip</strong> + files (<a href="{{url_for('species.populations.expression-data.data_review')}}#file-types" + title="Details for the data expectations.">Help</a>)</li> + </ol> +</div> + +<div class="row"> + <form action="{{url_for( + 'species.populations.expression-data.upload_file', + species_id=species.SpeciesId, + population_id=population.Id)}}" + method="POST" + enctype="multipart/form-data" + id="frm-upload-expression-data"> + {{flash_messages("error-expr-data")}} + + <div class="form-group"> + <legend class="heading">File Type</legend> + + <div class="radio"> + <label for="filetype_average" class="form-check-label"> + <input type="radio" name="filetype" value="average" id="filetype_average" + required="required" class="form-check-input" /> + Average</label> + <p class="form-text text-muted"> + <small>The averages data …</small></p> + </div> + + <div class="radio"> + <label for="filetype_standard_error" class="form-check-label"> + <input type="radio" name="filetype" value="standard-error" + id="filetype_standard_error" required="required" + class="form-check-input" /> + Standard Error + </label> + <p class="form-text text-muted"> + <small>The standard errors computed from the averages …</small></p> + </div> + </div> + + <div class="form-group"> + <span id="no-file-error" class="alert-danger" style="display: none;"> + No file selected + </span> + <label for="file_upload" class="form-label">Select File</label> + <input type="file" name="qc_text_file" id="file_upload" + accept="text/plain, text/tab-separated-values, application/zip" + class="form-control"/> + <p class="form-text text-muted"> + <small>Select the file to upload.</small></p> + </div> + + <button type="submit" + class="btn btn-primary" + data-toggle="modal" + data-target="#upload-progress-indicator">upload file</button> + </form> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} + +{%block javascript%} +<script type="text/javascript" src="/static/js/upload_progress.js"></script> +<script type="text/javascript"> + function setup_formdata(form) { + var formdata = new FormData(); + formdata.append( + "qc_text_file", + form.querySelector("input[type='file']").files[0]); + formdata.append( + "filetype", + selected_filetype( + Array.from(form.querySelectorAll("input[type='radio']")))); + return formdata; + } + + setup_upload_handlers( + "frm-upload-expression-data", make_data_uploader(setup_formdata)); +</script> +{%endblock%} diff --git a/uploader/templates/expression-data/select-population.html b/uploader/templates/expression-data/select-population.html new file mode 100644 index 0000000..8555e27 --- /dev/null +++ b/uploader/templates/expression-data/select-population.html @@ -0,0 +1,29 @@ +{%extends "expression-data/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} +{%from "populations/macro-select-population.html" import select_population_form%} + +{%block title%}Expression Data{%endblock%} + +{%block pagetitle%}Expression Data{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>You have selected the species. Now you need to select the population that + the expression data belongs to.</p> +</div> + +<div class="row"> + {{select_population_form(url_for( + "species.populations.expression-data.select_population", + species_id=species.SpeciesId), + populations)}} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} diff --git a/uploader/templates/genotypes/create-dataset.html b/uploader/templates/genotypes/create-dataset.html new file mode 100644 index 0000000..10331c1 --- /dev/null +++ b/uploader/templates/genotypes/create-dataset.html @@ -0,0 +1,82 @@ +{%extends "genotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Genotypes — Create Dataset{%endblock%} + +{%block pagetitle%}Genotypes — Create Dataset{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="create-dataset"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.genotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}">Create Dataset</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <form id="frm-geno-create-dataset" + method="POST" + action="{{url_for('species.populations.genotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}"> + <legend>Create a new Genotype Dataset</legend> + + <div class="form-group"> + <label for="txt-geno-dataset-name" class="form-label">Name</label> + <input type="text" + id="txt-geno-dataset-name" + name="geno-dataset-name" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>This is a short representative, but constrained name for the genotype + dataset.<br /> + The field will only accept letters ('A-Za-z'), numbers (0-9), hyphens + and underscores. Any other character will cause the name to be + rejected.</p></small> + </div> + + <div class="form-group"> + <label for="txt-geno-dataset-fullname" class="form-label">Full Name</label> + <input type="text" + id="txt-geno-dataset-fullname" + name="geno-dataset-fullname" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>This is a longer, more descriptive name for your dataset.</p></small> + </div> + + <div class="form-group"> + <label for="txt-geno-dataset-shortname" + class="form-label">Short Name</label> + <input type="text" + id="txt-geno-dataset-shortname" + name="geno-dataset-shortname" + class="form-control" /> + <small class="form-text text-muted"> + <p>A short name for your dataset. If you leave this field blank, the + short name will be set to the same value as the + "<strong>Name</strong>" field above.</p></small> + </div> + + <div class="form-group"> + <input type="submit" + class="btn btn-primary" + value="create dataset" /> + </div> + </form> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/genotypes/list-genotypes.html b/uploader/templates/genotypes/list-genotypes.html index 0efc693..e4c39eb 100644 --- a/uploader/templates/genotypes/list-genotypes.html +++ b/uploader/templates/genotypes/list-genotypes.html @@ -90,7 +90,56 @@ </div> <div class="row"> + <h2>Genotype Datasets</h2> + <p>The genotype data is organised under various genotype datasets. You can + click on the link for the relevant dataset to view a little more information + about it.</p> + + {%if dataset is not none%} + <table class="table"> + <thead> + <tr> + <th>Name</th> + <th>Full Name</th> + </tr> + </thead> + + <tbody> + <tr> + <td>{{dataset.Name}}</td> + <td><a href="{{url_for('species.populations.genotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}" + title="View details regarding and manage dataset '{{dataset.FullName}}'"> + {{dataset.FullName}}</a></td> + </tr> + </tbody> + </table> + {%else%} + <p class="text-warning"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + There is no genotype dataset defined for this population. + </p> + <p> + <a href="{{url_for('species.populations.genotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}" + title="Create a new genotype dataset for the '{{population.FullName}}' population for the '{{species.FullName}}' species." + class="btn btn-primary"> + create new genotype dataset</a></p> + {%endif%} +</div> +<div class="row text-warning"> + <p> + <span class="glyphicon glyphicon-exclamation-sign"></span> + <strong>NOTE</strong>: Currently the GN2 (and related) system(s) expect a + single genotype dataset. If there is more than one, the system apparently + fails in unpredictable ways. + </p> + <p>Fix this to allow multiple datasets, each with a different assembly from + all the rest.</p> </div> {%endblock%} diff --git a/uploader/templates/genotypes/view-dataset.html b/uploader/templates/genotypes/view-dataset.html new file mode 100644 index 0000000..e7ceb36 --- /dev/null +++ b/uploader/templates/genotypes/view-dataset.html @@ -0,0 +1,61 @@ +{%extends "genotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Genotypes: View Dataset{%endblock%} + +{%block pagetitle%}Genotypes: View Dataset{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="view-dataset"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.genotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">view dataset</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <h2>Genotype Dataset Details</h2> + <table class="table"> + <thead> + <tr> + <th>Name</th> + <th>Full Name</th> + </tr> + </thead> + + <tbody> + <tr> + <td>{{dataset.Name}}</td> + <td>{{dataset.FullName}}</td> + </tr> + </tbody> + </table> +</div> + +<div class="row text-warning"> + <h2>Assembly Details</h2> + + <p>Maybe include the assembly details here if found to be necessary.</p> +</div> + +<div class="row"> + <h2>Genotype Data</h2> + + <p class="text-danger"> + Provide link to enable uploading of genotype data here.</p> +</div> + +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/index.html b/uploader/templates/index.html index 0fa4a1b..d6f57eb 100644 --- a/uploader/templates/index.html +++ b/uploader/templates/index.html @@ -5,12 +5,6 @@ {%block pagetitle%}Home{%endblock%} -{%block breadcrumb%} -<li class="breadcrumb-item active"> - <a href="{{url_for('base.index')}}">Home</a> -</li> -{%endblock%} - {%block contents%} <div class="row"> @@ -50,7 +44,7 @@ <h2>Populations</h2> <p>Your studies will probably focus on a particular subset of the entire - species you are interested in &ndash your population.</p> + species you are interested in – your population.</p> <p>Populations are a way to organise the species data so as to link data to specific know populations for a particular species, e.g. The BXD population of mice (Mus musculus)</p> diff --git a/uploader/templates/macro-table-pagination.html b/uploader/templates/macro-table-pagination.html new file mode 100644 index 0000000..292c531 --- /dev/null +++ b/uploader/templates/macro-table-pagination.html @@ -0,0 +1,26 @@ +{%macro table_pagination(start_at, page_count, total_count, base_uri, name)%} +{%set ns = namespace(forward_uri=base_uri, back_uri=base_uri)%} +{%set ns.forward_uri="brr"%} + <div class="row"> + <div class="col-md-2" style="text-align: start;"> + {%if start_at > 0%} + <a href="{{base_uri + + '?start_at='+((start_at-page_count)|string) + + '&count='+(page_count|string)}}"> + <span class="glyphicon glyphicon-backward"></span> + Previous + </a> + {%endif%} + </div> + <div class="col-md-8" style="text-align: center;"> + Displaying {{name}} {{start_at+1}} to {{start_at+page_count if start_at+page_count < total_count else total_count}} of {{total_count}}</div> + <div class="col-md-2" style="text-align: end;"> + {%if start_at + page_count < total_count%} + <a href="{{base_uri + + '?start_at='+((start_at+page_count)|string) + + '&count='+(page_count|string)}}"> + Next<span class="glyphicon glyphicon-forward"></span></a> + {%endif%} + </div> + </div> +{%endmacro%} diff --git a/uploader/templates/parse_results.html b/uploader/templates/parse_results.html deleted file mode 100644 index 46fbaaf..0000000 --- a/uploader/templates/parse_results.html +++ /dev/null @@ -1,30 +0,0 @@ -{%extends "base.html"%} -{%from "errors_display.html" import errors_display%} - -{%block title%}Parse Results{%endblock%} - -{%block contents%} -<h1 class="heading">{{job_name}}: parse results</h2> - -{%if user_aborted%} -<span class="alert-warning">Job aborted by the user</span> -{%endif%} - -{{errors_display(errors, "No errors found in the file", "We found the following errors", True)}} - -{%if errors | length == 0 and not user_aborted %} -<form method="post" action="{{url_for('dbinsert.select_platform')}}"> - <input type="hidden" name="job_id" value="{{job_id}}" /> - <input type="submit" value="update database" class="btn btn-primary" /> -</form> -{%endif%} - -{%if errors | length > 0 or user_aborted %} -<br /> -<a href="{{url_for('expression-data.index.upload_file')}}" title="Back to index page." - class="btn btn-primary"> - Go back -</a> -{%endif%} - -{%endblock%} diff --git a/uploader/templates/phenotypes/add-phenotypes.html b/uploader/templates/phenotypes/add-phenotypes.html new file mode 100644 index 0000000..196bc69 --- /dev/null +++ b/uploader/templates/phenotypes/add-phenotypes.html @@ -0,0 +1,231 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "phenotypes/macro-display-pheno-dataset-card.html" import display_pheno_dataset_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="add-phenotypes"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">View Datasets</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <form id="frm-add-phenotypes" + method="POST" + enctype="multipart/form-data" + action="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}"> + <legend>Add New Phenotypes</legend> + + <div class="form-text help-block"> + <p>Select the zip file bundle containing information on the phenotypes you + wish to upload, then click the "Upload Phenotypes" button below to + upload the data.</p> + <p>See the <a href="#section-file-formats">File Formats</a> section below + to get an understanding of what is expected of the bundle files you + upload.</p> + <p><strong>This will not update any existing phenotypes!</strong></p> + </div> + + <div class="form-group"> + <label for="finput-phenotypes-bundle" class="form-label"> + Phenotypes Bundle</label> + <input type="file" + id="finput-phenotypes-bundle" + name="phenotypes-bundle" + accept="application/zip, .zip" + required="required" + class="form-control" /> + </div> + + <div class="form-group"> + <input type="submit" + value="upload phenotypes" + class="btn btn-primary" /> + </div> + </form> +</div> + +<div class="row"> + <h2 class="heading" id="section-file-formats">File Formats</h2> + <p>We accept an extended form of the + <a href="https://kbroman.org/qtl2/assets/vignettes/input_files.html#format-of-the-data-files" + title="R/qtl2 software input file format documentation"> + input files' format used with the R/qtl2 software</a> as a single ZIP + file</p> + <p>The files that are used for this feature are: + <ul> + <li>the <em>control</em> file</li> + <li><em>pheno</em> file(s)</li> + <li><em>phenocovar</em> file(s)</li> + <li><em>phenose</em> files(s)</li> + </ul> + </p> + <p>Other files within the bundle will be ignored, for this feature.</p> + <p>The following section will detail the expectations for each of the + different file types within the uploaded ZIP file bundle for phenotypes:</p> + + <h3 class="subheading">Control File</h3> + <p>There <strong>MUST be <em>one, and only one</em></strong> file that acts + as the control file. This file can be: + <ul> + <li>a <em>JSON</em> file, or</li> + <li>a <em>YAML</em> file.</li> + </ul> + </p> + + <p>The control file is useful for defining things about the bundle such as:</p> + <ul> + <li>The field separator value (default: <code>sep: ','</code>). There can + only ever be one field separator and it <strong>MUST</strong> be the same + one for <strong>ALL</strong> files in the bundle.</li> + <li>The comment character (default: <code>comment.char: '#'</code>). Any + line that starts with this character will be considered a comment line and + be ignored in its entirety.</li> + <li>Code for missing values (default: <code>na.strings: 'NA'</code>). You + can specify more than one code to indicate missing values, e.g. + <code>{…, "na.strings": ["NA", "N/A", "-"], …}</code></li> + </ul> + + <h3 class="subheading"><em>pheno</em> File(s)</h3> + <p>These files are the main data files. You must have at least one of these + files in your bundle for it to be valid for this step.</p> + <p>The data is a matrix of <em>individuals × phenotypes</em> by default, as + below:<br /> + <code> + id,10001,10002,10003,10004,…<br /> + BXD1,61.400002,54.099998,483,49.799999,…<br /> + BXD2,49,50.099998,403,45.5,…<br /> + BXD5,62.5,53.299999,501,62.900002,…<br /> + BXD6,53.099998,55.099998,403,NA,…<br /> + ⋮<br /></code> + </p> + <p>If the <code>pheno_transposed</code> value is set to <code>True</code>, + then the data will be a <em>phenotypes × individuals</em> matrix as in the + example below:<br /> + <code> + id,BXD1,BXD2,BXD5,BXD6,…<br /> + 10001,61.400002,49,62.5,53.099998,…<br /> + 10002,54.099998,50.099998,53.299999,55.099998,…<br /> + 10003,483,403,501,403,…<br /> + 10004,49.799999,45.5,62.900002,NA,…<br /> + ⋮ + </code> + </p> + + + <h3 class="subheading"><em>phenocovar</em> File(s)</h3> + <p>At least one phenotypes metadata file with the metadata values such as + descriptions, PubMed Identifier, publication titles (if present), etc.</p> + <p>The data in this/these file(s) is a matrix of + <em>phenotypes × phenotypes-covariates</em>. The first column is always the + phenotype names/identifiers — same as in the R/qtl2 format.</p> + <p><em>phenocovar</em> files <strong>should never be transposed</strong>!</p> + <p>This file <strong>MUST</strong> be present in the bundle, and have data for + the bundle to be considered valid by our system for this step.<br /> + In addition to that, the following are the fields that <strong>must be + present</strong>, and + have values, in the file before the file is considered valid: + <ul> + <li><em>description</em>: A description for each phenotype. Useful + for users to know what the phenotype is about.</li> + <li><em>units</em>: The units of measurement for the phenotype, + e.g. milligrams for brain weight, centimetres/millimetres for + tail-length, etc.</li> + </ul></p> + + <p>The following <em>optional</em> fields can also be provided: + <ul> + <li><em>pubmedid</em>: A PubMed Identifier for the publication where + the phenotype is published. If this field is not provided, the system will + assume your phenotype is not published.</li> + </ul> + </p> + <p>These files will be marked up in the control file with the + <code>phenocovar</code> key, as in the examples below: + <ol> + <li>JSON: single file<br /> + <code>{<br /> + ⋮,<br /> + "phenocovar": "your_covariates_file.csv",<br /> + ⋮<br /> + } + </code> + </li> + <li>JSON: multiple files<br /> + <code>{<br /> + ⋮,<br /> + "phenocovar": [<br /> + "covariates_file_01.csv",<br /> + "covariates_file_01.csv",<br /> + ⋮<br /> + ],<br /> + ⋮<br /> + } + </code> + </li> + <li>YAML: single file or<br /> + <code> + ⋮<br /> + phenocovar: your_covariates_file.csv<br /> + ⋮ + </code> + </li> + <li>YAML: multiple files<br /> + <code> + ⋮<br /> + phenocovar:<br /> + - covariates_file_01.csv<br /> + - covariates_file_02.csv<br /> + - covariates_file_03.csv<br /> + …<br /> + ⋮ + </code> + </li> + </ol> + </p> + + <h3 class="subheading"><em>phenose</em> and <em>phenonum</em> File(s)</h3> + <p>These are extensions to the R/qtl2 standard, i.e. these types ofs file are + not supported by the original R/qtl2 file format</p> + <p>We use these files to upload the standard errors (<em>phenose</em>) when + the data file (<em>pheno</em>) is average data. In that case, the + <em>phenonum</em> file(s) contains the number of individuals that were + involved when computing the averages.</p> + <p>Both types of files are matrices of <em>individuals × phenotypes</em> by + default. Like the related <em>pheno</em> files, if + <code>pheno_transposed: True</code>, then the file will be a matrix of + <em>phenotypes × individuals</em>.</p> +</div> + +<div class="row text-warning"> + <h3 class="subheading">Notes for Devs (well… Fred, really.)</h3> + <p>Use the following resources for automated retrieval of certain data</p> + <ul> + <li><a href="https://www.ncbi.nlm.nih.gov/pmc/tools/developers/" + title="NCBI APIs: Retrieve articles' metadata etc."> + NCBI APIS</a></li> + </ul> +</div> + +{%endblock%} + +{%block sidebarcontents%} +{{display_pheno_dataset_card(species, population, dataset)}} +{%endblock%} diff --git a/uploader/templates/phenotypes/base.html b/uploader/templates/phenotypes/base.html new file mode 100644 index 0000000..3bc5dea --- /dev/null +++ b/uploader/templates/phenotypes/base.html @@ -0,0 +1,12 @@ +{%extends "populations/base.html"%} + +{%block lvl3_breadcrumbs%} +<li {%if activelink=="phenotypes"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.index')}}">Phenotypes</a> +</li> +{%block lvl4_breadcrumbs%}{%endblock%} +{%endblock%} diff --git a/uploader/templates/phenotypes/create-dataset.html b/uploader/templates/phenotypes/create-dataset.html new file mode 100644 index 0000000..93de92f --- /dev/null +++ b/uploader/templates/phenotypes/create-dataset.html @@ -0,0 +1,106 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="create-dataset"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}">Create Datasets</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>Create a new phenotype dataset.</p> +</div> + +<div class="row"> + <form id="frm-create-pheno-dataset" + action="{{url_for('species.populations.phenotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}" + method="POST"> + + <div class="form-group"> + <label class="form-label" for="txt-dataset-name">Name</label> + {%if errors["dataset-name"] is defined%} + <small class="form-text text-muted danger"> + <p>{{errors["dataset-name"]}}</p></small> + {%endif%} + <input type="text" + name="dataset-name" + id="txt-dataset-name" + value="{{original_formdata.get('dataset-name') or (population.InbredSetCode + 'Publish')}}" + {%if errors["dataset-name"] is defined%} + class="form-control danger" + {%else%} + class="form-control" + {%endif%} + required="required" /> + <small class="form-text text-muted"> + <p>A short representative name for the dataset.</p> + <p>Recommended: Use the population code and append "Publish" at the end. + <br />This field will only accept names composed of + letters ('A-Za-z'), numbers (0-9), hyphens and underscores.</p> + </small> + </div> + + <div class="form-group"> + <label class="form-label" for="txt-dataset-fullname">Full Name</label> + {%if errors["dataset-fullname"] is defined%} + <small class="form-text text-muted danger"> + <p>{{errors["dataset-fullname"]}}</p></small> + {%endif%} + <input id="txt-dataset-fullname" + name="dataset-fullname" + type="text" + value="{{original_formdata.get('dataset-fullname', '')}}" + {%if errors["dataset-fullname"] is defined%} + class="form-control danger" + {%else%} + class="form-control" + {%endif%} + required="required" /> + <small class="form-text text-muted"> + <p>A longer, descriptive name for the dataset — useful for humans. + </p></small> + </div> + + <div class="form-group"> + <label class="form-label" for="txt-dataset-shortname">Short Name</label> + <input id="txt-dataset-shortname" + name="dataset-shortname" + type="text" + class="form-control" + value="{{original_formdata.get('dataset-shortname') or (population.InbredSetCode + ' Publish')}}" /> + <small class="form-text text-muted"> + <p>An optional, short name for the dataset. <br /> + If this is not provided, it will default to the value provided for the + <strong>Name</strong> field above.</p></small> + </div> + + <div class="form-group"> + <input type="submit" + class="btn btn-primary" + value="create phenotype dataset" /> + </div> + + </form> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/phenotypes/index.html b/uploader/templates/phenotypes/index.html new file mode 100644 index 0000000..0c691e6 --- /dev/null +++ b/uploader/templates/phenotypes/index.html @@ -0,0 +1,26 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-select-species.html" import select_species_form%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>This section deals with phenotypes that + <span class="text-warning"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + … what are the characteristics of these phenotypes? …</span></p> + <p>Select the species to begin the process of viewing/uploading data about + your phenotypes</p> +</div> + +<div class="row"> + {{select_species_form(url_for("species.populations.phenotypes.index"), + species)}} +</div> +{%endblock%} diff --git a/uploader/templates/phenotypes/list-datasets.html b/uploader/templates/phenotypes/list-datasets.html new file mode 100644 index 0000000..2eaf43a --- /dev/null +++ b/uploader/templates/phenotypes/list-datasets.html @@ -0,0 +1,65 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="list-datasets"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.list_datasets', + species_id=species.SpeciesId, + population_id=population.Id)}}">List Datasets</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + {%if datasets | length > 0%} + <p>The dataset(s) available for this population is/are:</p> + + <table class="table"> + <thead> + <tr> + <th>Name</th> + <th>Full Name</th> + <th>Short Name</th> + </tr> + </thead> + + <tbody> + {%for dataset in datasets%} + <tr> + <td><a href="{{url_for('species.populations.phenotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">{{dataset.Name}}</a></td> + <td>{{dataset.FullName}}</td> + <td>{{dataset.ShortName}}</td> + </tr> + {%endfor%} + </tbody> + </table> + {%else%} + <p class="text-warning"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + There is no dataset for this population!</p> + <p><a href="{{url_for('species.populations.phenotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}" + class="btn btn-primary" + title="Create a new phenotype dataset.">create dataset</a></p> + {%endif%} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/phenotypes/macro-display-pheno-dataset-card.html b/uploader/templates/phenotypes/macro-display-pheno-dataset-card.html new file mode 100644 index 0000000..11b108b --- /dev/null +++ b/uploader/templates/phenotypes/macro-display-pheno-dataset-card.html @@ -0,0 +1,31 @@ +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%macro display_pheno_dataset_card(species, population, dataset)%} +{{display_population_card(species, population)}} + +<div class="card"> + <div class="card-body"> + <h5 class="card-title">Phenotypes' Dataset</h5> + <div class="card-text"> + <table class="table"> + <tbody> + <tr> + <td>Name</td> + <td>{{dataset.Name}}</td> + </tr> + + <tr> + <td>Full Name</td> + <td>{{dataset.FullName}}</td> + </tr> + + <tr> + <td>Short Name</td> + <td>{{dataset.ShortName}}</td> + </tr> + </tbody> + </table> + </div> + </div> +</div> +{%endmacro%} diff --git a/uploader/templates/phenotypes/select-population.html b/uploader/templates/phenotypes/select-population.html new file mode 100644 index 0000000..eafd4a7 --- /dev/null +++ b/uploader/templates/phenotypes/select-population.html @@ -0,0 +1,28 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} +{%from "populations/macro-select-population.html" import select_population_form%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>Select the population for your phenotypes to view and manage the phenotype + datasets that relate to it.</p> +</div> + +<div class="row"> + {{select_population_form(url_for("species.populations.phenotypes.select_population", + species_id=species.SpeciesId), + populations)}} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} diff --git a/uploader/templates/phenotypes/view-dataset.html b/uploader/templates/phenotypes/view-dataset.html new file mode 100644 index 0000000..b136bb6 --- /dev/null +++ b/uploader/templates/phenotypes/view-dataset.html @@ -0,0 +1,96 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "macro-table-pagination.html" import table_pagination%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="view-dataset"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.view_dataset', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}">View Datasets</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>The basic dataset details are:</p> + + <table class="table"> + <thead> + <tr> + <th>Name</th> + <th>Full Name</th> + <th>Short Name</th> + </tr> + </thead> + + <tbody> + <tr> + <td>{{dataset.Name}}</td> + <td>{{dataset.FullName}}</td> + <td>{{dataset.ShortName}}</td> + </tr> + </tbody> + </table> +</div> + +<div class="row"> + <p><a href="{{url_for('species.populations.phenotypes.add_phenotypes', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id)}}" + title="Add a bunch of phenotypes" + class="btn btn-primary">Add phenotypes</a></p> +</div> + +<div class="row"> + <h2>Phenotype Data</h2> + + <p>This dataset has a total of {{phenotype_count}} phenotypes.</p> + + {{table_pagination(start_from, count, phenotype_count, url_for('species.populations.phenotypes.view_dataset', species_id=species.SpeciesId, population_id=population.Id, dataset_id=dataset.Id), "phenotypes")}} + + <table class="table"> + <thead> + <tr> + <th>#</th> + <th>Record</th> + <th>Description</th> + </tr> + </thead> + + <tbody> + {%for pheno in phenotypes%} + <tr> + <td>{{pheno.sequence_number}}</td> + <td><a href="{{url_for('species.populations.phenotypes.view_phenotype', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id, + xref_id=pheno['pxr.Id'])}}" + title="View phenotype details"> + {{pheno.InbredSetCode}}_{{pheno["pxr.Id"]}}</a></td> + <td>{{pheno.Post_publication_description or pheno.Pre_publication_abbreviation or pheno.Original_description}}</td> + </tr> + {%else%} + <tr><td colspan="5"></td></tr> + {%endfor%} + </tbody> + </table> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/phenotypes/view-phenotype.html b/uploader/templates/phenotypes/view-phenotype.html new file mode 100644 index 0000000..99bb8e5 --- /dev/null +++ b/uploader/templates/phenotypes/view-phenotype.html @@ -0,0 +1,126 @@ +{%extends "phenotypes/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "populations/macro-display-population-card.html" import display_population_card%} + +{%block title%}Phenotypes{%endblock%} + +{%block pagetitle%}Phenotypes{%endblock%} + +{%block lvl4_breadcrumbs%} +<li {%if activelink=="view-phenotype"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.phenotypes.view_phenotype', + species_id=species.SpeciesId, + population_id=population.Id, + dataset_id=dataset.Id, + xref_id=xref_id)}}">View Datasets</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <div class="panel panel-default"> + <div class="panel-heading"><strong>Basic Phenotype Details</strong></div> + + <table class="table"> + <tbody> + <tr> + <td><strong>Phenotype</strong></td> + <td>{{phenotype.Post_publication_description or phenotype.Pre_publication_abbreviation or phenotype.Original_description}} + </tr> + <tr> + <td><strong>Cross-Reference ID</strong></td> + <td>{{phenotype.xref_id}}</td> + </tr> + <tr> + <td><strong>Collation</strong></td> + <td>{{dataset.FullName}}</td> + </tr> + <tr> + <td><strong>Units</strong></td> + <td>{{phenotype.Units}}</td> + </tr> + </tbody> + </table> + + <form action="#edit-delete-phenotype" + method="POST" + id="frm-delete-phenotype"> + + <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> + <input type="hidden" name="population_id" value="{{population.Id}}" /> + <input type="hidden" name="dataset_id" value="{{dataset.Id}}" /> + <input type="hidden" name="phenotype_id" value="{{phenotype.Id}}" /> + + <div class="btn-group btn-group-justified"> + <div class="btn-group"> + {%if "group:resource:edit-resource" in privileges%} + <input type="submit" + title="Edit the values for the phenotype. This is meant to be used when you need to update only a few values." + class="btn btn-primary not-implemented" + value="edit" /> + {%endif%} + </div> + <div class="btn-group"></div> + <div class="btn-group"> + {%if "group:resource:delete-resource" in privileges%} + <input type="submit" + title="Delete the entire phenotype. This is useful when you need to change data for most or all of the fields for this phenotype." + class="btn btn-danger not-implemented" + value="delete" /> + {%endif%} + </div> + </div> + </form> + </div> +</div> + +<div class="row"> + <div class="panel panel-default"> + <div class="panel-heading"><strong>Phenotype Data</strong></div> + {%if "group:resource:view-resource" in privileges%} + <table class="table"> + <thead> + <tr> + <th>#</th> + <th>Sample</th> + <th>Value</th> + <th>Symbol</th> + <th>SE</th> + <th>N</th> + </tr> + </thead> + + <tbody> + {%for item in phenotype.data%} + <tr> + <td>{{loop.index}}</td> + <td>{{item.StrainName}}</td> + <td>{{item.value}}</td> + <td>{{item.Symbol or "-"}}</td> + <td>{{item.error or "-"}}</td> + <td>{{item.count or "-"}}</td> + </tr> + {%endfor%} + </tbody> + </table> + {%else%} + <p class="text-danger"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + You do not currently have privileges to view this phenotype in greater + detail. + </p> + {%endif%} + </div> +</div> + +{%endblock%} + +{%block sidebarcontents%} +{{display_population_card(species, population)}} +{%endblock%} diff --git a/uploader/templates/platforms/base.html b/uploader/templates/platforms/base.html new file mode 100644 index 0000000..dac965f --- /dev/null +++ b/uploader/templates/platforms/base.html @@ -0,0 +1,13 @@ +{%extends "species/base.html"%} + +{%block lvl3_breadcrumbs%} +<li {%if activelink=="platforms"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.populations.platforms.index')}}"> + Sequencing Platforms</a> +</li> +{%block lvl4_breadcrumbs%}{%endblock%} +{%endblock%} diff --git a/uploader/templates/platforms/create-platform.html b/uploader/templates/platforms/create-platform.html new file mode 100644 index 0000000..0866d5e --- /dev/null +++ b/uploader/templates/platforms/create-platform.html @@ -0,0 +1,124 @@ +{%extends "platforms/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} + +{%block title%}Platforms — Create Platforms{%endblock%} + +{%block pagetitle%}Platforms — Create Platforms{%endblock%} + +{%block lvl3_breadcrumbs%} +<li {%if activelink=="create-platform"%} + class="breadcrumb-item active" + {%else%} + class="breadcrumb-item" + {%endif%}> + <a href="{{url_for('species.platforms.create_platform', + species_id=species.SpeciesId)}}">create platform</a> +</li> +{%endblock%} + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <h2>Create New Platform</h2> + + <p>You can create a new genetic sequencing platform below.</p> +</div> + +<div class="row"> + <form id="frm-create-platform" + method="POST" + action="{{url_for('species.platforms.create_platform', + species_id=species.SpeciesId)}}"> + + <div class="form-group"> + <label for="txt-geo-platform" class="form-label">GEO Platform</label> + <input type="text" + id="txt-geo-platform" + name="geo-platform" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>This is the platform's + <a href="https://www.ncbi.nlm.nih.gov/geo/browse/?view=platforms&tax={{species.TaxonomyId}}" + title="Platforms for '{{species.FullName}}' on NCBI"> + accession value on NCBI</a>. If you do not know the value, click the + link and search on NCBI for species '{{species.FullName}}'.</p></small> + </div> + + <div class="form-group"> + <label for="txt-platform-name" class="form-label">Platform Name</label> + <input type="text" + id="txt-platform-name" + name="platform-name" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>This is name of the genetic sequencing platform.</p></small> + </div> + + <div class="form-group"> + <label for="txt-platform-shortname" class="form-label"> + Platform Short Name</label> + <input type="text" + id="txt-platform-shortname" + name="platform-shortname" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>Use the following conventions for this field: + <ol> + <li>Start with a 4-letter vendor code, e.g. "Affy" for "Affymetrix", "Illu" for "Illumina", etc.</li> + <li>Append an underscore to the 4-letter vendor code</li> + <li>Use the name of the array given by the vendor, e.g. U74AV2, MOE430A, etc.</li> + </ol> + </p> + </small> + </div> + + <div class="form-group"> + <label for="txt-platform-title" class="form-label">Platform Title</label> + <input type="text" + id="txt-platform-title" + name="platform-title" + required="required" + class="form-control" /> + <small class="form-text text-muted"> + <p>The full platform title. Sometimes, this is the same as the Platform + Name above.</p></small> + </div> + + <div class="form-group"> + <label for="txt-go-tree-value" class="form-label">GO Tree Value</label> + <input type="text" + id="txt-go-tree-value" + name="go-tree-value" + class="form-control" /> + <small class="form-text text-muted"> + <p>This is a Chip identification value useful for analysis with the + <strong> + <a href="https://www.geneweaver.org/" + title="Go to the GeneWeaver site." + target="_blank">GeneWeaver</a></strong> + and + <strong> + <a href="https://www.webgestalt.org/" + title="Go to the WEB-based GEne SeT AnaLysis Toolkit site." + target="_blank">WebGestalt</a></strong> + tools.<br /> + This can be left blank for custom platforms.</p></small> + </div> + + <div class="form-group"> + <input type="submit" + value="create new platform" + class="btn btn-primary" /> + </div> + </form> +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} diff --git a/uploader/templates/platforms/index.html b/uploader/templates/platforms/index.html new file mode 100644 index 0000000..35b6464 --- /dev/null +++ b/uploader/templates/platforms/index.html @@ -0,0 +1,21 @@ +{%extends "platforms/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-select-species.html" import select_species_form%} + +{%block title%}Platforms{%endblock%} + +{%block pagetitle%}Platforms{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>In this section, you will be able to view and manage the sequencing + platforms that are currently supported by GeneNetwork.</p> +</div> + +<div class="row"> + {{select_species_form(url_for("species.platforms.index"), species)}} +</div> +{%endblock%} diff --git a/uploader/templates/platforms/list-platforms.html b/uploader/templates/platforms/list-platforms.html new file mode 100644 index 0000000..718dd1d --- /dev/null +++ b/uploader/templates/platforms/list-platforms.html @@ -0,0 +1,93 @@ +{%extends "platforms/base.html"%} +{%from "flash_messages.html" import flash_all_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} + +{%block title%}Platforms — List Platforms{%endblock%} + +{%block pagetitle%}Platforms — List Platforms{%endblock%} + + +{%block contents%} +{{flash_all_messages()}} + +<div class="row"> + <p>View the list of the genetic sequencing platforms that are currently + supported by GeneNetwork.</p> + <p>If you cannot find the platform you wish to use, you can add it by clicking + the "New Platform" button below.</p> + <p><a href="{{url_for('species.platforms.create_platform', + species_id=species.SpeciesId)}}" + title="Create a new genetic sequencing platform for species {{species.FullName}}" + class="btn btn-primary">Create Platform</a></p> +</div> + +<div class="row"> + <h2>Supported Platforms</h2> + {%if platforms is defined and platforms | length > 0%} + <p>There are {{total_platforms}} platforms supported by GeneNetwork</p> + + <div class="row"> + <div class="col-md-2" style="text-align: start;"> + {%if start_from > 0%} + <a href="{{url_for('species.platforms.list_platforms', + species_id=species.SpeciesId, + start_from=start_from-count, + count=count)}}"> + <span class="glyphicon glyphicon-backward"></span> + Previous + </a> + {%endif%} + </div> + <div class="col-md-8" style="text-align: center;"> + Displaying platforms {{start_from+1}} to {{start_from+count if start_from+count < total_platforms else total_platforms}} of + {{total_platforms}} + </div> + <div class="col-md-2" style="text-align: end;"> + {%if start_from + count < total_platforms%} + <a href="{{url_for('species.platforms.list_platforms', + species_id=species.SpeciesId, + start_from=start_from+count, + count=count)}}"> + Next + <span class="glyphicon glyphicon-forward"></span> + </a> + {%endif%} + </div> + </div> + + <table class="table"> + <thead> + <tr> + <th>#</th> + <th>Platform Name</th> + <th><a href="https://www.ncbi.nlm.nih.gov/geo/browse/?view=platforms&tax={{species.TaxonomyId}}" + title="Gene Expression Omnibus: Platforms section" + target="_blank">GEO Platform</a></th> + <th>Title</th> + </tr> + </thead> + + <tbody> + {%for platform in platforms%} + <tr> + <td>{{platform.sequence_number}}</td> + <td>{{platform.GeneChipName}}</td> + <td><a href="https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc={{platform.GeoPlatform}}" + title="View platform on the Gene Expression Omnibus" + target="_blank">{{platform.GeoPlatform}}</a></td> + <td>{{platform.Title}}</td> + </tr> + {%endfor%} + </tbody> + </table> + {%else%} + <p class="text-warning"> + <span class="glyphicon glyphicon-exclamation-sign"></span> + There are no platforms supported at this time!</p> + {%endif%} +</div> +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} diff --git a/uploader/templates/populations/macro-display-population-card.html b/uploader/templates/populations/macro-display-population-card.html index e68f8e3..79f7925 100644 --- a/uploader/templates/populations/macro-display-population-card.html +++ b/uploader/templates/populations/macro-display-population-card.html @@ -7,25 +7,39 @@ <div class="card-body"> <h5 class="card-title">Population</h5> <div class="card-text"> - <dl> - <dt>Name</dt> - <dd>{{population.Name}}</dd> + <table class="table"> + <tbody> + <tr> + <td>Name</td> + <td>{{population.Name}}</td> + </tr> - <dt>Full Name</dt> - <dd>{{population.FullName}}</dd> + <tr> + <td>Full Name</td> + <td>{{population.FullName}}</td> + </tr> - <dt>Code</dt> - <dd>{{population.InbredSetCode}}</dd> + <tr> + <td>Code</td> + <td>{{population.InbredSetCode}}</td> + </tr> - <dt>Genetic Type</dt> - <dd>{{population.GeneticType}}</dd> + <tr> + <td>Genetic Type</td> + <td>{{population.GeneticType}}</td> + </tr> - <dt>Family</dt> - <dd>{{population.Family}}</dd> + <tr> + <td>Family</td> + <td>{{population.Family}}</td> + </tr> - <dt>Description</dt> - <dd>{{population.Description or "-"}}</dd> - </dl> + <tr> + <td>Description</td> + <td>{{(population.Description or "")[0:500]}}…</td> + </tr> + </tbody> + </table> </div> </div> </div> diff --git a/uploader/templates/rqtl2/create-tissue-success.html b/uploader/templates/populations/rqtl2/create-tissue-success.html index d6fe154..d6fe154 100644 --- a/uploader/templates/rqtl2/create-tissue-success.html +++ b/uploader/templates/populations/rqtl2/create-tissue-success.html diff --git a/uploader/templates/populations/rqtl2/index.html b/uploader/templates/populations/rqtl2/index.html new file mode 100644 index 0000000..ec6ffb8 --- /dev/null +++ b/uploader/templates/populations/rqtl2/index.html @@ -0,0 +1,54 @@ +{%extends "base.html"%} +{%from "flash_messages.html" import flash_messages%} + +{%block title%}Data Upload{%endblock%} + +{%block contents%} +<h1 class="heading">R/qtl2 data upload</h1> + +<h2>R/qtl2 Upload</h2> + +<div class="row"> + <form method="POST" action="{{url_for('expression-data.rqtl2.select_species')}}" + id="frm-rqtl2-upload"> + <legend class="heading">upload R/qtl2 bundle</legend> + {{flash_messages("error-rqtl2")}} + + <div class="form-group"> + <label for="select:species" class="form-label">Species</label> + <select id="select:species" + name="species_id" + required="required" + class="form-control"> + <option value="">Select species</option> + {%for spec in species%} + <option value="{{spec.SpeciesId}}">{{spec.MenuName}}</option> + {%endfor%} + </select> + <small class="form-text text-muted"> + Data that you upload to the system should belong to a know species. + Here you can select the species that you wish to upload data for. + </small> + </div> + + <input type="submit" class="btn btn-primary" value="submit" /> + </form> +</div> + +<div class="row"> + <h2 class="heading">R/qtl2 Bundles</h2> + + <div class="explainer"> + <p>This feature combines and extends the two upload methods below. Instead of + uploading one item at a time, the R/qtl2 bundle you upload can contain both + the genotypes data (samples/individuals/cases and their data) and the + expression data.</p> + <p>The R/qtl2 bundle, additionally, can contain extra metadata, that neither + of the methods below can handle.</p> + + <a href="{{url_for('expression-data.rqtl2.select_species')}}" + title="Upload a zip bundle of R/qtl2 files"> + <button class="btn btn-primary">upload R/qtl2 bundle</button></a> + </div> +</div> +{%endblock%} diff --git a/uploader/templates/rqtl2/no-such-job.html b/uploader/templates/populations/rqtl2/no-such-job.html index b17004f..b17004f 100644 --- a/uploader/templates/rqtl2/no-such-job.html +++ b/uploader/templates/populations/rqtl2/no-such-job.html diff --git a/uploader/templates/rqtl2/rqtl2-job-error.html b/uploader/templates/populations/rqtl2/rqtl2-job-error.html index 9817518..9817518 100644 --- a/uploader/templates/rqtl2/rqtl2-job-error.html +++ b/uploader/templates/populations/rqtl2/rqtl2-job-error.html diff --git a/uploader/templates/rqtl2/rqtl2-job-results.html b/uploader/templates/populations/rqtl2/rqtl2-job-results.html index 4ecd415..4ecd415 100644 --- a/uploader/templates/rqtl2/rqtl2-job-results.html +++ b/uploader/templates/populations/rqtl2/rqtl2-job-results.html diff --git a/uploader/templates/rqtl2/rqtl2-job-status.html b/uploader/templates/populations/rqtl2/rqtl2-job-status.html index e896f88..e896f88 100644 --- a/uploader/templates/rqtl2/rqtl2-job-status.html +++ b/uploader/templates/populations/rqtl2/rqtl2-job-status.html diff --git a/uploader/templates/rqtl2/rqtl2-qc-job-error.html b/uploader/templates/populations/rqtl2/rqtl2-qc-job-error.html index 90e8887..90e8887 100644 --- a/uploader/templates/rqtl2/rqtl2-qc-job-error.html +++ b/uploader/templates/populations/rqtl2/rqtl2-qc-job-error.html diff --git a/uploader/templates/rqtl2/rqtl2-qc-job-results.html b/uploader/templates/populations/rqtl2/rqtl2-qc-job-results.html index b3c3a8f..b3c3a8f 100644 --- a/uploader/templates/rqtl2/rqtl2-qc-job-results.html +++ b/uploader/templates/populations/rqtl2/rqtl2-qc-job-results.html diff --git a/uploader/templates/rqtl2/rqtl2-qc-job-status.html b/uploader/templates/populations/rqtl2/rqtl2-qc-job-status.html index f4a6266..f4a6266 100644 --- a/uploader/templates/rqtl2/rqtl2-qc-job-status.html +++ b/uploader/templates/populations/rqtl2/rqtl2-qc-job-status.html diff --git a/uploader/templates/rqtl2/rqtl2-qc-job-success.html b/uploader/templates/populations/rqtl2/rqtl2-qc-job-success.html index f126835..f126835 100644 --- a/uploader/templates/rqtl2/rqtl2-qc-job-success.html +++ b/uploader/templates/populations/rqtl2/rqtl2-qc-job-success.html diff --git a/uploader/templates/populations/rqtl2/select-geno-dataset.html b/uploader/templates/populations/rqtl2/select-geno-dataset.html new file mode 100644 index 0000000..3233abc --- /dev/null +++ b/uploader/templates/populations/rqtl2/select-geno-dataset.html @@ -0,0 +1,69 @@ +{%extends "base.html"%} +{%from "flash_messages.html" import flash_messages%} + +{%block title%}Upload R/qtl2 Bundle{%endblock%} + +{%block contents%} +<h2 class="heading">Select Genotypes Dataset</h2> + +<div class="row"> + <p>Your R/qtl2 files bundle could contain a "geno" specification. You will + therefore need to select from one of the existing Genotype datasets or + create a new one.</p> + <p>This is the dataset where your data will be organised under.</p> +</div> + +<div class="row"> + <form id="frm-upload-rqtl2-bundle" + action="{{url_for('expression-data.rqtl2.select_geno_dataset', + species_id=species.SpeciesId, + population_id=population.InbredSetId)}}" + method="POST" + enctype="multipart/form-data"> + <legend class="heading">select from existing genotype datasets</legend> + + <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> + <input type="hidden" name="population_id" + value="{{population.InbredSetId}}" /> + <input type="hidden" name="rqtl2_bundle_file" + value="{{rqtl2_bundle_file}}" /> + + {{flash_messages("error-rqtl2-select-geno-dataset")}} + + <div class="form-group"> + <legend>Datasets</legend> + <label for="select:geno-datasets" class="form-label">Dataset</label> + <select id="select:geno-datasets" + name="geno-dataset-id" + required="required" + {%if datasets | length == 0%} + disabled="disabled" + {%endif%} + class="form-control" + aria-describedby="help-geno-dataset-select-dataset"> + <option value="">Select dataset</option> + {%for dset in datasets%} + <option value="{{dset['Id']}}">{{dset["Name"]}} ({{dset["FullName"]}})</option> + {%endfor%} + </select> + <span id="help-geno-dataset-select-dataset" class="form-text text-muted"> + Select from the existing genotype datasets for species + {{species.SpeciesName}} ({{species.FullName}}). + </span> + </div> + + <button type="submit" class="btn btn-primary">select dataset</button> + </form> +</div> + +<div class="row"> + <p>If the genotype dataset you need does not currently exist for your dataset, + go the <a href="{{url_for( + 'species.populations.genotypes.create_dataset', + species_id=species.SpeciesId, + population_id=population.Id)}}" + title="Create a new genotypes dataset for {{species.FullName}}"> + genotypes page to create the genotype dataset</a></p> +</div> + +{%endblock%} diff --git a/uploader/templates/populations/rqtl2/select-population.html b/uploader/templates/populations/rqtl2/select-population.html new file mode 100644 index 0000000..ded425f --- /dev/null +++ b/uploader/templates/populations/rqtl2/select-population.html @@ -0,0 +1,57 @@ +{%extends "expression-data/index.html"%} +{%from "flash_messages.html" import flash_messages%} +{%from "species/macro-display-species-card.html" import display_species_card%} + +{%block title%}Select Grouping/Population{%endblock%} + +{%block contents%} +<h1 class="heading">Select grouping/population</h1> + +<div class="row"> + <p>The data is organised in a hierarchical form, beginning with + <em>species</em> at the very top. Under <em>species</em> the data is + organised by <em>population</em>, sometimes referred to as <em>grouping</em>. + (In some really old documents/systems, you might see this referred to as + <em>InbredSet</em>.)</p> + <p>In this section, you get to define what population your data is to be + organised by.</p> +</div> + +<div class="row"> + <form method="POST" + action="{{url_for('expression-data.rqtl2.select_population', + species_id=species.SpeciesId)}}"> + <legend class="heading">select grouping/population</legend> + {{flash_messages("error-select-population")}} + + <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> + + <div class="form-group"> + <label for="select:inbredset" class="form-label">population</label> + <select id="select:inbredset" + name="inbredset_id" + required="required" + class="form-control"> + <option value="">Select a grouping/population</option> + {%for pop in populations%} + <option value="{{pop.InbredSetId}}"> + {{pop.InbredSetName}} ({{pop.FullName}})</option> + {%endfor%} + </select> + <span class="form-text text-muted">Select the population for your data from + the list below.</span> + </div> + + <button type="submit" class="btn btn-primary" />select population</button> +</form> +</div> + +{%endblock%} + +{%block sidebarcontents%} +{{display_species_card(species)}} +{%endblock%} + + +{%block javascript%} +{%endblock%} diff --git a/uploader/templates/rqtl2/select-probeset-dataset.html b/uploader/templates/populations/rqtl2/select-probeset-dataset.html index 74f8f69..74f8f69 100644 --- a/uploader/templates/rqtl2/select-probeset-dataset.html +++ b/uploader/templates/populations/rqtl2/select-probeset-dataset.html diff --git a/uploader/templates/rqtl2/select-probeset-study-id.html b/uploader/templates/populations/rqtl2/select-probeset-study-id.html index e3fd9cc..e3fd9cc 100644 --- a/uploader/templates/rqtl2/select-probeset-study-id.html +++ b/uploader/templates/populations/rqtl2/select-probeset-study-id.html diff --git a/uploader/templates/rqtl2/select-tissue.html b/uploader/templates/populations/rqtl2/select-tissue.html index fe3080a..fe3080a 100644 --- a/uploader/templates/rqtl2/select-tissue.html +++ b/uploader/templates/populations/rqtl2/select-tissue.html diff --git a/uploader/templates/rqtl2/summary-info.html b/uploader/templates/populations/rqtl2/summary-info.html index 0adba2e..0adba2e 100644 --- a/uploader/templates/rqtl2/summary-info.html +++ b/uploader/templates/populations/rqtl2/summary-info.html diff --git a/uploader/templates/rqtl2/upload-rqtl2-bundle-step-01.html b/uploader/templates/populations/rqtl2/upload-rqtl2-bundle-step-01.html index 9d45c5f..9d45c5f 100644 --- a/uploader/templates/rqtl2/upload-rqtl2-bundle-step-01.html +++ b/uploader/templates/populations/rqtl2/upload-rqtl2-bundle-step-01.html diff --git a/uploader/templates/rqtl2/upload-rqtl2-bundle-step-02.html b/uploader/templates/populations/rqtl2/upload-rqtl2-bundle-step-02.html index 8210ed0..8210ed0 100644 --- a/uploader/templates/rqtl2/upload-rqtl2-bundle-step-02.html +++ b/uploader/templates/populations/rqtl2/upload-rqtl2-bundle-step-02.html diff --git a/uploader/templates/rqtl2/create-geno-dataset-success.html b/uploader/templates/rqtl2/create-geno-dataset-success.html deleted file mode 100644 index bb6d63d..0000000 --- a/uploader/templates/rqtl2/create-geno-dataset-success.html +++ /dev/null @@ -1,55 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Upload R/qtl2 Bundle{%endblock%} - -{%block contents%} -<h2 class="heading">Select Genotypes Dataset</h2> - -<div class="explainer"> - <p>You successfully created the genotype dataset with the following - information. - <dl> - <dt>ID</dt> - <dd>{{geno_dataset.id}}</dd> - - <dt>Name</dt> - <dd>{{geno_dataset.name}}</dd> - - <dt>Full Name</dt> - <dd>{{geno_dataset.fname}}</dd> - - <dt>Short Name</dt> - <dd>{{geno_dataset.sname}}</dd> - - <dt>Created On</dt> - <dd>{{geno_dataset.today}}</dd> - - <dt>Public?</dt> - <dd>{%if geno_dataset.public == 0%}No{%else%}Yes{%endif%}</dd> - </dl> - </p> -</div> - -<div class="row"> - <form id="frm-upload-rqtl2-bundle" - action="{{url_for('expression-data.rqtl2.select_dataset_info', - species_id=species.SpeciesId, - population_id=population.InbredSetId)}}" - method="POST" - enctype="multipart/form-data"> - <legend class="heading">select from existing genotype datasets</legend> - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - <input type="hidden" name="population_id" - value="{{population.InbredSetId}}" /> - <input type="hidden" name="rqtl2_bundle_file" - value="{{rqtl2_bundle_file}}" /> - <input type="hidden" name="geno-dataset-id" - value="{{geno_dataset.id}}" /> - - <button type="submit" class="btn btn-primary">continue</button> - </form> -</div> - -{%endblock%} diff --git a/uploader/templates/rqtl2/create-probe-dataset-success.html b/uploader/templates/rqtl2/create-probe-dataset-success.html deleted file mode 100644 index 03b75c7..0000000 --- a/uploader/templates/rqtl2/create-probe-dataset-success.html +++ /dev/null @@ -1,59 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Upload R/qtl2 Bundle{%endblock%} - -{%block contents%} -<h2 class="heading">Create ProbeSet Dataset</h2> - -<div class="row"> - <p>You successfully created the ProbeSet dataset with the following - information. - <dl> - <dt>Averaging Method</dt> - <dd>{{avgmethod.Name}}</dd> - - <dt>ID</dt> - <dd>{{dataset.datasetid}}</dd> - - <dt>Name</dt> - <dd>{{dataset.name2}}</dd> - - <dt>Full Name</dt> - <dd>{{dataset.fname}}</dd> - - <dt>Short Name</dt> - <dd>{{dataset.sname}}</dd> - - <dt>Created On</dt> - <dd>{{dataset.today}}</dd> - - <dt>DataScale</dt> - <dd>{{dataset.datascale}}</dd> - </dl> - </p> -</div> - -<div class="row"> - <form id="frm-upload-rqtl2-bundle" - action="{{url_for('expression-data.rqtl2.select_dataset_info', - species_id=species.SpeciesId, - population_id=population.InbredSetId)}}" - method="POST" - enctype="multipart/form-data"> - <legend class="heading">Create ProbeSet dataset</legend> - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - <input type="hidden" name="population_id" - value="{{population.InbredSetId}}" /> - <input type="hidden" name="rqtl2_bundle_file" value="{{rqtl2_bundle_file}}" /> - <input type="hidden" name="geno-dataset-id" value="{{geno_dataset.Id}}" /> - <input type="hidden" name="tissueid" value="{{tissue.Id}}" /> - <input type="hidden" name="probe-study-id" value="{{study.Id}}" /> - <input type="hidden" name="probe-dataset-id" value="{{dataset.datasetid}}" /> - - <button type="submit" class="btn btn-primary">continue</button> - </form> -</div> - -{%endblock%} diff --git a/uploader/templates/rqtl2/create-probe-study-success.html b/uploader/templates/rqtl2/create-probe-study-success.html deleted file mode 100644 index e293f6f..0000000 --- a/uploader/templates/rqtl2/create-probe-study-success.html +++ /dev/null @@ -1,49 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Upload R/qtl2 Bundle{%endblock%} - -{%block contents%} -<h2 class="heading">Create ProbeSet Study</h2> - -<div class="row"> - <p>You successfully created the ProbeSet study with the following - information. - <dl> - <dt>ID</dt> - <dd>{{study.id}}</dd> - - <dt>Name</dt> - <dd>{{study.name}}</dd> - - <dt>Full Name</dt> - <dd>{{study.fname}}</dd> - - <dt>Short Name</dt> - <dd>{{study.sname}}</dd> - - <dt>Created On</dt> - <dd>{{study.today}}</dd> - </dl> - </p> - - <form id="frm-upload-rqtl2-bundle" - action="{{url_for('expression-data.rqtl2.select_dataset_info', - species_id=species.SpeciesId, - population_id=population.InbredSetId)}}" - method="POST" - enctype="multipart/form-data"> - <legend class="heading">Create ProbeSet study</legend> - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - <input type="hidden" name="population_id" - value="{{population.InbredSetId}}" /> - <input type="hidden" name="rqtl2_bundle_file" value="{{rqtl2_bundle_file}}" /> - <input type="hidden" name="geno-dataset-id" value="{{geno_dataset.Id}}" /> - <input type="hidden" name="probe-study-id" value="{{study.studyid}}" /> - - <button type="submit" class="btn btn-primary">continue</button> - </form> -</div> - -{%endblock%} diff --git a/uploader/templates/rqtl2/index.html b/uploader/templates/rqtl2/index.html deleted file mode 100644 index 8ce13bf..0000000 --- a/uploader/templates/rqtl2/index.html +++ /dev/null @@ -1,36 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Data Upload{%endblock%} - -{%block contents%} -<h1 class="heading">R/qtl2 data upload</h1> - -<h2>R/qtl2 Upload</h2> - -<form method="POST" action="{{url_for('expression-data.rqtl2.select_species')}}" - id="frm-rqtl2-upload"> - <legend class="heading">upload R/qtl2 bundle</legend> - {{flash_messages("error-rqtl2")}} - - <div class="form-group"> - <label for="select:species" class="form-label">Species</label> - <select id="select:species" - name="species_id" - required="required" - class="form-control"> - <option value="">Select species</option> - {%for spec in species%} - <option value="{{spec.SpeciesId}}">{{spec.MenuName}}</option> - {%endfor%} - </select> - <small class="form-text text-muted"> - Data that you upload to the system should belong to a know species. - Here you can select the species that you wish to upload data for. - </small> - </div> - - <button type="submit" class="btn btn-primary" />submit</button> -</form> - -{%endblock%} diff --git a/uploader/templates/rqtl2/select-geno-dataset.html b/uploader/templates/rqtl2/select-geno-dataset.html deleted file mode 100644 index 1db51e0..0000000 --- a/uploader/templates/rqtl2/select-geno-dataset.html +++ /dev/null @@ -1,144 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Upload R/qtl2 Bundle{%endblock%} - -{%block contents%} -<h2 class="heading">Select Genotypes Dataset</h2> - -<div class="row"> - <p>Your R/qtl2 files bundle contains a "geno" specification. You will - therefore need to select from one of the existing Genotype datasets or - create a new one.</p> - <p>This is the dataset where your data will be organised under.</p> -</div> - -<div class="row"> - <form id="frm-upload-rqtl2-bundle" - action="{{url_for('expression-data.rqtl2.select_geno_dataset', - species_id=species.SpeciesId, - population_id=population.InbredSetId)}}" - method="POST" - enctype="multipart/form-data"> - <legend class="heading">select from existing genotype datasets</legend> - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - <input type="hidden" name="population_id" - value="{{population.InbredSetId}}" /> - <input type="hidden" name="rqtl2_bundle_file" - value="{{rqtl2_bundle_file}}" /> - - {{flash_messages("error-rqtl2-select-geno-dataset")}} - - <div class="form-group"> - <legend>Datasets</legend> - <label for="select:geno-datasets" class="form-label">Dataset</label> - <select id="select:geno-datasets" - name="geno-dataset-id" - required="required" - {%if datasets | length == 0%} - disabled="disabled" - {%endif%} - class="form-control" - aria-describedby="help-geno-dataset-select-dataset"> - <option value="">Select dataset</option> - {%for dset in datasets%} - <option value="{{dset['Id']}}">{{dset["Name"]}} ({{dset["FullName"]}})</option> - {%endfor%} - </select> - <span id="help-geno-dataset-select-dataset" class="form-text text-muted"> - Select from the existing genotype datasets for species - {{species.SpeciesName}} ({{species.FullName}}). - </span> - </div> - - <button type="submit" class="btn btn-primary">select dataset</button> - </form> -</div> - -<div class="row"> - <p style="color:#FE3535; padding-left:20em; font-weight:bolder;">OR</p> -</div> - -<div class="row"> - <form id="frm-upload-rqtl2-bundle" - action="{{url_for('expression-data.rqtl2.create_geno_dataset', - species_id=species.SpeciesId, - population_id=population.InbredSetId)}}" - method="POST" - enctype="multipart/form-data"> - <legend class="heading">create a new genotype dataset</legend> - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - <input type="hidden" name="population_id" - value="{{population.InbredSetId}}" /> - <input type="hidden" name="rqtl2_bundle_file" - value="{{rqtl2_bundle_file}}" /> - - {{flash_messages("error-rqtl2-create-geno-dataset")}} - - <div class="form-group"> - <label for="txt:dataset-name" class="form-label">Name</label> - <input type="text" - id="txt:dataset-name" - name="dataset-name" - maxlength="100" - required="required" - class="form-control" - aria-describedby="help-geno-dataset-name" /> - <span id="help-geno-dataset-name" class="form-text text-muted"> - Provide the new name for the genotype dataset, e.g. "BXDGeno" - </span> - </div> - - <div class="form-group"> - <label for="txt:dataset-fullname" class="form-label">Full Name</label> - <input type="text" - id="txt:dataset-fullname" - name="dataset-fullname" - required="required" - maxlength="100" - class="form-control" - aria-describedby="help-geno-dataset-fullname" /> - - <span id="help-geno-dataset-fullname" class="form-text text-muted"> - Provide a longer name that better describes the genotype dataset, e.g. - "BXD Genotypes" - </span> - </div> - - <div class="form-group"> - <label for="txt:dataset-shortname" class="form-label">Short Name</label> - <input type="text" - id="txt:dataset-shortname" - name="dataset-shortname" - maxlength="100" - class="form-control" - aria-describedby="help-geno-dataset-shortname" /> - - <span id="help-geno-dataset-shortname" class="form-text text-muted"> - Provide a short name for the genotype dataset. This is optional. If not - provided, we'll default to the same value as the "Name" above. - </span> - </div> - - <div class="form-group"> - <input type="checkbox" - id="chk:dataset-public" - name="dataset-public" - checked="checked" - class="form-check" - aria-describedby="help-geno-datasent-public" /> - <label for="chk:dataset-public" class="form-check-label">Public?</label> - - <span id="help-geno-dataset-public" class="form-text text-muted"> - Specify whether the dataset will be available publicly. Check to make the - dataset publicly available and uncheck to limit who can access the dataset. - </span> - </div> - - <button type="submit" class="btn btn-primary">create new dataset</button> - </form> -</div> - -{%endblock%} diff --git a/uploader/templates/rqtl2/select-population.html b/uploader/templates/rqtl2/select-population.html deleted file mode 100644 index 7d27303..0000000 --- a/uploader/templates/rqtl2/select-population.html +++ /dev/null @@ -1,136 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} - -{%block title%}Select Grouping/Population{%endblock%} - -{%block contents%} -<h1 class="heading">Select grouping/population</h1> - -<div class="explainer"> - <p>The data is organised in a hierarchical form, beginning with - <em>species</em> at the very top. Under <em>species</em> the data is - organised by <em>population</em>, sometimes referred to as <em>grouping</em>. - (In some really old documents/systems, you might see this referred to as - <em>InbredSet</em>.)</p> - <p>In this section, you get to define what population your data is to be - organised by.</p> -</div> - -<form method="POST" - action="{{url_for('expression-data.rqtl2.select_population', species_id=species.SpeciesId)}}"> - <legend class="heading">select grouping/population</legend> - {{flash_messages("error-select-population")}} - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - - <div class="form-group"> - <label for="select:inbredset" class="form-label">population</label> - <select id="select:inbredset" - name="inbredset_id" - required="required" - class="form-control"> - <option value="">Select a grouping/population</option> - {%for pop in populations%} - <option value="{{pop.InbredSetId}}"> - {{pop.InbredSetName}} ({{pop.FullName}})</option> - {%endfor%} - </select> - <span class="form-text text-muted">If you are adding data to an already existing - population, simply pick the population from this drop-down selector. If - you cannot find your population from this list, try the form below to - create a new one..</span> - </div> - - <button type="submit" class="btn btn-primary" />select population</button> -</form> - -<p style="color:#FE3535; padding-left:20em; font-weight:bolder;">OR</p> - -<form method="POST" - action="{{url_for('expression-data.rqtl2.create_population', species_id=species.SpeciesId)}}"> - <legend class="heading">create new grouping/population</legend> - {{flash_messages("error-create-population")}} - - <input type="hidden" name="species_id" value="{{species.SpeciesId}}" /> - - <div class="form-group"> - <legend class="heading">mandatory</legend> - - <div class="form-group"> - <label for="txt:inbredset-name" class="form-label">name</label> - <input id="txt:inbredset-name" - name="inbredset_name" - type="text" - required="required" - maxlength="30" - placeholder="Enter grouping/population name" - class="form-control" /> - <span class="form-text text-muted">This is a short name that identifies the - population. Useful for menus, and quick scanning.</span> - </div> - - <div class="form-group"> - <label for="txt:" class="form-label">full name</label> - <input id="txt:inbredset-fullname" - name="inbredset_fullname" - type="text" - required="required" - maxlength="100" - placeholder="Enter the grouping/population's full name" - class="form-control" /> - <span class="form-text text-muted">This can be the same as the name above, or can - be longer. Useful for documentation, and human communication.</span> - </div> - </div> - - <div class="form-group"> - <legend class="heading">optional</legend> - - <div class="form-group"> - <label for="num:public" class="form-label">public?</label> - <select id="num:public" - name="public" - class="form-control"> - <option value="0">0 - Only accessible to authorised users</option> - <option value="1">1 - Publicly accessible to all users</option> - <option value="2" selected> - 2 - Publicly accessible to all users</option> - </select> - <span class="form-text text-muted">This determines whether the - population/grouping will appear on the menus for users.</span> - </div> - - <div class="form-group"> - <label for="txt:inbredset-family" class="form-label">family</label> - <input id="txt:inbredset-family" - name="inbredset_family" - type="text" - placeholder="I am not sure what this is about." - class="form-control" /> - <span class="form-text text-muted">I do not currently know what this is about. - This is a failure on my part to figure out what this is and provide a - useful description. Please feel free to remind me.</span> - </div> - - <div class="form-group"> - <label for="txtarea:" class="form-label">Description</label> - <textarea id="txtarea:description" - name="description" - rows="5" - placeholder="Enter a description of this grouping/population" - class="form-control"></textarea> - <span class="form-text text-muted"> - A long-form description of what the population consists of. Useful for - humans.</span> - </div> - </div> - - <button type="submit" class="btn btn-primary" /> - create grouping/population</button> -</form> - -{%endblock%} - - -{%block javascript%} -{%endblock%} diff --git a/uploader/templates/select_species.html b/uploader/templates/select_species.html deleted file mode 100644 index 1642401..0000000 --- a/uploader/templates/select_species.html +++ /dev/null @@ -1,92 +0,0 @@ -{%extends "base.html"%} -{%from "flash_messages.html" import flash_messages%} -{%from "upload_progress_indicator.html" import upload_progress_indicator%} - -{%block title%}expression data: select species{%endblock%} - -{%block contents%} -{{upload_progress_indicator()}} - -<h2 class="heading">expression data: select species</h2> - -<div class="row"> - <form action="{{url_for('expression-data.index.upload_file')}}" - method="POST" - enctype="multipart/form-data" - id="frm-upload-expression-data"> - <legend class="heading">upload expression data</legend> - {{flash_messages("error-expr-data")}} - - <div class="form-group"> - <label for="select_species01" class="form-label">Species</label> - <select id="select_species01" - name="speciesid" - required="required" - class="form-control"> - <option value="">Select species</option> - {%for aspecies in species%} - <option value="{{aspecies.SpeciesId}}">{{aspecies.MenuName}}</option> - {%endfor%} - </select> - </div> - - <div class="form-group"> - <legend class="heading">file type</legend> - - <div class="form-check"> - <input type="radio" name="filetype" value="average" id="filetype_average" - required="required" class="form-check-input" /> - <label for="filetype_average" class="form-check-label">average</label> - </div> - - <div class="form-check"> - <input type="radio" name="filetype" value="standard-error" - id="filetype_standard_error" required="required" - class="form-check-input" /> - <label for="filetype_standard_error" class="form-check-label"> - standard error - </label> - </div> - </div> - - <div class="form-group"> - <span id="no-file-error" class="alert-danger" style="display: none;"> - No file selected - </span> - <label for="file_upload" class="form-label">select file</label> - <input type="file" name="qc_text_file" id="file_upload" - accept="text/plain, text/tab-separated-values, application/zip" - class="form-control"/> - </div> - - <button type="submit" - class="btn btn-primary" - data-toggle="modal" - data-target="#upload-progress-indicator">upload file</button> - </form> -</div> -{%endblock%} - - -{%block javascript%} -<script type="text/javascript" src="static/js/upload_progress.js"></script> -<script type="text/javascript"> - function setup_formdata(form) { - var formdata = new FormData(); - formdata.append( - "speciesid", - form.querySelector("#select_species01").value) - formdata.append( - "qc_text_file", - form.querySelector("input[type='file']").files[0]); - formdata.append( - "filetype", - selected_filetype( - Array.from(form.querySelectorAll("input[type='radio']")))); - return formdata; - } - - setup_upload_handlers( - "frm-upload-expression-data", make_data_uploader(setup_formdata)); -</script> -{%endblock%} diff --git a/uploader/templates/species/macro-display-species-card.html b/uploader/templates/species/macro-display-species-card.html index 857c0f0..166c7b9 100644 --- a/uploader/templates/species/macro-display-species-card.html +++ b/uploader/templates/species/macro-display-species-card.html @@ -3,13 +3,19 @@ <div class="card-body"> <h5 class="card-title">Species</h5> <div class="card-text"> - <dl> - <dt>Common Name</dt> - <dd>{{species.SpeciesName}}</dd> + <table class="table"> + <tbody> + <tr> + <td>Common Name</td> + <td>{{species.SpeciesName}}</td> + </tr> - <dt>Scientific Name</dt> - <dd>{{species.FullName}}</dd> - </dl> + <tr> + <td>Scientific Name</td> + <td>{{species.FullName}}</td> + </tr> + </tbody> + </table> </div> </div> </div> diff --git a/uploader/templates/species/macro-select-species.html b/uploader/templates/species/macro-select-species.html index 6955134..dd086c0 100644 --- a/uploader/templates/species/macro-select-species.html +++ b/uploader/templates/species/macro-select-species.html @@ -1,8 +1,6 @@ {%macro select_species_form(form_action, species)%} {%if species | length > 0%} <form method="GET" action="{{form_action}}"> - <legend>Select Species</legend> - <div class="form-group"> <label for="select-species" class="form-label">Species</label> <select id="select-species" diff --git a/uploader/ui.py b/uploader/ui.py index 4115b02..1994056 100644 --- a/uploader/ui.py +++ b/uploader/ui.py @@ -8,6 +8,7 @@ def make_template_renderer(default): template, **{ **kwargs, + "activemenu": default, "activelink": kwargs.get("activelink", default) }) return render_template |