| Age | Commit message (Collapse) | Author | |
|---|---|---|---|
| 11 days | Add some phenocovar metadata. | Frederick Muriuki Muriithi | |
| 11 days | Fetch a jobs errors from its errors lists. | Frederick Muriuki Muriithi | |
| 11 days | Save phenocovar errors in redis as they are found. | Frederick Muriuki Muriithi | |
| 11 days | Pass the redis connection and fully qualified job id | Frederick Muriuki Muriithi | |
| Pass the redis connection on to the function used to build main since it might need to use a connection to redis. Also pass the computed fully qualified job id rather than recomputing it every time. Update dependent functions to take the new arguments. | |||
| 11 days | Cleanup linting and type (mypy) errors. | Frederick Muriuki Muriithi | |
| 12 days | Wrap everything in try-catch to handle errors gracefully. | Frederick Muriuki Muriithi | |
| Put everything in the build_main function within a try-catch block to ensure we capture all exceptions that might occur in different scripts and log them out. This helps with debugging errors in the asynchronous scripts. | |||
| 12 days | Add emphasis to important point. | Frederick Muriuki Muriithi | |
| 12 days | Fix typo. | Frederick Muriuki Muriithi | |
| 12 days | Send logs to STDOUT too. | Frederick Muriuki Muriithi | |
| 12 days | Launch job and redirect to job status page. | Frederick Muriuki Muriithi | |
| 12 days | Add UI to view status of the job as it runs. | Frederick Muriuki Muriithi | |
| 12 days | Return 0 on completion of QC processing. | Frederick Muriuki Muriithi | |
| 12 days | Change from 'pending' to 'started' on entry to script. | Frederick Muriuki Muriithi | |
| Update the status of the job from 'pending' to 'started' at the entry-point of the script. | |||
| 2024-11-22 | Remove local code that does db connection. | Frederick Muriuki Muriithi | |
| Reuse the code in gn-libs for connecting to the database, rather than using the replicated local code. | |||
| 2024-11-22 | Use gn-libs code for db connection. | Frederick Muriuki Muriithi | |
| Use the code in gn-libs to connect to the database, rather than a local module. | |||
| 2024-11-22 | Add gn-libs as a dependency. | Frederick Muriuki Muriithi | |
| 2024-10-24 | Reduce number of local variables created in function. | Frederick Muriuki Muriithi | |
| Use the values in the argparse.Namespace object directly to avoid creating many local variables. | |||
| 2024-10-24 | Add `jobid` to extraction directory | Frederick Muriuki Muriithi | |
| Add the job id to help "group" files that belong to the same job. | |||
| 2024-10-24 | Check pheno, phenose, and phenonum files with logging | Frederick Muriuki Muriithi | |
| Add logging to the `qc_pheno_file` function so that we get the messages pushed to redis for every file that is being checked for errors. | |||
| 2024-10-24 | Check phenocovar files with logging | Frederick Muriuki Muriithi | |
| Build a function-scope logger using the new `scripts.redis_logger.RedisMessageListHandler` log handler to log-out the messages for each phenocovar file being processed in a more thread-safe way. | |||
| 2024-10-24 | Fix linting and typing errors. | Frederick Muriuki Muriithi | |
| 2024-10-24 | Add a `--loglevel` argument to select loglevel in scripts | Frederick Muriuki Muriithi | |
| To enable selection of log levels within scripts, this commit adds the `--loglevel` argument to the list of arguments that can be passed to scripts. | |||
| 2024-10-24 | Parse entire namespace object rather than individual values. | Frederick Muriuki Muriithi | |
| 2024-10-24 | Move logger creation to `build_main` function | Frederick Muriuki Muriithi | |
| Since the module-level loggers are built mostly the same, move the creation of the logger to the more general function to reduce repetition. | |||
| 2024-10-24 | Add handler to send log messages to specific Redis list | Frederick Muriuki Muriithi | |
| Class `RedisMessageListHandler` builds a log handler that pushes any messages logged out to the redis list specified by its `fullyqualifiedkey` argument. | |||
| 2024-10-24 | Update Redis prefix settings. | Frederick Muriuki Muriithi | |
| 2024-10-24 | Don't compute settings in configuration file. | Frederick Muriuki Muriithi | |
| 2024-10-23 | Add documentation on background jobs. | Frederick Muriuki Muriithi | |
| 2024-10-22 | Document emergent general QC structure for CSV files. | Frederick Muriuki Muriithi | |
| 2024-10-22 | Refactor `qc_pheno_file` and reuse it for different file types. | Frederick Muriuki Muriithi | |
| The QC/QA steps taken by the `qc_pheno_file` function are very similar across the "pheno", "phenose" and "phenonum" files. This commit makes the `qc_pheno_file` function a higher-order function and we pass the file-type specific check(s) as a callable (function) to be used for the QC/QA process. | |||
| 2024-10-22 | Check for errors in `pheno` files. | Frederick Muriuki Muriithi | |
| 2024-10-21 | Check `phenocovar` files for errors. | Frederick Muriuki Muriithi | |
| 2024-10-21 | Add utilities for reading text files and csv files | Frederick Muriuki Muriithi | |
| The function `read_text_file(…)` will read the raw text file and return the content line by line. The function `read_csv_file(…)` builds on `read_text_file(…)` to read a file as a character-separated-values file. The field separator, and comment_char characters can be provided to customise how the CSV file is read. | |||
| 2024-10-17 | UI: Create background job when user uploads file. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Cleanup: Delete all extracted files after processing. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Leave TODO notes for what needs to be done. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Fetch samples from database | Frederick Muriuki Muriithi | |
| Fetch the samples from the database. These will be used to verify that the samples in the phenotype files already exist in the database and are valid. | |||
| 2024-10-17 | Undo transpose for any transposed files | Frederick Muriuki Muriithi | |
| To reduce the complexity involved in the processing of the files, we undo any transposition of the CSV files for those files that are marked as transposed. | |||
| 2024-10-17 | Pass new arguments to QC function. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Add `speciesid` and `populationid` arguments to the script. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Add the working directory argument to the script. | Frederick Muriuki Muriithi | |
| Add a `--working-dir` argument to allow passing a directory where the script process the files within. If not provided, it defaults to a directory within the systems temporary directory. | |||
| 2024-10-17 | Extract the R/qtl2 bundle for processing. | Frederick Muriuki Muriithi | |
| To enable processing of the files individually, this commit will enable the extraction of the files into a known working directory in which all further processing will take place. | |||
| 2024-10-17 | Extract common functions. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Save errors for each file in lists. Parallelise error checking. | Frederick Muriuki Muriithi | |
| * Save the errors for each file in a redis list for that file. * Make error checking parallel, i.e. ensure every file of a particular type is checked completely independent of other files of the same type. | |||
| 2024-10-17 | Rewrite the QC code for R/qtl2 | Frederick Muriuki Muriithi | |
| 2024-10-17 | Ignore comment lines. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Implement sha256 hashing over file contents. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Add test for function to compute file hash. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Rename test module. | Frederick Muriuki Muriithi | |
| 2024-10-17 | mypy: Fix some typing issues. | Frederick Muriuki Muriithi | |
 |
index : gn-uploader | |
| gn-uploader: Data uploader for GeneNetwork |
| aboutsummaryrefslogtreecommitdiff |