| Age | Commit message (Collapse) | Author | |
|---|---|---|---|
| 12 days | Save 'pheno' errors in redis as they are found. | Frederick Muriuki Muriithi | |
| 12 days | Add some phenocovar metadata. | Frederick Muriuki Muriithi | |
| 12 days | Save phenocovar errors in redis as they are found. | Frederick Muriuki Muriithi | |
| 12 days | Pass the redis connection and fully qualified job id | Frederick Muriuki Muriithi | |
| Pass the redis connection on to the function used to build main since it might need to use a connection to redis. Also pass the computed fully qualified job id rather than recomputing it every time. Update dependent functions to take the new arguments. | |||
| 12 days | Wrap everything in try-catch to handle errors gracefully. | Frederick Muriuki Muriithi | |
| Put everything in the build_main function within a try-catch block to ensure we capture all exceptions that might occur in different scripts and log them out. This helps with debugging errors in the asynchronous scripts. | |||
| 13 days | Fix typo. | Frederick Muriuki Muriithi | |
| 13 days | Send logs to STDOUT too. | Frederick Muriuki Muriithi | |
| 13 days | Return 0 on completion of QC processing. | Frederick Muriuki Muriithi | |
| 13 days | Change from 'pending' to 'started' on entry to script. | Frederick Muriuki Muriithi | |
| Update the status of the job from 'pending' to 'started' at the entry-point of the script. | |||
| 2024-11-22 | Use gn-libs code for db connection. | Frederick Muriuki Muriithi | |
| Use the code in gn-libs to connect to the database, rather than a local module. | |||
| 2024-10-24 | Reduce number of local variables created in function. | Frederick Muriuki Muriithi | |
| Use the values in the argparse.Namespace object directly to avoid creating many local variables. | |||
| 2024-10-24 | Add `jobid` to extraction directory | Frederick Muriuki Muriithi | |
| Add the job id to help "group" files that belong to the same job. | |||
| 2024-10-24 | Check pheno, phenose, and phenonum files with logging | Frederick Muriuki Muriithi | |
| Add logging to the `qc_pheno_file` function so that we get the messages pushed to redis for every file that is being checked for errors. | |||
| 2024-10-24 | Check phenocovar files with logging | Frederick Muriuki Muriithi | |
| Build a function-scope logger using the new `scripts.redis_logger.RedisMessageListHandler` log handler to log-out the messages for each phenocovar file being processed in a more thread-safe way. | |||
| 2024-10-24 | Fix linting and typing errors. | Frederick Muriuki Muriithi | |
| 2024-10-24 | Parse entire namespace object rather than individual values. | Frederick Muriuki Muriithi | |
| 2024-10-24 | Move logger creation to `build_main` function | Frederick Muriuki Muriithi | |
| Since the module-level loggers are built mostly the same, move the creation of the logger to the more general function to reduce repetition. | |||
| 2024-10-22 | Refactor `qc_pheno_file` and reuse it for different file types. | Frederick Muriuki Muriithi | |
| The QC/QA steps taken by the `qc_pheno_file` function are very similar across the "pheno", "phenose" and "phenonum" files. This commit makes the `qc_pheno_file` function a higher-order function and we pass the file-type specific check(s) as a callable (function) to be used for the QC/QA process. | |||
| 2024-10-22 | Check for errors in `pheno` files. | Frederick Muriuki Muriithi | |
| 2024-10-21 | Check `phenocovar` files for errors. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Cleanup: Delete all extracted files after processing. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Leave TODO notes for what needs to be done. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Fetch samples from database | Frederick Muriuki Muriithi | |
| Fetch the samples from the database. These will be used to verify that the samples in the phenotype files already exist in the database and are valid. | |||
| 2024-10-17 | Undo transpose for any transposed files | Frederick Muriuki Muriithi | |
| To reduce the complexity involved in the processing of the files, we undo any transposition of the CSV files for those files that are marked as transposed. | |||
| 2024-10-17 | Pass new arguments to QC function. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Add `speciesid` and `populationid` arguments to the script. | Frederick Muriuki Muriithi | |
| 2024-10-17 | Add the working directory argument to the script. | Frederick Muriuki Muriithi | |
| Add a `--working-dir` argument to allow passing a directory where the script process the files within. If not provided, it defaults to a directory within the systems temporary directory. | |||
| 2024-10-17 | Extract the R/qtl2 bundle for processing. | Frederick Muriuki Muriithi | |
| To enable processing of the files individually, this commit will enable the extraction of the files into a known working directory in which all further processing will take place. | |||
| 2024-10-17 | Extract common functions. | Frederick Muriuki Muriithi | |
| 2024-10-14 | Initialise background script for running QC on phenotype bundles. | Frederick Muriuki Muriithi | |
| 2024-10-14 | BugFix: Use provided prefix | Frederick Muriuki Muriithi | |
| Use the provided prefix rather than calling `jobs.jobsnamespace()` function that depends of an app context existing. | |||
| 2024-10-14 | Make addition of arguments independent of each other. | Frederick Muriuki Muriithi | |
| 2024-08-16 | Log out correct parameters. | Frederick Muriuki Muriithi | |
| 2024-08-13 | Bug: cross reference with NULL cM when "gmap" file is absent | Frederick Muriuki Muriithi | |
| The "gmap" file might not exist in some bundles. In those instances, cross-reference the data without including the genotypes' physical positions (cM). | |||
| 2024-08-08 | Fix bugs and pass in logger to functions. | Frederick Muriuki Muriithi | |
| 2024-08-06 | Pass logger on to inner functions | Frederick Muriuki Muriithi | |
| Pass the logger forward to inner functions to help with debugging things. | |||
| 2024-07-25 | Rename module: qc_app --> uploader | Frederick Muriuki Muriithi | |
| 2024-07-05 | bug: Return a hashable key, not a dict. | Frederick Muriuki Muriithi | |
| 2024-07-02 | Call correct method. | Frederick Muriuki Muriithi | |
| 2024-07-02 | Ensure no duplicated values for the query. | Frederick Muriuki Muriithi | |
| 2024-04-03 | Reduce size of data inserted per query | Frederick Muriuki Muriithi | |
| Reduce the size of data inserted per query since MariDB allows a packet with a maximum size of 1GB. This should hopefully resolve the …OperationalError: (2006, 'Server has gone away') error. | |||
| 2024-02-15 | Filter out NULL values | Frederick Muriuki Muriithi | |
| 2024-02-05 | Fix linting and type errors. | Frederick Muriuki Muriithi | |
| 2024-01-23 | Update scripts to use redis prefix. | Frederick Muriuki Muriithi | |
| 2024-01-22 | scripts: Pass in logger to get detailed updates. | Frederick Muriuki Muriithi | |
| 2024-01-19 | scripts: Process R/qtl2 bundle | Frederick Muriuki Muriithi | |
| Build script to start the processing of the R/qtl2 bundle. | |||
| 2024-01-16 | Filter out items with values of `None`. | Frederick Muriuki Muriithi | |
| The ProbeSetData table does not allow `NULL` for the `value` column for good reason - if there is no value, just don't have a record for that particular sample. | |||
| 2024-01-15 | Update imports: `take` was moved to different package. | Frederick Muriuki Muriithi | |
| 2024-01-15 | Process the `pheno` files. | Frederick Muriuki Muriithi | |
| Process the data in the pheno files and insert it into the database. | |||
| 2024-01-15 | Initialise `install_phenos` script. | Frederick Muriuki Muriithi | |
 |
index : gn-uploader | |
| gn-uploader: Data uploader for GeneNetwork |
| aboutsummaryrefslogtreecommitdiff |