about summary refs log tree commit diff
path: root/scripts
AgeCommit message (Collapse)Author
2026-10-22Move scripts into the uploader packagepi coding agent
Move the top-level `scripts` package to `uploader/scripts` so that it becomes the `uploader.scripts` package. All files are moved unchanged; references are updated in a follow-up commit.
2026-10-15Deduplicate logging.Frederick Muriuki Muriithi
2026-10-15Relax line-argument annotations to any iterablepi coding agent
Accept `Iterable[str]` for the lines arguments instead of `Iterator[str]`, since lists of lines are valid inputs, and accept `Sequence[str]` for the sample names. This fixes the mypy type errors without changing behaviour.
2026-10-15Sort collected genotype symbols for deterministic outputpi coding agent
The genotype symbols were converted to a tuple straight from a set, so their order depended on the set's iteration order. That made the output (and the JSON emitted by the CLI) non-deterministic, and caused the multi-processed result to differ from the single-threaded result once symbols were merged across chunks. Sort the symbols so the output is stable and identical for both implementations. The set of symbols, which is what consumers care about, is unchanged.
2026-10-15Implement multi-processed file statistics collectionpi coding agent
Implement `file_stats_multiprocessing`, which streams the file lines into bounded chunks, processes each chunk in a worker process and merges the per-chunk statistics in file order so that the result matches `file_stats_singlethread`. The worker (`_process_chunk`) is a module-level function that only depends on picklable data (headers, configuration and the chunk of lines), so it works with any multiprocessing start method. Also fix the file-size threshold check in `compute_statistics`: it compared the size in megabytes against a threshold expressed in bytes, making the multi-processed branch effectively unreachable. Compare bytes to bytes instead.
2026-10-15Add chunk statistics merge helperspi coding agent
Add `_merge_chunk_stats` to combine the statistics from two consecutive chunks of a file while preserving file order, and `_merge_all_chunk_stats` to fold an iterable of chunk statistics into a single value. For transposed files, the markers of the last markers line win, falling back to the earlier chunk when a later chunk saw no markers line. For untransposed files, the markers are concatenated in order. Counters are summed, genotype symbols are unioned, dot-geno fields are combined (later values override earlier ones) and errors are concatenated in order.
2026-10-15Refactor: extract per-line accumulation helperpi coding agent
Move the per-line processing loop out of `file_stats_singlethread` into a reusable `_accumulate` helper that returns a `_ChunkStats` value. This makes the accumulation logic available to both the single-threaded and the multi-processed implementations and allows a single source of truth for the processing logic. The new module-level `_make_line_splitter` and `_make_is_skippable_line` helpers build the closures from plain configuration values, so callers no longer need to pass unpicklable lambdas around. `file_stats_singlethread` is re-expressed in terms of the extracted helpers and its signature simplified to `(fileconfigs, headers, lines)`.
2026-10-15Populate blank lines.Frederick Muriuki Muriithi
2026-10-15Bugfix: Previously comparing str to tuple.Frederick Muriuki Muriithi
Split the line in order to compare with tuple of headers.
2026-10-14Refactor design to allow processing file with multiple processes.Frederick Muriuki Muriithi
If the file is larger (in bytes) than the value set in the `FILE_SIZE_THRESHOLD` constant, then the file will be chunked and processed with multiple processes, using the multiprocessing library, otherwise, it will be processed in a single-thread process.
2026-10-14Log out error to help with troubleshooting failures.Frederick Muriuki Muriithi
2026-10-02Pre-process a transposed file.Frederick Muriuki Muriithi
2026-10-02Output as JSON string.Frederick Muriuki Muriithi
2026-10-02Narrow data collected about file(s).Frederick Muriuki Muriithi
2026-10-02Separate pre-processing of transposed and untransposed lines.Frederick Muriuki Muriithi
We pre-process the lines differently when working with transponed lines than we do for untransposed lines -- the code reflects that moving forward.
2026-10-02Refactor: Simplify code by reusing common functionality.Frederick Muriuki Muriithi
2026-10-02Update calls to CSV processing functions.Frederick Muriuki Muriithi
- Update the imports - Pass in function rather than comment-char value
2026-10-02Update `LineError` type.Frederick Muriuki Muriithi
2026-10-01Add basic header validation.Frederick Muriuki Muriithi
Validate whether there are columns with the names given on the CLI, and if not, add a message note to that effect.
2026-10-01Handle line-level errors.Frederick Muriuki Muriithi
2026-10-01Fix type errors.Frederick Muriuki Muriithi
2026-10-01Fix pylint issues to get a clean working state.Frederick Muriuki Muriithi
2026-10-01Use correct validator.Frederick Muriuki Muriithi
2026-09-29Add "Accept" header for content negotiation.Frederick Muriuki Muriithi
2026-09-28Allow negative values for phenotype traits.Frederick Muriuki Muriithi
2026-09-28Compute initial stats.Frederick Muriuki Muriithi
2026-09-28Add content-specific CLI options for genotype CSV files.Frederick Muriuki Muriithi
2026-09-28Add format-specific CLI options for CSV files.Frederick Muriuki Muriithi
2026-09-28BugFix: Myriad of minor bugfixes.Frederick Muriuki Muriithi
2026-09-28Process the untransposed files.Frederick Muriuki Muriithi
2026-09-28BugFix: Fix typing.Frederick Muriuki Muriithi
2026-09-28BugFix: Check for equality rather than membership. Update keys.Frederick Muriuki Muriithi
2026-09-28Add function to read file contents.Frederick Muriuki Muriithi
2026-09-28Bugfix: (Syntax) Remove commasFrederick Muriuki Muriithi
2026-09-28Add missing imports.Frederick Muriuki Muriithi
2026-09-28Initialise scripts for genotype uploads files' processing.Frederick Muriuki Muriithi
2026-09-04Ignore an issue caught by pylint.Frederick Muriuki Muriithi
2026-04-16Improve error messages.Frederick Muriuki Muriithi
To help users better understand what the problem with their data is, the errors need to be clearer to them. This commit fixes some error messages.
2026-04-16Fix bug with cascade of name checks.Frederick Muriuki Muriithi
The `meta["data_name"]` statement was leading to an exception in any case where the "data_name" key is not provided. This defeats the purpose of allowing either "dataname" or "data_name". This commit fixes that, while handling the case where neither one of the two are provided.
2026-04-15Update to error log, because it's an error, even though we return 0.Frederick Muriuki Muriithi
2026-04-15Move tracing details to INFO loglevel rather than debug.Frederick Muriuki Muriithi
We need to trace the auth process so as to help with debugging resource creation failures.
2026-04-15Pull helper function out of main().Frederick Muriuki Muriithi
2026-04-09Use multiple threads to speed up the computations.Frederick Muriuki Muriithi
2026-04-09Specify the --permu_output option to QTLReaperFrederick Muriuki Muriithi
Specify the --permu_output option to QTLReaper to prevent it from attempting to write to the working directory, which in the guix containers is a READONLY file system.
2026-04-09pylint: non-assigned expressions - silence linter.Frederick Muriuki Muriithi
2026-04-09Prompt user for a name for the resource object.Frederick Muriuki Muriithi
2026-03-25Add the DATASET-ID argument to command-line interfaces.Frederick Muriuki Muriithi
2026-03-11Check exit code of the Rust QTLReaper script.Frederick Muriuki Muriithi
2026-03-11Check whether files exist before attempting to operate on them.Frederick Muriuki Muriithi
2026-03-11Set "RUST_BACKTRACE" envvar if loglevel is DEBUG.Frederick Muriuki Muriithi