| Age | Commit message (Collapse) | Author |
|
Move the top-level `scripts` package to `uploader/scripts` so that it becomes
the `uploader.scripts` package. All files are moved unchanged; references are
updated in a follow-up commit.
|
|
|
|
Accept `Iterable[str]` for the lines arguments instead of `Iterator[str]`,
since lists of lines are valid inputs, and accept `Sequence[str]` for the
sample names. This fixes the mypy type errors without changing behaviour.
|
|
The genotype symbols were converted to a tuple straight from a set, so their
order depended on the set's iteration order. That made the output (and the
JSON emitted by the CLI) non-deterministic, and caused the multi-processed
result to differ from the single-threaded result once symbols were merged
across chunks. Sort the symbols so the output is stable and identical for
both implementations. The set of symbols, which is what consumers care
about, is unchanged.
|
|
Implement `file_stats_multiprocessing`, which streams the file lines into
bounded chunks, processes each chunk in a worker process and merges the
per-chunk statistics in file order so that the result matches
`file_stats_singlethread`.
The worker (`_process_chunk`) is a module-level function that only depends on
picklable data (headers, configuration and the chunk of lines), so it works
with any multiprocessing start method.
Also fix the file-size threshold check in `compute_statistics`: it compared
the size in megabytes against a threshold expressed in bytes, making the
multi-processed branch effectively unreachable. Compare bytes to bytes
instead.
|
|
Add `_merge_chunk_stats` to combine the statistics from two consecutive
chunks of a file while preserving file order, and `_merge_all_chunk_stats`
to fold an iterable of chunk statistics into a single value.
For transposed files, the markers of the last markers line win, falling back
to the earlier chunk when a later chunk saw no markers line. For
untransposed files, the markers are concatenated in order. Counters are
summed, genotype symbols are unioned, dot-geno fields are combined (later
values override earlier ones) and errors are concatenated in order.
|
|
Move the per-line processing loop out of `file_stats_singlethread` into a
reusable `_accumulate` helper that returns a `_ChunkStats` value. This
makes the accumulation logic available to both the single-threaded and the
multi-processed implementations and allows a single source of truth for the
processing logic.
The new module-level `_make_line_splitter` and `_make_is_skippable_line`
helpers build the closures from plain configuration values, so callers no
longer need to pass unpicklable lambdas around.
`file_stats_singlethread` is re-expressed in terms of the extracted helpers
and its signature simplified to `(fileconfigs, headers, lines)`.
|
|
|
|
Split the line in order to compare with tuple of headers.
|
|
If the file is larger (in bytes) than the value set in the
`FILE_SIZE_THRESHOLD` constant, then the file will be chunked and
processed with multiple processes, using the multiprocessing library,
otherwise, it will be processed in a single-thread process.
|
|
|
|
|
|
|
|
|
|
We pre-process the lines differently when working with transponed
lines than we do for untransposed lines -- the code reflects that
moving forward.
|
|
|
|
- Update the imports
- Pass in function rather than comment-char value
|
|
|
|
Validate whether there are columns with the names given on the CLI,
and if not, add a message note to that effect.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
To help users better understand what the problem with their data is,
the errors need to be clearer to them. This commit fixes some error
messages.
|
|
The `meta["data_name"]` statement was leading to an exception in any
case where the "data_name" key is not provided. This defeats the
purpose of allowing either "dataname" or "data_name". This commit
fixes that, while handling the case where neither one of the two are
provided.
|
|
|
|
We need to trace the auth process so as to help with debugging
resource creation failures.
|
|
|
|
|
|
Specify the --permu_output option to QTLReaper to prevent it from
attempting to write to the working directory, which in the guix
containers is a READONLY file system.
|
|
|
|
|
|
|
|
|
|
|
|
|