about summary refs log tree commit diff
path: root/scripts/cli
diff options
context:
space:
mode:
authorpi coding agent <pi@localhost>2026-10-15 20:44:47 +0000
committerFrederick Muriuki Muriithi <fredmanglis@gmail.com>2026-10-15 16:00:32 -0500
commit930dfee072eeba7482fbd2828efd8fa3922b138f (patch)
tree3f620daff3b1f263b79fba4fea00da6334ab3c95 /scripts/cli
parent43fc51a9191306ddb48d94518f6d85987d0fb3e6 (diff)
downloadgn-uploader-930dfee072eeba7482fbd2828efd8fa3922b138f.tar.gz
Implement multi-processed file statistics collection
Implement `file_stats_multiprocessing`, which streams the file lines into
bounded chunks, processes each chunk in a worker process and merges the
per-chunk statistics in file order so that the result matches
`file_stats_singlethread`.

The worker (`_process_chunk`) is a module-level function that only depends on
picklable data (headers, configuration and the chunk of lines), so it works
with any multiprocessing start method.

Also fix the file-size threshold check in `compute_statistics`: it compared
the size in megabytes against a threshold expressed in bytes, making the
multi-processed branch effectively unreachable. Compare bytes to bytes
instead.
Diffstat (limited to 'scripts/cli')
0 files changed, 0 insertions, 0 deletions