diff options
| author | pi coding agent <pi@localhost> | 2026-10-15 20:44:47 +0000 |
|---|---|---|
| committer | Frederick Muriuki Muriithi <fredmanglis@gmail.com> | 2026-10-15 16:00:32 -0500 |
| commit | 930dfee072eeba7482fbd2828efd8fa3922b138f (patch) | |
| tree | 3f620daff3b1f263b79fba4fea00da6334ab3c95 /scripts/insert_samples.py | |
| parent | 43fc51a9191306ddb48d94518f6d85987d0fb3e6 (diff) | |
| download | gn-uploader-930dfee072eeba7482fbd2828efd8fa3922b138f.tar.gz | |
Implement multi-processed file statistics collection
Implement `file_stats_multiprocessing`, which streams the file lines into bounded chunks, processes each chunk in a worker process and merges the per-chunk statistics in file order so that the result matches `file_stats_singlethread`. The worker (`_process_chunk`) is a module-level function that only depends on picklable data (headers, configuration and the chunk of lines), so it works with any multiprocessing start method. Also fix the file-size threshold check in `compute_statistics`: it compared the size in megabytes against a threshold expressed in bytes, making the multi-processed branch effectively unreachable. Compare bytes to bytes instead.
Diffstat (limited to 'scripts/insert_samples.py')
0 files changed, 0 insertions, 0 deletions
