diff options
| author | Alexander_Kabui <alexanderkabui@proton.me> | 2026-10-13 15:11:33 +0300 |
|---|---|---|
| committer | Alexander_Kabui <alexanderkabui@proton.me> | 2026-10-14 07:27:03 +0300 |
| commit | 53c3f26ec2fdf209876a8a8eab561acc6fd755d4 (patch) | |
| tree | 25fc2be0d09e37da759be90fc3590f1496afaa89 /test/python/test_version.py | |
| parent | a6003ffbfe40aadaa1d9548bdc3294987b97d304 (diff) | |
| download | genenetwork4-53c3f26ec2fdf209876a8a8eab561acc6fd755d4.tar.gz | |
* src/indexer/sidecar.rs (sharding): one sidecar shard per dataset.
A monolithic sidecar rewrote and re-read the whole identity map each run; sharding it by dataset under <index>.sidecar.json plus <index>.sidecar.d/ lets a run load and rewrite only the shards it touches and keeps resident memory at O(largest dataset). * src/indexer/run.rs: crawl the SQL sources one dataset at a time through a sink and commit every 5000 documents instead of buffering the whole crawl; a bounded-channel variant was tried and reverted (several times slower). * src/indexer/rows.rs: order and group the crawl streams by dataset. * src/bin/gn-indexer.rs: follow the sharded sink wiring.
Diffstat (limited to 'test/python/test_version.py')
0 files changed, 0 insertions, 0 deletions
