You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2026/09/14 04:52:46 UTC
[PR] [python] Search vector index shards concurrently [paimon]
TheR1sing3un opened a new pull request, #9805:
URL: https://github.com/apache/paimon/pull/9805
### Purpose
Vector index readers run their native search synchronously before returning a completed Future, so collecting those futures still opens and searches shards one at a time.
Add the Python query/table option `vector.search.parallelism` (positive integer, default `1`) to schedule complete shard open/search/close operations with a bounded thread pool for both single and batch vector queries. Results are merged in split order, persisted metric checks are synchronized without serializing index loading, and pending tasks are cancelled while started tasks finish cleanup on failure. Single-shard and serial searches keep the direct path.
### Tests
- `python -m pytest -q pypaimon/tests/vector_index_parallel_search_test.py pypaimon/tests/vector_search_filter_test.py pypaimon/tests/vector_metric_consistency_test.py pypaimon/tests/vindex_vector_index_test.py pypaimon/tests/lumina_vector_index_test.py pypaimon/tests/batch_vector_raw_scan_test.py`: 112 passed, 1 skipped.
- New coverage exercises concurrent index loading, worker limits, single/batch result ordering, offset pre-filters, metric mismatch and search-error cleanup, option validation, table/query precedence, and serial fast paths.
- Flake8, license-header checks, and `git diff --check` passed.
### Benchmark
macOS 26.4.1 arm64, Python 3.9.6, paimon-vindex 0.4.0, NumPy 2.0.2. Each native IVF-flat shard contains 4,096 vectors of 64 dimensions with 16 clusters. Queries use Top-K=10 and nprobe=4; per-shard Vindex I/O parallelism is fixed at 1.
The temporary harness runs the production index read/merge path against local index files. Timings include index opening, native initialization/search, result conversion, merging, and closing; table planning and scalar pre-filter evaluation are excluded. Each process performs one warm-up and five measured iterations; values below are medians. A delay of 1 ms is injected per positional read to isolate storage-latency sensitivity; this is not a live object-store measurement.
| Shards | Queries | Read delay | Master (ms) | Parallelism 1 (ms) | 2 (ms) | 4 (ms) | 8 (ms) |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 8 | 1 | 1 ms | 61.957 | 61.746 | 32.263 | 16.291 | 8.787 |
| 8 | 8 | 0 ms | 2.827 | 2.883 | 2.998 | 4.005 | 4.218 |
| 8 | 8 | 1 ms | 165.151 | 165.171 | 83.661 | 42.373 | 22.892 |
| 16 | 8 | 1 ms | 331.570 | 331.942 | 167.545 | 86.012 | 44.061 |
At 16 shards / 8 queries, parallelism 4 improves latency by 3.85x and parallelism 8 by 7.53x versus master. Peak process RSS for parallelism 1/2/4/8 was 139.8/139.6/153.6/155.7 MiB. All result IDs and score bytes matched the serial baseline, and open stream counts never exceeded the configured parallelism.
Local tiny searches do not benefit from additional threads, which motivates retaining the default of 1. A separate 50-iteration local single-query check measured 1.075 ms on master versus 1.085 ms with parallelism 1. Real gains depend on shard size, storage, native thread settings, and concurrent query load.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Search vector index shards concurrently [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9805:
URL: https://github.com/apache/paimon/pull/9805#discussion_r4011606527
##########
paimon-python/pypaimon/table/source/vector_search_read.py:
##########
@@ -451,6 +460,43 @@ def _eval_batch(self, row_range_start, row_range_end, vector_index_files,
future.add_done_callback(lambda _: reader.close())
return future
+ def _search_index_splits(self, splits, query, search_limit, pre_filters, batch=False):
+ # Native readers finish their search before returning a completed Future.
+ # Schedule the entire open/search/close operation, not just Future.result().
+ key = "vector.search.parallelism"
Review Comment:
Why introduce a new option? Any reusing from Java?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Search vector index shards concurrently [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9805:
URL: https://github.com/apache/paimon/pull/9805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Search vector index shards concurrently [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #9805:
URL: https://github.com/apache/paimon/pull/9805#discussion_r4011835987
##########
paimon-python/pypaimon/table/source/vector_search_read.py:
##########
@@ -451,6 +460,43 @@ def _eval_batch(self, row_range_start, row_range_end, vector_index_files,
future.add_done_callback(lambda _: reader.close())
return future
+ def _search_index_splits(self, splits, query, search_limit, pre_filters, batch=False):
+ # Native readers finish their search before returning a completed Future.
+ # Schedule the entire open/search/close operation, not just Future.result().
+ key = "vector.search.parallelism"
Review Comment:
reuse CoreOptions.GLOBAL_INDEX_THREAD_NUM now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org