You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JunRuiLee (via GitHub)" <gi...@apache.org> on 2026/07/20 05:24:18 UTC
[PR] [core] Rerank primary-key vector candidates with exact distances [paimon-rust]
JunRuiLee opened a new pull request, #550:
URL: https://github.com/apache/paimon-rust/pull/550
### Purpose
Part of #514. Adds **exact rerank** to primary-key vector search, mirroring Java `PrimaryKeyVectorRead`'s refine/rerank. On a PK-vector table, `execute_read()` materializes rows in file-local physical coordinates; this lets a configured refine factor over-fetch approximate (ANN) candidates and reorder them by exact distance for higher recall, while leaving the no-refine path byte-identical.
Stacked on #537 (C FFI materialized read); its two commits appear in this diff until #537 merges. Review/merge after #537.
### Brief change log
- **Dual-limit bucket search**: `bucket_search` returns separate indexed/exact candidate lists, each bounded independently — the ANN/indexed channel over-fetches (`indexed_limit = limit × refine_factor`) while the exact-fallback channel stays bounded by the final `limit`. `PkVectorOrchestrator::search_candidates` takes the dual limits and returns an `OrchestratorSearchResult`; `merge_candidates` runs a global Top-K over the union of the two channels.
- **Rerank**: when a positive refine factor is configured, `rerank_indexed_positional` rereads the over-fetched ANN candidates' vectors from their data files by file-local position (reusing the position read path), recomputes exact distance, and re-runs the global Top-K. Exact-fallback candidates are already exact and are not reranked.
- **Refine-factor resolution** reuses the existing option-key precedence (`refine_factor` / `refine-factor` / `rerank_factor` / `rerank-factor`, with `fields.<col>.` → index-type → `ivf.` → bare prefix order); factor 0 (unset) leaves `indexed_limit == limit`, so output is byte-identical to the no-rerank path.
- **Fail-loud validation**: `append_batch_vectors` rejects null child elements in vector arrays (never a defaulted 0.0), `build_indexed_splits` validates `split_index` is in range before indexing, and the rerank kernel fails loud on ownership/dedup/NULL/dimension/file-not-in-plan violations — never a panic or a silently corrupted result.
### Tests
- Dual-limit bucket behavior; `merge_candidates` union Top-K; rerank kernel (ownership order, dedup, NULL element, dimension mismatch, file-not-in-plan); end-to-end refine integration matching an exact brute-force ground truth.
- **Regression invariant**: `refine_factor == 0` preserves the exact pre-rerank output (baseline + Java-fixture read-back tests unchanged).
- Fail-loud coverage: null vector element, out-of-range `split_index`.
- `cargo test -p paimon` green (1803 tests); `cargo build -p paimon-datafusion` clean (cross-crate `Send` gate); `cargo clippy -D warnings` and `cargo fmt --check` clean.
### API and Format
No on-disk format change, no new result columns. `refine_factor == 0` (the default) is byte-identical to the current output.
### Documentation
Code comments only; no user-facing docs change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rerank primary-key vector candidates with exact distances [paimon-rust]
Posted by "JunRuiLee (via GitHub)" <gi...@apache.org>.
JunRuiLee commented on code in PR #550:
URL: https://github.com/apache/paimon-rust/pull/550#discussion_r3612957999
##########
crates/paimon/src/table/vector_search_builder.rs:
##########
@@ -557,12 +566,29 @@ impl<'a> VectorSearchBuilder<'a> {
},
);
- let candidates = PkVectorOrchestrator::new(reader)
+ // Resolve the refine factor from the query options first, then fall back to
+ // the table options; a positive factor over-fetches indexed (approximate)
+ // candidates so the exact rerank below has a wider pool to reorder. Factor 0
+ // (unset) leaves `indexed_limit == limit`, byte-identical to the no-rerank
+ // path. The two option maps are kept distinct (query options passed
+ // separately from table options) so a broad query key cannot be overridden
+ // by a more specific table key: query options take precedence as a whole.
+ // `search_options` above is the merged view used only to drive the ANN read.
+ let refine_factor = configured_refine_factor(
Review Comment:
Fixed in 1ab7337. `configured_refine_factor` / `indexed_search_limit` are now resolved **before** `PkVectorScan::plan()` and the empty-plan early return, so an invalid factor fails loud regardless of whether the table currently has searchable data — matching Java, which resolves this in the `PrimaryKeyVectorRead` constructor before planning. Added an empty-table regression test covering both invalid forms you named (`0` → "must be positive", and a non-integer → "must be an integer") and both option sources (query and table options, since the fix moved resolution of both ahead of planning).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rerank primary-key vector candidates with exact distances [paimon-rust]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #550:
URL: https://github.com/apache/paimon-rust/pull/550#discussion_r3612677337
##########
crates/paimon/src/table/vector_search_builder.rs:
##########
@@ -557,12 +566,29 @@ impl<'a> VectorSearchBuilder<'a> {
},
);
- let candidates = PkVectorOrchestrator::new(reader)
+ // Resolve the refine factor from the query options first, then fall back to
+ // the table options; a positive factor over-fetches indexed (approximate)
+ // candidates so the exact rerank below has a wider pool to reorder. Factor 0
+ // (unset) leaves `indexed_limit == limit`, byte-identical to the no-rerank
+ // path. The two option maps are kept distinct (query options passed
+ // separately from table options) so a broad query key cannot be overridden
+ // by a more specific table key: query options take precedence as a whole.
+ // `search_options` above is the merged view used only to drive the ANN read.
+ let refine_factor = configured_refine_factor(
Review Comment:
`configured_refine_factor` is reached only after `PkVectorScan::plan()` and the `plan.splits.is_empty()` early return above. As a result, an empty PK-vector table accepts `refine_factor=0` or `refine_factor=abc` and returns a normal empty reader, while the identical query starts failing as soon as the table has a searchable split. Java resolves this option in the `PrimaryKeyVectorRead` constructor before planning, and configuration validity should not depend on whether data currently exists. Please resolve the factor/search limit before the empty-plan return (ideally before planning/index I/O) and add an empty-plan regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Rerank primary-key vector candidates with exact distances [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #550:
URL: https://github.com/apache/paimon-rust/pull/550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org