You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JunRuiLee (via GitHub)" <gi...@apache.org> on 2026/07/16 06:07:23 UTC
[PR] [core] Add primary-key vector search read integration [paimon-rust]
JunRuiLee opened a new pull request, #526:
URL: https://github.com/apache/paimon-rust/pull/526
> **Draft, stacked on #517.** This branch is based on #517 (`[core] Add primary-key vector read kernel`). Until #517 merges, the diff here includes #517's commits; it will shrink to the integration change once #517 lands and this is rebased.
### Purpose
Wire the crate-private PK-vector read kernel (#517) into the first production caller: real snapshot / index-manifest planning, the public vector-search routing branch, real ANN + exact reader dependencies, and best-first row materialization. This is the read-side integration counterpart of Apache Paimon Java apache/paimon#8579 (`PrimaryKeyVectorScan` / `PrimaryKeyVectorRead`), without the residual scalar predicate (apache/paimon#8603) and refine/rerank (apache/paimon#8591), which are follow-ups.
### Brief change log
- `spec/core_options.rs`: PK-vector index config options — columns / index-type / distance-metric accessors, with fail-loud validation.
- `vindex/pkvector`: thread the ANN segment and `GlobalIndexSearchMode::Fast` skip through `bucket_search`; extend `BucketAnnSegment` with the resolved path / file size / index-meta the real ANN adapter needs.
- `table/pk_vector_scan.rs`: `PkVectorScan` — snapshot-consistent `(partition, bucket)` planning plus index-manifest scan, producing `PkVectorSearchSplit`s.
- `table/pk_vector_data_file_reader.rs`: real positional exact `PkVectorReader` over the data-file vector column.
- `table/vector_search_builder.rs`: PK-vector routing branch in `execute_scored` / `execute_read`; real vindex ANN scorer; wires the orchestrator as the first production caller (removes the kernel's `dead_code` allowances).
- `table/pk_vector_orchestrator.rs`: `search_candidates` returns bucket-tagged candidates; split-scoped exact-reader factory.
### Tests
- `cargo test -p paimon` — synthetic unit/e2e coverage plus a narrow Java-written baseline fixture (`testdata/pkvector/baseline/`) read back end-to-end. `cargo fmt --all --check` and `cargo clippy -p paimon --lib --tests -- -D warnings` are clean.
### Out of scope (follow-ups)
- Residual scalar `WHERE` predicate (apache/paimon#8603).
- Refine / rerank (apache/paimon#8591).
### API and Format
No storage-format change. Adds the public PK-vector search routing on the existing vector-search builder; no existing public API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add primary-key vector search read integration [paimon-rust]
Posted by "JunRuiLee (via GitHub)" <gi...@apache.org>.
JunRuiLee closed pull request #526: [core] Add primary-key vector search read integration
URL: https://github.com/apache/paimon-rust/pull/526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add primary-key vector search read integration [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #526:
URL: https://github.com/apache/paimon-rust/pull/526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add primary-key vector search read integration [paimon-rust]
Posted by "JunRuiLee (via GitHub)" <gi...@apache.org>.
JunRuiLee commented on PR #526:
URL: https://github.com/apache/paimon-rust/pull/526#issuecomment-4993449026
Thanks for catching this. Fixed in `59900be`: `bucket_search` now validates ANN segment payload path uniqueness and rejects any source data file covered by multiple ANN segments, mirroring Java `PkVectorBucketIndexState`.
Also added focused tests:
- `test_rejects_duplicate_ann_segment_path`
- `test_rejects_source_file_covered_by_multiple_segments`
Validation passed:
- `cargo fmt --check`
- `cargo test -p paimon vindex::pkvector::bucket --lib`
- `cargo clippy -p paimon --all-targets -- -D warnings`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add primary-key vector search read integration [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #526:
URL: https://github.com/apache/paimon-rust/pull/526#issuecomment-4992716175
Rust directly groups ANN payloads by bucket, without validating the uniqueness of payload files—as Java’s `PkVectorBucketIndexState` does—or ensuring that the same source data file cannot be overwritten by two ANN segments at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org