You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JunRuiLee (via GitHub)" <gi...@apache.org> on 2026/07/20 12:30:14 UTC
[PR] [core] Batch primary-key vector search over a shared plan [paimon-rust]
JunRuiLee opened a new pull request, #560:
URL: https://github.com/apache/paimon-rust/pull/560
### Purpose
Part of #514, and the second of three PRs split out of the original combined change on the primary-key vector `execute_read` path. This PR adds **batch multi-query search**: N query vectors share one snapshot/manifest plan, segment preload, and opened readers. Single-query output stays byte-identical (single == batch-of-one).
> **Stacked PR.** Base is `pk-vec/1-streaming-exact-fallback` (#559). Until that lands on `main`, the diff here also contains its commit; review the two top commits in isolation, or wait for the parent to merge. Full stack:
> 1. streaming exact fallback — #559
> 2. **this PR** — batch multi-query
> 3. parallel search — #556 (stacked on this)
### Brief change log
- `feat(vindex)`: the bucket-local ANN search gains a batch entry so N queries share one pass over each ANN segment and each uncovered file. The ANN searcher builds the live-row mask once (query-independent), opens one reader per segment, and drives `visit_batch_vector_search`; the exact searcher scores all queries in one stream pass. A single query short-circuits to the existing single-query path so its result is unchanged. The batch ANN scorer dispatches Lumina vs. vindex readers per segment backend, matching the single-query path.
- `feat(table)`: `BatchVectorSearchBuilder` gains `with_filter`/`with_projection` and an `execute_read` returning one Arrow stream per query in input order (empty streams preserve arity; any query error fails the whole call). The builder factors the query-independent plan (snapshot, segment preload, residual allow-list, backend, ANN scorer) into a shared step reused across queries. The single-query `plan_and_search_pk_candidates` becomes a thin batch-of-one wrapper whose output is byte-identical. The scored `execute()` stays fail-loud on the primary-key path, and a data-evolution `execute_read` validates the target column exists and is a FLOAT vector column (`ARRAY`/`VECTOR`) up front so an unknown/scalar/non-float column fails loud instead of returning an empty stream.
### Tests
- Single == batch-of-one at every layer; per-query heaps independent under a shared residual/DV; arity preserved on an empty snapshot; malformed query, zero limit, and reserved/unknown/scalar/non-float columns fail loud (core + C FFI coverage).
- `cargo test -p paimon` green; `cargo build -p paimon-datafusion` clean; `cargo clippy -p paimon --all-targets -D warnings` and `cargo fmt --check` clean.
### API and Format
No on-disk format change, no new result columns. Single-query output is byte-identical. New public surface: `BatchVectorSearchBuilder::{with_filter, with_projection, execute_read}`.
### Documentation
Code comments only; no user-facing docs change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Batch primary-key vector search over a shared plan [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #560:
URL: https://github.com/apache/paimon-rust/pull/560#issuecomment-5035386414
**1. Missing authorization check**
> `BatchVectorSearchBuilder::execute()` no longer calls `CoreOptions::ensure_read_authorized()`, although the previous implementation performed this check before validation and empty-table fast paths. This method returns data-derived row IDs and scores outside `TableScan` / `TableRead`, so a table with `query-auth.enabled=true` can bypass the fail-closed read authorization through the batch scored-search API. Please restore the authorization check at the beginning of `execute()` and add a batch-specific regression test.
**2. `ARRAY<FLOAT>` with Lumina is incorrectly handled as vindex**
> `pk_vector_query_dimension()` unconditionally uses `VindexVectorIndexOptions` for every `ARRAY<FLOAT>` column. For a valid PK-vector configuration using `lumina` or `lumina-vector-ann`, this fails with `Unsupported vindex index type` before planning, including on an empty table. Please resolve the dimension according to the configured backend: use the effective `lumina.index.dimension` for Lumina and `VindexVectorIndexOptions` only for vindex backends. A regression test covering `ARRAY<FLOAT>` with Lumina would also be useful.
**3. Default projection no longer rejects reserved-column conflicts**
> The previous implementation validated the resolved field list after applying the projection, which also caught reserved-name conflicts in the default projection. The new resolver only validates explicitly requested projection names. As a result, a table containing a user column such as `__paimon_search_score`, `_PKEY_VECTOR_POSITION`, or `_ROW_ID` can silently pass on an empty result and may produce duplicate or incorrectly interpreted metadata columns for non-empty results. Please restore the post-resolution reserved-field validation for both the single-query and batch resolvers, along with the removed default-projection regression tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Batch primary-key vector search over a shared plan [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #560:
URL: https://github.com/apache/paimon-rust/pull/560
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Batch primary-key vector search over a shared plan [paimon-rust]
Posted by "JunRuiLee (via GitHub)" <gi...@apache.org>.
JunRuiLee commented on PR #560:
URL: https://github.com/apache/paimon-rust/pull/560#issuecomment-5036028466
Addressed in fd72a9a (fixes) and 5fd4157 (cleanup):
**1. Missing authorization check** — `BatchVectorSearchBuilder::execute()` now calls `ensure_read_authorized()` at the very top, before config validation and any fast path, matching `execute_read` and the single-query builder. A `query-auth.enabled=true` table previously returned empty results on an empty snapshot and bypassed the fail-closed guard; it now fails closed. Added a batch-specific regression test.
**2. `ARRAY<FLOAT>` with Lumina** — `pk_vector_query_dimension()` now resolves the dimension per the configured backend: Lumina via `LuminaVectorIndexOptions` (`lumina.index.dimension`), vindex/IVF via `VindexVectorIndexOptions`. A valid Lumina `ARRAY<FLOAT>` PK-vector column is no longer rejected as `Unsupported vindex index type` before planning (including on an empty table). Added a regression test covering `ARRAY<FLOAT>` + Lumina.
**3. Default projection reserved-column conflicts** — both `resolve_materialize_read_type` resolvers (single-query and batch) now validate the *resolved* field list against the reserved metadata names, so the default all-columns projection also rejects a colliding user column, not only names in an explicit projection. The explicit pre-check is kept so a requested reserved name that is not a real column still fails as a reserved-column error rather than "column not found". Restored the default-projection regression tests for both resolvers.
Separately, dropped the stale `_PKEY_VECTOR_SCORE` reserved name (5fd4157): it was the score column's pre-rename name and nothing emits it anymore — the injected metadata columns are `__paimon_search_score`, `_PKEY_VECTOR_POSITION`, and `_ROW_ID`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Batch primary-key vector search over a shared plan [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #560:
URL: https://github.com/apache/paimon-rust/pull/560#issuecomment-5034195415
- `ARRAY<FLOAT>` is a valid PK-vector column, but batch searches only perform query validation in advance for `VECTOR<FLOAT>`.
- For empty tables or empty execution plans, queries on `ARRAY<FLOAT>` that contain NaN or Inf will be returned as normal empty results rather than reporting an invalid input. The minimal regression I added consistently failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Batch primary-key vector search over a shared plan [paimon-rust]
Posted by "JunRuiLee (via GitHub)" <gi...@apache.org>.
JunRuiLee commented on PR #560:
URL: https://github.com/apache/paimon-rust/pull/560#issuecomment-5034753724
> * `ARRAY<FLOAT>` is a valid PK-vector column, but batch searches only perform query validation in advance for `VECTOR<FLOAT>`.
> * For empty tables or empty execution plans, queries on `ARRAY<FLOAT>` that contain NaN or Inf will be returned as normal empty results rather than reporting an invalid input. The minimal regression I added consistently failed.
Thanks, fixed. Batch PK-vector query validation now also covers `ARRAY<FLOAT>` before planning, using the configured index dimension, so NaN/Inf queries fail loud even for empty plans. Added a regression test for the empty `ARRAY<FLOAT>` table case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org