You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/09/16 09:04:28 UTC
[I] [Bug] Vector search drops rows when the scalar index cannot fully evaluate the filter [paimon]
LuciferYang opened a new issue, #9883:
URL: https://github.com/apache/paimon/issues/9883
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
master
### Compute Engine
Any (data-evolution vector search: local / Spark / Flink)
### Minimal reproduce step
On a data-evolution table with a vector index, run a vector search whose WHERE filter the scalar global index cannot evaluate. Two shapes trigger it:
1. The filter uses a function the scalar index does not support, e.g. `IS NOT NULL` on a multivalue-indexed array column, or references a column that has no scalar index.
2. The filter is a partially-evaluable `AND`, e.g. `a = 1 AND tags IS NOT NULL` where `tags IS NOT NULL` is unsupported.
### What doesn't meet your expectations?
Rows that match the filter go missing from the result (shape 1), or the top-K is polluted by rows that do not match the filter and matching rows are displaced out of it (shape 2).
Root cause: vector search delegates pre-filtering to the scalar global index. `scalarMatchedRows` returned an empty bitmap when the index could not evaluate the predicate, and that bitmap was AND-ed into every index split, so every index-covered row was dropped. For a partially-evaluable `AND`, the evaluator drops the unsupported conjunct and returns a superset that the index path trusts as exact. The index read path has no final-read filter, so an inexact bitmap can neither be corrected nor ignored.
### Anything else?
No response
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Vector search drops rows when the scalar index cannot fully evaluate the filter [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on issue #9883:
URL: https://github.com/apache/paimon/issues/9883#issuecomment-5715366565
The separate Spark-boundary case (non-convertible residual on a truncated top-K) is filed as #9931.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Vector search drops rows when the scalar index cannot fully evaluate the filter [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on issue #9883:
URL: https://github.com/apache/paimon/issues/9883#issuecomment-5715345869
## Follow-up design for the partially-evaluable case
#9909 fixed the fully-unevaluable case: when the scalar global index cannot evaluate the predicate at all, `scalarMatchedRows` returns `null` and the ranges are routed to the raw search where the exact filter decides. This issue stays open for the partially-evaluable case.
### The remaining problem
When a WHERE filter has one conjunct the index can evaluate and one it cannot, `GlobalIndexEvaluator` silently drops the unevaluable conjunct and returns a non-null result that is a superset of the true matches. The vector index path trusts that bitmap as the exact include set (it has no final-read filter), so the top-K is computed over the superset: rows that fail the dropped conjunct can take top-K slots, and truly-matching rows ranked below them are lost.
Two shapes, both reproducible:
- different fields: `a = 1 AND tags IS NOT NULL`, where `IS NOT NULL` on a multivalue-indexed array is unsupported;
- same field: `id >= 0 AND CAST(id AS BIGINT) > 0`, where the CAST conjunct is unsupported.
### Field coverage is not sufficient
An earlier attempt gated "keep the index result" on whether every field the filter references contributed (`contributingFieldIds`). That is not enough: the same-field example has `contributingFieldIds = {id}` covering every referenced field, yet the result is still a superset because a second conjunct on `id` was dropped. Exactness is a property of the whole predicate evaluation, not of field coverage.
### Where it lives (per engine)
The gap is at the evaluator level and is engine-agnostic. Anything that reaches the vector-search `filter` is affected, and both example predicates do reach it:
- Spark pushes convertible predicates into the scan; `SparkV2FilterConverter` converts both `CAST(id AS BIGINT) > 0` (a cast transform) and `array IS NOT NULL`, so they reach the evaluator.
- Flink's `vector_search` procedure parses the whole `where` string into the Paimon predicate; supported functions reach the evaluator, and an inexpressible one such as CAST fails the call rather than silently dropping.
- Local and batch reads pass the full predicate directly.
One case is out of scope here: a genuinely non-convertible Spark predicate (a UDF, a column-to-column comparison, an unresolvable cast) stays as a Spark residual, never reaches Paimon, and Spark post-filters the already-truncated top-K, which cannot refill displaced rows. That is a separate Spark-boundary concern, not the "scalar index cannot evaluate" problem tracked here; I will file it separately.
### Proposed design
1. **Exactness signal in the evaluator.** Add an `exact` flag sourced from the reader/result (default exact; an approximate reader reports inexact), propagated bottom-up in `combineResults`: an AND is exact only if no child was dropped and every contributing child is exact; an OR is exact if all children are exact (it already requires all present); an empty result is exact. A boolean is sufficient. Sourcing it from the reader, rather than inferring it from "the reader returned a present result", keeps the guarantee under future approximate or ANN-style readers.
2. **Remediation, gated on the flag.** When the result is exact, use it as today. When it is inexact, keep the index (ANN) path but re-check: apply the supported part as a pre-filter, over-retrieve more than K candidates (the existing refine-factor path already over-retrieves), re-apply the exact filter on the candidate rows (the candidate read path can already read filter columns and run the exact predicate), take the top-K, and iteratively retrieve more if fewer than K survive. Fall back to the raw search when the residual is selective enough that refilling approaches a full scan. This keeps the index fast path for a selective supported conjunct and stays correct.
All four read paths (local, batch, Spark, Flink) inherit this from the shared base, so it is a single fix.
### Alternatives considered
- Route every inexact predicate to a full raw search, like the fully-unevaluable case. Simplest, but it loses the index acceleration for a selective supported conjunct, so it is kept only as the safety valve above.
- Gate on `contributingFieldIds`. Insufficient, as shown above.
Feedback welcome, especially on the `exact` flag versus a residual predicate, and on candidate re-check versus raw fallback as the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Vector search drops rows when the scalar index cannot fully evaluate the filter [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on issue #9883:
URL: https://github.com/apache/paimon/issues/9883#issuecomment-5710616065
Scope update: #9909 fixes the fully-unevaluable half of this issue (the scalar index cannot evaluate the predicate at all, so the old empty bitmap dropped every index-covered row). It intentionally does not close this issue.
Still open here: a partially-evaluable predicate keeps the index fast path and can return a wrong top-K, because the scalar evaluator drops the unsupported conjunct and returns a non-null superset. Two shapes:
- different fields, e.g. `a = 1 AND tags IS NOT NULL` where the array conjunct is unsupported;
- same field, e.g. `id >= 0 AND CAST(id AS BIGINT) > 0` where the CAST conjunct is unsupported.
Field coverage (`contributingFieldIds`) is not sufficient to prove exactness: the second shape covers every referenced field yet is still inexact. A fix needs an expression-level exactness signal or residual predicate from the evaluator, then either raw fallback or candidate re-checking with extra retrieval to refill the filtered top-K (filtering only the initial K would still miss qualifying rows displaced from that candidate set).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Vector search drops rows when the scalar index cannot fully evaluate the filter [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on issue #9883:
URL: https://github.com/apache/paimon/issues/9883#issuecomment-5966545700
@JingsongLi I think we can close this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Vector search drops rows when the scalar index cannot fully evaluate the filter [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #9883: [Bug] Vector search drops rows when the scalar index cannot fully evaluate the filter
URL: https://github.com/apache/paimon/issues/9883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Vector search drops rows when the scalar index cannot fully evaluate the filter [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #9883:
URL: https://github.com/apache/paimon/issues/9883#issuecomment-5709072480
Do not break performance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org