You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/06/18 10:48:33 UTC
[PR] [python] Support native batch vector search in Lumina reader [paimon]
XiaoHongbo-Hope opened a new pull request, #8280:
URL: https://github.com/apache/paimon/pull/8280
### Purpose
### Tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support native batch vector search in Lumina reader [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #8280:
URL: https://github.com/apache/paimon/pull/8280#discussion_r3452689603
##########
paimon-python/pypaimon/table/source/vector_search_read.py:
##########
@@ -191,32 +218,33 @@ def read_batch(self, splits):
return [GlobalIndexResult.create_empty() for _ in range(n)]
pre_filter = self._pre_filter(splits)
Review Comment:
> This PR currently conflicts with master, and this part needs a careful rebase rather than just taking the PR-side implementation. The current master split `read_batch` into indexed and raw paths: indexed splits are searched through `_eval`, raw splits are still scanned via `_read_raw_search`, and the two results are merged per query. This new native-batch path should be applied only to the indexed splits, using the per-index-split pre-filters, then still merge each query result with the raw fallback. Otherwise batch vector search will either drop raw-split rows or try to treat `RawVectorSearchSplit` as an indexed split.
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support native batch vector search in Lumina reader [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8280:
URL: https://github.com/apache/paimon/pull/8280#issuecomment-4775914945
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support native batch vector search in Lumina reader [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8280:
URL: https://github.com/apache/paimon/pull/8280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support native batch vector search in Lumina reader [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8280:
URL: https://github.com/apache/paimon/pull/8280#discussion_r3451147515
##########
paimon-python/pypaimon/table/source/vector_search_read.py:
##########
@@ -191,32 +218,33 @@ def read_batch(self, splits):
return [GlobalIndexResult.create_empty() for _ in range(n)]
pre_filter = self._pre_filter(splits)
Review Comment:
This PR currently conflicts with master, and this part needs a careful rebase rather than just taking the PR-side implementation. The current master split `read_batch` into indexed and raw paths: indexed splits are searched through `_eval`, raw splits are still scanned via `_read_raw_search`, and the two results are merged per query. This new native-batch path should be applied only to the indexed splits, using the per-index-split pre-filters, then still merge each query result with the raw fallback. Otherwise batch vector search will either drop raw-split rows or try to treat `RawVectorSearchSplit` as an indexed split.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org