You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "chenghuichen (via GitHub)" <gi...@apache.org> on 2026/04/19 10:54:57 UTC
[PR] [tantivy] Fix unnecessary .store file generation in full-text index [paimon]
chenghuichen opened a new pull request, #7670:
URL: https://github.com/apache/paimon/pull/7670
### Purpose
Tantivy is used purely as an inverted index in Paimon, so `.store` files are never needed or read. The original implementation mistakenly set `row_id` as `stored=True`, wasting ~26% of archive size per index file.
This PR removes `.set_stored()` from the schema and filters `.store` files when packing the archive, and updates the Python reader accordingly.
### Tests
Re-using `JavaPyTantivyE2ETest#testTantivyFullTextIndexWrite` +`test_read_tantivy_full_text_inde`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [tantivy] Fix unnecessary .store file generation in full-text index [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #7670:
URL: https://github.com/apache/paimon/pull/7670
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [tantivy] Fix unnecessary .store file generation in full-text index [paimon]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on code in PR #7670:
URL: https://github.com/apache/paimon/pull/7670#discussion_r3114853088
##########
paimon-python/pypaimon/globalindex/tantivy/tantivy_full_text_global_index_reader.py:
##########
@@ -152,13 +152,27 @@ def visit_full_text_search(self, full_text_search) -> Optional[ScoredGlobalIndex
searcher = self._searcher
query = self._index.parse_query(query_text, ["text"])
- results = searcher.search(query, limit)
+ scored_results = searcher.search(query, limit)
+ if not scored_results.hits:
+ return DictBasedScoredIndexResult({})
+
+ addr_to_score: Dict[tuple, float] = {
+ (addr.segment_ord, addr.doc): score
+ for score, addr in scored_results.hits
+ }
+
Review Comment:
This fallback looks like a potentially large performance regression for broad queries. We only need the `row_id` for the top-`limit` hits in `scored_results`, but this second search asks tantivy-py to collect up to `searcher.num_docs` matches ordered by `row_id`. For a common term on a large shard, a `limit=10` lookup can now degenerate into scanning/materializing almost the full match set just to recover 10 ids. Could we keep `row_id` stored until batch fast-field access is available in the shipped tantivy-py version, or add a direct fast-field read path instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [tantivy] Fix unnecessary .store file generation in full-text index [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on code in PR #7670:
URL: https://github.com/apache/paimon/pull/7670#discussion_r3115313164
##########
paimon-python/pypaimon/globalindex/tantivy/tantivy_full_text_global_index_reader.py:
##########
@@ -152,13 +152,27 @@ def visit_full_text_search(self, full_text_search) -> Optional[ScoredGlobalIndex
searcher = self._searcher
query = self._index.parse_query(query_text, ["text"])
- results = searcher.search(query, limit)
+ scored_results = searcher.search(query, limit)
+ if not scored_results.hits:
+ return DictBasedScoredIndexResult({})
+
+ addr_to_score: Dict[tuple, float] = {
+ (addr.segment_ord, addr.doc): score
+ for score, addr in scored_results.hits
+ }
+
Review Comment:
Thanks for the review! Based on feedback from the tantivy-py community, v0.26 with fast-field access should be released in a few weeks. I think it makes sense to hold this PR for now — I'll track the community progress and ping you once the code is updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [tantivy] Fix unnecessary .store file generation in full-text index [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on code in PR #7670:
URL: https://github.com/apache/paimon/pull/7670#discussion_r3115313164
##########
paimon-python/pypaimon/globalindex/tantivy/tantivy_full_text_global_index_reader.py:
##########
@@ -152,13 +152,27 @@ def visit_full_text_search(self, full_text_search) -> Optional[ScoredGlobalIndex
searcher = self._searcher
query = self._index.parse_query(query_text, ["text"])
- results = searcher.search(query, limit)
+ scored_results = searcher.search(query, limit)
+ if not scored_results.hits:
+ return DictBasedScoredIndexResult({})
+
+ addr_to_score: Dict[tuple, float] = {
+ (addr.segment_ord, addr.doc): score
+ for score, addr in scored_results.hits
+ }
+
Review Comment:
Thanks for the review! Based on feedback from the tantivy-py community, v0.26 with fast-field access should be released in a few weeks (see https://github.com/quickwit-oss/tantivy-py/pull/641). I think it makes sense to hold this PR for now — I'll track the community progress and ping you once the code is updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [tantivy] Fix unnecessary .store file generation in full-text index [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on PR #7670:
URL: https://github.com/apache/paimon/pull/7670#issuecomment-4384859633
`tantivy-py` 0.26 is out. Updated to use `fast_field_values()`, the workaround is removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [tantivy] Fix unnecessary .store file generation in full-text index [paimon]
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on PR #7670:
URL: https://github.com/apache/paimon/pull/7670#issuecomment-4389425035
@JingsongLi The [support_directory](https://github.com/JingsongLi/tantivy-py/tree/support_directory) branch is currently 29 commits behind upstream 0.26.0, which means it's missing the `fast_field_values` API that our code depends on. Could we rebase it onto 0.26.0?
Alternatively we could wait for 0.27 and switch to the official release — upstream is going to replace the custom directory approach with BlobStore (https://github.com/quickwit-oss/tantivy-py/pull/632). But the timeline is uncertain, and if this code ships before that, backward compatibility becomes a headache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [tantivy] Fix unnecessary .store file generation in full-text index [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #7670:
URL: https://github.com/apache/paimon/pull/7670#issuecomment-4525719517
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org