You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/07/01 02:53:11 UTC
[PR] [python] Support Tantivy full-text global index build [paimon]
JingsongLi opened a new pull request, #8403:
URL: https://github.com/apache/paimon/pull/8403
## Summary
Support building Tantivy full-text global indexes from PyPaimon and align the local Python reader with the Java Tantivy query semantics where possible.
## Changes
- Add a PyPaimon Tantivy full-text global index writer that produces Java-compatible archived Tantivy index files and sparse index option metadata.
- Reuse the generic Python global-index build path for Tantivy full-text indexes, including row-range sharding and non-string column validation.
- Align Python full-text query behavior with Java for match boost, fuzziness, boost/demote queries, boolean score composition, and explicit failures for unsupported Tantivy options.
- Fail fast when Python build is asked to use `tantivy.tokenizer=jieba`, because tantivy-py does not expose the same Rust `tantivy_jieba` tokenizer used by Java.
- Update full-text global-index docs with SQL/Python tabs for build/drop/query examples and document Python's `jieba` build limitation.
## Testing
- [x] `python -m pytest -q paimon-python/pypaimon/tests/global_index_build_test.py paimon-python/pypaimon/tests/vector_search_filter_test.py`
- [x] `python -m compileall -q paimon-python/pypaimon/globalindex/tantivy paimon-python/pypaimon/globalindex/create_global_index.py paimon-python/pypaimon/tests/global_index_build_test.py paimon-python/pypaimon/tests/vector_search_filter_test.py`
- [x] `git diff --check`
## Notes
PyPaimon can query Java/SQL-built `jieba` Tantivy indexes for match queries when the Python `jieba` package is installed, but Python index construction intentionally rejects `tantivy.tokenizer=jieba` to avoid silently producing a non-equivalent index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support Tantivy full-text global index build [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8403:
URL: https://github.com/apache/paimon/pull/8403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org