You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/06/02 07:11:21 UTC
[PR] [tantivy] Support tokenizer options for full text index [paimon]
JingsongLi opened a new pull request, #8074:
URL: https://github.com/apache/paimon/pull/8074
## Summary
This PR adds configurable tokenizer support for Tantivy full-text global indexes, including ngram tokenization for short fragment matching and Jieba tokenization for Chinese text.
## Changes
- Add persisted Tantivy full-text tokenizer options for `default`, `ngram`, and `jieba`, including ngram gram bounds, prefix-only mode, and lower-case behavior.
- Wire tokenizer options through Java global index creation, JNI writer/searcher APIs, Rust Tantivy analyzers, and metadata serialization.
- Teach PyPaimon to read tokenizer metadata, register the ngram analyzer, and query Jieba-tokenized indexes through Python `jieba` when installed.
- Extend Java/Python interoperability and tokenizer option coverage, including ngram and Jieba scenarios.
- Document tokenizer configuration in global index, PyPaimon, CLI, Flink, Spark, and Tantivy README docs.
## Testing
- `cargo check` in `paimon-tantivy/paimon-tantivy-jni/rust`
- `mvn -pl paimon-tantivy/paimon-tantivy-index -am -Pfast-build -DfailIfNoTests=false -Dflink.forkCount=0 -Dtest=TantivyFullTextIndexOptionsTest test -q`
- `PYTHONPATH=. /Users/lijingsong/.venvs/paimon-py311/bin/pytest pypaimon/tests/vector_search_filter_test.py::TantivyFullTextIndexOptionsTest -q`
- `/Users/lijingsong/.venvs/paimon-py311/bin/flake8 --config=./dev/cfg.ini pypaimon/globalindex/tantivy/tantivy_full_text_global_index_reader.py pypaimon/tests/e2e/java_py_read_write_test.py pypaimon/tests/vector_search_filter_test.py`
- `git diff --check origin/master..HEAD`
## Notes
- PyPaimon Jieba query support requires the optional Python `jieba` package. It is not added as a hard dependency.
- A full `./dev/lint-python.sh -i flake8` run was attempted, but the local process was killed by the OS (`Killed: 9`) before producing lint findings; the Python files changed by this PR pass flake8 directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [tantivy] Support configurable full-text analyzers [paimon]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on PR #8074:
URL: https://github.com/apache/paimon/pull/8074#issuecomment-4608653230
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [tantivy] Support configurable full-text analyzers [paimon]
Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on PR #8074:
URL: https://github.com/apache/paimon/pull/8074#issuecomment-4601775856
## Review Summary
Overall well-designed PR — tokenizer names validated against explicit allowlists at all three layers (Java, Rust, Python), good defense in depth.
### Must Fix
**JSON injection in JNI convenience constructors** (`TantivyIndexWriter.java:42-52`, `TantivySearcher.java:147-159`)
The 6-arg `TantivyIndexWriter` constructor and `TantivySearcher.defaultConfigJson()` build JSON via raw string concatenation without escaping `tokenizerName`:
```java
"{\"tokenizer\":\"" + tokenizerName + "\",\"ngramMinGram\":" + ...
```
A crafted tokenizer name like `simple","ngramMaxGram":999999,"tokenizer":"ngram` would inject arbitrary fields into the JSON parsed by Rust's serde (last key wins by default), bypassing Java-side validation entirely.
The main production path through `TantivyFullTextIndexOptions.toNativeConfigJson()` properly escapes via `appendJsonString()`, but these convenience constructors are public API and bypass it.
**Suggested fix:** Route all JSON construction through `toNativeConfigJson()`, or remove the convenience constructors and force callers through `TantivyFullTextIndexOptions`.
### Nice to Have (not blocking)
1. **No upper bound on `ngramMaxGram` / `maxTokenLength`** — validated as `> 0` but no ceiling. `max-gram=1000000` would generate O(text_length × max_gram) tokens. Consider adding caps (e.g., max_gram ≤ 10, max_token_length ≤ 1024).
2. **Cross-language tokenizer_name contract** — the `tokenizer_name()` / `_needs_custom` logic is independently implemented in Rust and Python. They agree today, but a comment documenting the contract would help prevent drift.
3. **Duplicated JSON construction** — `TantivyIndexWriter` and `TantivySearcher` both manually build the same JSON template. Could be consolidated into one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [tantivy] Support configurable full-text analyzers [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8074:
URL: https://github.com/apache/paimon/pull/8074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org