You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/07/05 07:44:40 UTC
[PR] Prepare full-text index 0.1 API and storage [paimon-full-text]
JingsongLi opened a new pull request, #7:
URL: https://github.com/apache/paimon-full-text/pull/7
## Summary
Prepare the standalone Paimon full-text index for a 0.1 release by stabilizing the storage envelope, simplifying the public query API to JSON strings, and completing seek-on-demand read support across Rust, C, Java, and Python.
## Changes
- Stabilize the v1 storage envelope with metadata validation, Tantivy version checks, a 16 MiB header limit, and a golden file for the binary envelope layout.
- Replace eager in-memory Tantivy directory loading with a seek-on-demand archive directory, reader reuse, read metrics, prewarm support, and a small-block read cache.
- Expand indexing/search support for multi-field documents, repeated named fields, fuzzy/phrase/boolean/multi-match/boost-demotion queries, tokenizer options, and 64-bit Roaring filters.
- Simplify public search APIs to accept query JSON strings directly and remove Java/Python `FullTextQuery` helper classes.
- Align C, JNI, Java, and Python I/O around single positional `pread` callbacks that must be safe for concurrent calls.
- Remove stale design/plan documents and update current README/API/storage docs.
## Testing
- [x] `cargo clippy --workspace --all-targets -- -D warnings`
- [x] `cargo test --workspace`
- [x] `cargo build -p paimon-ftindex-ffi -p paimon-ftindex-jni`
- [x] `PYTHONPATH=python PAIMON_FTINDEX_LIB_PATH=/Users/lijingsong/IdeaProjects/paimon-full-text/target/debug/libpaimon_ftindex_ffi.dylib python3 -m pytest -q python/tests`
- [x] `PAIMON_FTINDEX_JNI_LIB_PATH=/Users/lijingsong/IdeaProjects/paimon-full-text/target/debug/libpaimon_ftindex_jni.dylib mvn -q -f java/pom.xml test`
## Notes
Breaking API cleanup before 0.1: Java and Python query builder classes were removed; callers now pass the query DSL as a JSON string to `search`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prepare full-text index 0.1 API and storage [paimon-full-text]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #7:
URL: https://github.com/apache/paimon-full-text/pull/7#discussion_r3524514631
##########
core/src/tokenizer.rs:
##########
@@ -36,19 +37,19 @@ impl Default for TokenizerConfig {
fn default() -> Self {
Self {
tokenizer: TokenizerKind::Default,
- ngram_min_gram: 2,
- ngram_max_gram: 2,
+ ngram_min_gram: 3,
+ ngram_max_gram: 3,
ngram_prefix_only: false,
jieba_search_mode: true,
jieba_ordinal_position: true,
lower_case: true,
max_token_length: 40,
- ascii_folding: false,
- stem: false,
+ ascii_folding: true,
+ stem: true,
language: "english".to_string(),
- remove_stop_words: false,
+ remove_stop_words: true,
stop_words: Vec::new(),
- with_position: true,
+ with_position: false,
Review Comment:
This changes the library defaults from the previous API/Paimon adapter assumptions (`ngram` 2/2, `ascii-folding=false`, `stem=false`, `remove-stop-words=false`, `with-position=true`) to English-analysis defaults with `with_position=false`. That is not just a local default change: the Java Paimon adapter in apache/paimon PR #8463 serializes sparse native options by omitting values that are default on the Paimon side. With these new defaults, a default Paimon full-text index would be built without positions, so phrase queries fail with `phrase query requires positions`; `ngram` without explicit grams also becomes 3/3 instead of Paimon's 2/2, and stemming/stop-word/ascii-folding semantics silently change. Please either keep the standalone 0.1 defaults aligned with the Paimon adapter defaults, or update the adapter/integration contract so it always passes the full intended defaults (and add a regression test covering default phrase search/ngram defaults).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prepare full-text index 0.1 API and storage [paimon-full-text]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #7:
URL: https://github.com/apache/paimon-full-text/pull/7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org