You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/06/21 03:00:58 UTC
[PR] [core] Add structured full-text query DSL [paimon]
JingsongLi opened a new pull request, #8308:
URL: https://github.com/apache/paimon/pull/8308
## Summary
Add a single-column structured full-text query DSL for Paimon full-text search. The DSL supports match, phrase, fuzzy, boost, and boolean queries while preserving the existing query text/operator APIs and Spark SQL function shape.
## Changes
- Add `FullTextQuery` as a canonical Java representation for structured full-text queries, including JSON parsing/serialization and validation.
- Teach `FullTextSearch`, Java builders, Spark scan pushdown, and the test full-text reader to preserve structured query semantics end to end.
- Add Tantivy JNI JSON search entry points and build Tantivy query objects for match, phrase, fuzzy, boost, and boolean query forms.
- Reject `multi_match` explicitly for now because the current full-text index is single-column.
- Document structured DSL usage for Spark SQL and Java API.
## Testing
- `mvn -pl paimon-common,paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=FullTextQueryTest,FullTextSearchBuilderTest test`
- `cargo check` in `paimon-tantivy/paimon-tantivy-jni/rust`
- `mvn -pl paimon-tantivy/paimon-tantivy-index -am -Pfast-build -DfailIfNoTests=false -Dtest=TantivyFullTextGlobalIndexTest test`
- `mvn -pl paimon-spark/paimon-spark-common,paimon-spark/paimon-spark3-common,paimon-spark/paimon-spark-ut -am -Pspark3,fast-build -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.FullTextSearchTest -Dtest=none test`
- `mvn -pl paimon-common,paimon-core,paimon-tantivy/paimon-tantivy-index,paimon-spark/paimon-spark-common,paimon-spark/paimon-spark3-common,paimon-spark/paimon-spark-ut -am -Pspark3 -DskipTests -DfailIfNoTests=false compile`
- `git diff --check`
## Notes
`multi_match` is reserved for future multi-column full-text indexes. Tantivy fuzzy search currently supports fuzziness up to 2; `max_expansions` and `prefix_length` are rejected unless they keep the default values because Tantivy 0.22 does not expose those controls through the public API used here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Align full-text search with JSON DSL [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8308:
URL: https://github.com/apache/paimon/pull/8308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Align full-text search with JSON DSL [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8308:
URL: https://github.com/apache/paimon/pull/8308#issuecomment-4764198031
Addressed the blocker in `2e325b67e`: full-text scans now only include global index types whose factory declares full-text support, and the Tantivy/test full-text factories opt in. I also added a regression test covering a full-text index and a btree index on the same `content` column; it failed before the fix with `Expected exactly one index file per shard` and now passes.\n\nValidation run locally:\n- `mvn -pl paimon-common,paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=FullTextQueryTest,FullTextSearchBuilderTest test`\n- `mvn -pl paimon-tantivy/paimon-tantivy-index -am -Pfast-build -DskipTests -DfailIfNoTests=false compile`\n- `mvn -pl paimon-common,paimon-core,paimon-tantivy/paimon-tantivy-index -DskipTests -DfailIfNoTests=false spotless:check`\n- `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Align full-text search with JSON DSL [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8308:
URL: https://github.com/apache/paimon/pull/8308#issuecomment-4764245416
Addressed the PyPaimon blocker in `3ceef0054`: `FullTextScanImpl` now filters scanned index files to `tantivy-fulltext`, matching the only full-text reader supported by PyPaimon. I added a regression test with `tantivy-fulltext` and `btree` entries on the same `content` column/range; it failed before the fix with both files in the split and now passes.\n\nValidation run locally:\n- `python -m pytest paimon-python/pypaimon/tests/vector_search_filter_test.py -q -k full_text`\n- `python -m pytest paimon-python/pypaimon/tests/vector_search_filter_test.py -q`\n- `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org