You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/10/10 12:00:59 UTC
[PR] [Python] Delegate local full-text search to Rust core [paimon]
JingsongLi opened a new pull request, #10494:
URL: https://github.com/apache/paimon/pull/10494
### Purpose
Delegate existing local full-text searches to Rust core for eligible REST Data Evolution tables with row tracking and Parquet files. Query parsing, snapshot/index planning, scalar filtering, raw scans and ranking run in core; Python forwards builder configuration and wraps scored results.
Depends on https://github.com/apache/paimon-rust/pull/1095. This PR is draft until that dependency reaches Rust main. CI keeps installing Rust main; no fork/branch pin is introduced.
- Reuse the existing Python builder/result API and full-text analyzer option normalization.
- Qualify support before searching. Tables outside the route or predicates not representable by the typed FFI use the Python reader. Once native execution starts, errors propagate without a second search.
- Align the classic Python reader with current Java scalar FAST/FULL/DETAIL coverage and configurable candidate refinement, instead of forcing every scalar predicate through FULL reads.
- Pin classic read plans, attach scalar files to indexed splits, and require an existing full-text definition before raw search.
### Tests
- 106 passed with native plan/read/write/update/commit CI gates enabled against a wheel built from the paired Rust change.
- Actual native counters: 113 plans, 105 reads, 10 writes, 1 commit; row-ID, grouped, predicate, upsert, incremental and merge update categories all exercised.
- 98 new REST tests compare native/classic result IDs and scores, including all full-text/scalar modes, BTree/bitmap and partial coverage, Boolean/boost queries, pre-Top-K filters, candidate refinement, typed stop words, partition predicates, pinned/empty/error REST snapshots, schema-only renames, deletion vectors, text update invalidation, invalid limits and query authorization.
- Eligible-route tests prohibit Python full-text scanning. Missing-index and REST execution failures verify that native errors are not retried through Python.
- Existing search reader tests: 79 passed and 19 subtests passed.
- Flake8, Python 3.6 source syntax and diff whitespace checks: passed.
PK and distributed full-text searches retain their existing path. No new index type or storage format is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Delegate local full-text search to Rust core [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #10494:
URL: https://github.com/apache/paimon/pull/10494#discussion_r4237671641
##########
paimon-python/pypaimon/table/source/full_text_read.py:
##########
@@ -91,27 +92,71 @@ def read(self, splits: List[FullTextSearchSplit]) -> GlobalIndexResult:
splits_by_column.setdefault(split.column_name, []).append(split)
live_rows = global_index_live_row_filter.live_rows(
self._table, self._partition_filter)
- if self._filter is not None:
- from pypaimon.table.source.global_index_row_filter import matching_rows
-
- candidates = RoaringBitmap64()
- for split in index_splits:
- candidates.add_range(split.row_range_start, split.row_range_end)
- for row_range in _raw_row_ranges(raw_splits):
- candidates.add_range(row_range.from_, row_range.to)
- # Scalar indexes may prune this read, but incomplete scalar coverage
- # must not exclude rows covered by the full-text search plan.
- table = self._table.copy({"scalar-index.search-mode": "full"})
- matched = matching_rows(table, self._filter, candidates, self._partition_filter)
- live_rows = matched if live_rows is None else RoaringBitmap64.and_(live_rows, matched)
- indexed_result = self._eval_column_query(splits_by_column, live_rows)
+ matched = self._matched_indexed_rows(index_splits, live_rows)
+ indexed_result = self._eval_column_query(splits_by_column, matched if self._filter is not None else live_rows)
raw_ranges = _raw_row_ranges(raw_splits)
if self._filter is None:
raw_result = self._read_raw_search(raw_ranges, _index_type(index_splits))
else:
- raw_result = self._read_raw_search(raw_ranges, _index_type(index_splits), live_rows)
+ candidates = GlobalIndexResult.from_ranges(raw_ranges).results()
+ if live_rows is not None:
+ candidates = RoaringBitmap64.and_(candidates, live_rows)
+ matched = self._matching_rows(candidates)
+ raw_result = self._read_raw_search(raw_ranges, _index_type(index_splits), matched)
return indexed_result.or_(raw_result).top_k(self._limit)
+ def _matching_rows(self, candidates):
+ from pypaimon.table.source.global_index_row_filter import matching_rows
+
+ table = self._table.copy({'scalar-index.search-mode': 'full'})
+ return matching_rows(table, self._filter, candidates, self._partition_filter)
+
+ def _matched_indexed_rows(self, splits, live_rows):
+ if self._filter is None:
+ return live_rows
+ from pypaimon.common.options.core_options import GlobalIndexSearchMode
+ from pypaimon.globalindex.data_evolution_global_index_coverage import DataEvolutionGlobalIndexCoverage
+ from pypaimon.globalindex.data_evolution_global_index_scanner import DataEvolutionGlobalIndexScanner
+ from pypaimon.snapshot.time_travel_util import TimeTravelUtil
+
+ covered = RoaringBitmap64()
+ files = {}
+ for split in splits:
+ covered.add_range(split.row_range_start, split.row_range_end)
+ files.update((file.file_name, file) for file in split.scalar_index_files)
+ if live_rows is not None:
+ covered = RoaringBitmap64.and_(covered, live_rows)
+ if covered.is_empty():
+ return covered
+ snapshot = getattr(self, '_snapshot', None)
+ if snapshot is None:
+ snapshot = TimeTravelUtil.resolve_snapshot(self._table)
+ mode = self._table.options.scalar_index_search_mode()
+ scalar_files = list(files.values())
+ coverage = DataEvolutionGlobalIndexCoverage(self._table, snapshot, self._partition_filter, scalar_files)
+ raw_ranges = coverage.unindexed_ranges(self._table.fields, self._filter, mode)
+ unindexed = RoaringBitmap64.and_(GlobalIndexResult.from_ranges(raw_ranges).results(), covered)
+ decided = RoaringBitmap64.remove_all(covered, unindexed)
+ matched = RoaringBitmap64()
+ if not decided.is_empty():
+ scanner = DataEvolutionGlobalIndexScanner.create(self._table, index_files=scalar_files,
+ snapshot=snapshot, partition_filter=self._partition_filter)
+ evaluation = None
+ if scanner is not None:
+ try:
+ evaluation = scanner.scan_with_coverage(self._filter)
+ finally:
+ scanner.close()
+ if evaluation is not None:
+ candidates = RoaringBitmap64.and_(evaluation.result.results(), decided)
+ if evaluation.result.is_exact():
Review Comment:
[P2] Match Java/Rust candidate policy for bitmap text predicates
Using evaluation.result.is_exact() alone admits indexed bitmap matches when global-index.filter.refine-from-data=false. The bitmap reader marks contains, endsWith and LIKE results exact, but Java FilteredRowIdReader.isExact (paimon-core, lines 85-108) and Rust GlobalIndexScanner::is_exact_predicate deliberately classify these predicate leaves as candidate-only regardless of the selected index reader. Consequently the new native route excludes those indexed rows while the classic Python route retains them. This option defaults to false, so toggling read.native.enabled changes the result set for an otherwise supported REST table.
Reproduced at these fixed heads using the REST fixture from native_full_text_search_test.py: build the full-text content index and a bitmap index on label, use label contains 'eep' / endsWith 'eep' / LIKE '%eep', set refinement=false, and search 'paimon' with limit 64. With complete index coverage, classic returns row IDs {1,5,7}, native returns empty. After appending the unindexed tail, classic returns {1,5,7,9,11}, native returns {9,11}. I used the existing _compare helper, which also forbids a Python scan retry during native execution. All six bitmap/refinement-disabled comparisons fail; the 18 BTree/refinement-enabled controls pass.
Please apply the same recursive predicate-level exactness policy as current Java before accepting the scalar result here, rather than exempting bitmap predicates solely because the reader returns an exact bitmap. Also change the new full_text_scalar_filter_test.py expectation at lines 76-78, which currently encodes the bitmap exemption, and extend the REST/native parity tests to bitmap contains/endsWith/LIKE with refinement both disabled and enabled, including an unindexed tail. Rust already follows the Java policy and should not be relaxed just to match the classic path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Delegate local full-text search to Rust core [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #10494:
URL: https://github.com/apache/paimon/pull/10494#discussion_r4237931664
##########
paimon-python/pypaimon/table/source/full_text_read.py:
##########
@@ -91,27 +92,72 @@ def read(self, splits: List[FullTextSearchSplit]) -> GlobalIndexResult:
splits_by_column.setdefault(split.column_name, []).append(split)
live_rows = global_index_live_row_filter.live_rows(
self._table, self._partition_filter)
- if self._filter is not None:
- from pypaimon.table.source.global_index_row_filter import matching_rows
-
- candidates = RoaringBitmap64()
- for split in index_splits:
- candidates.add_range(split.row_range_start, split.row_range_end)
- for row_range in _raw_row_ranges(raw_splits):
- candidates.add_range(row_range.from_, row_range.to)
- # Scalar indexes may prune this read, but incomplete scalar coverage
- # must not exclude rows covered by the full-text search plan.
- table = self._table.copy({"scalar-index.search-mode": "full"})
- matched = matching_rows(table, self._filter, candidates, self._partition_filter)
- live_rows = matched if live_rows is None else RoaringBitmap64.and_(live_rows, matched)
- indexed_result = self._eval_column_query(splits_by_column, live_rows)
+ matched = self._matched_indexed_rows(index_splits, live_rows)
+ indexed_result = self._eval_column_query(splits_by_column, matched if self._filter is not None else live_rows)
raw_ranges = _raw_row_ranges(raw_splits)
if self._filter is None:
raw_result = self._read_raw_search(raw_ranges, _index_type(index_splits))
else:
- raw_result = self._read_raw_search(raw_ranges, _index_type(index_splits), live_rows)
+ candidates = GlobalIndexResult.from_ranges(raw_ranges).results()
+ if live_rows is not None:
+ candidates = RoaringBitmap64.and_(candidates, live_rows)
+ matched = self._matching_rows(candidates)
+ raw_result = self._read_raw_search(raw_ranges, _index_type(index_splits), matched)
return indexed_result.or_(raw_result).top_k(self._limit)
+ def _matching_rows(self, candidates):
+ from pypaimon.table.source.global_index_row_filter import matching_rows
+
+ table = self._table.copy({'scalar-index.search-mode': 'full'})
+ return matching_rows(table, self._filter, candidates, self._partition_filter)
+
+ def _matched_indexed_rows(self, splits, live_rows):
+ if self._filter is None:
+ return live_rows
+ from pypaimon.common.options.core_options import GlobalIndexSearchMode
+ from pypaimon.globalindex.data_evolution_global_index_coverage import DataEvolutionGlobalIndexCoverage
+ from pypaimon.globalindex.data_evolution_global_index_scanner import DataEvolutionGlobalIndexScanner
+ from pypaimon.snapshot.time_travel_util import TimeTravelUtil
+ from pypaimon.table.source.global_index_row_filter import is_exact
+
+ covered = RoaringBitmap64()
+ files = {}
+ for split in splits:
+ covered.add_range(split.row_range_start, split.row_range_end)
+ files.update((file.file_name, file) for file in split.scalar_index_files)
+ if live_rows is not None:
+ covered = RoaringBitmap64.and_(covered, live_rows)
+ if covered.is_empty():
+ return covered
+ snapshot = getattr(self, '_snapshot', None)
+ if snapshot is None:
+ snapshot = TimeTravelUtil.resolve_snapshot(self._table)
+ mode = self._table.options.scalar_index_search_mode()
+ scalar_files = list(files.values())
+ coverage = DataEvolutionGlobalIndexCoverage(self._table, snapshot, self._partition_filter, scalar_files)
+ raw_ranges = coverage.unindexed_ranges(self._table.fields, self._filter, mode)
+ unindexed = RoaringBitmap64.and_(GlobalIndexResult.from_ranges(raw_ranges).results(), covered)
+ decided = RoaringBitmap64.remove_all(covered, unindexed)
+ matched = RoaringBitmap64()
+ if not decided.is_empty():
+ scanner = DataEvolutionGlobalIndexScanner.create(self._table, index_files=scalar_files,
+ snapshot=snapshot, partition_filter=self._partition_filter)
+ evaluation = None
+ if scanner is not None:
+ try:
+ evaluation = scanner.scan_with_coverage(self._filter)
+ finally:
+ scanner.close()
+ if evaluation is not None:
+ candidates = RoaringBitmap64.and_(evaluation.result.results(), decided)
+ if is_exact(self._filter, evaluation.result):
Review Comment:
[P2] Provide exact BTree prefix results before dropping indexed candidates
This refinement gate now drops BTree `startsWith` candidates, including optimized `LIKE 'ke%'`, when `global-index.filter.refine-from-data=false`. The classic `BTreeIndexReader.visit_starts_with` still returns every non-null row with `is_exact=False` (btree_index_reader.py:250-251), so the new shared helper rejects it. In contrast, Java `BTreeIndexReader.visitStartsWith` performs an exact prefix interval query (lines 280-297), and Java `FilteredRowIdReader.isExact` does not classify StartsWith as candidate-only. Rust also performs the exact prefix query and is correct to retain those indexed matches. Consequently, ordinary prefix filters still change their result set when `read.native.enabled` is toggled. The option defaults to false. This is also a classic-route regression: before this PR the classic full-text reader refined these candidates instead of excluding them.
Reproduced on Python head `5d2c640095b7d88f8b6641cb424007aa784f2076` using a freshly rebuilt Rust wheel from `1ccf5baa1ad29e35f6220c3595cc24fdf4488b02`, with the existing REST/full-text fixture, FULL text/scalar modes and a label BTree index:
- With complete coverage, `label startsWith 'ke'` and `label LIKE 'ke%'` both return empty on classic but `{1,5,7}` on native.
- After appending the unindexed tail, classic returns `{9,11}` but native returns `{1,5,7,9,11}`.
- The independent 32-case prefix matrix (BTree/bitmap x StartsWith/optimized LIKE x matching/missing prefix x complete/partial coverage x refinement off/on) has four failures, all for matching BTree prefixes with refinement disabled. The bitmap, missing-prefix and refinement-enabled controls pass. The native comparison forbids a Python full-text scan retry.
- The four corresponding classic prefix cases pass on the PR base `1b49a70cfaf5d4430298c825b5ba9d499bddad01` with refinement disabled. Empty StartsWith adds two failures with the same BTree cause.
Please implement an exact prefix range query in the Python BTree reader, matching Java/Rust, and only then return an exact result. Simply marking the existing all-non-null bitmap exact would introduce false positives. Extend `test_optimized_bitmap_like_filters_stay_exact_without_refinement` to BTree too, covering both StartsWith and optimized prefix LIKE and an unindexed tail; keep Rust's Java-aligned behavior unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Delegate local full-text search to Rust core [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #10494:
URL: https://github.com/apache/paimon/pull/10494#discussion_r4237759603
##########
paimon-python/pypaimon/table/source/native_full_text_search.py:
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Local REST full-text search delegated to Rust core."""
+
+from pypaimon.catalog.table_query_auth import reject_search_under_query_auth
+from pypaimon.globalindex.full_text.native_full_text_global_index_reader import NativeFullTextIndexOptions
+from pypaimon.globalindex.vector_search_result import DictBasedScoredIndexResult
+from pypaimon.read.native_plan import _native_table, _predicate_to_native, native_method_available
+from pypaimon.table.file_store_table import FileStoreTable
+from pypaimon.write.native_commit import _rest_catalog_supported
+
+
+def try_native_full_text_search(builder):
+ """Qualify the route before execution; Rust owns scan, filtering and ranking."""
+ table = builder._table
+ reject_search_under_query_auth(table)
+ if (type(table) is not FileStoreTable
+ or not table.options.native_read_enabled()
+ or not table.options.data_evolution_enabled()
+ or not table.options.row_tracking_enabled()
+ or table.is_primary_key_table
+ or table.options.file_format() != 'parquet'
+ or not native_method_available('Table', 'new_full_text_search_builder')
+ or not _rest_catalog_supported(table)):
+ return None
+ if builder._limit is None or builder._limit <= 0:
+ raise ValueError('Limit must be positive, set via with_limit()')
+ # The full-text adapter accepts typed lists and booleans. Reuse its mapping
+ # before the generic table bridge stringifies options for Rust.
+ options = {'full-text.' + key: value for key, value in NativeFullTextIndexOptions.from_options(
+ table.options.options.to_map()).to_native_options().items()}
+ native = _native_table(table.copy(options)).new_full_text_search_builder()
+ if builder._field_name is not None and builder._query is not None:
+ native.with_query(builder._field_name, builder._query)
+ native.with_limit(builder._limit)
+ try:
+ if builder._partition_filter is not None:
+ native.with_partition_filter(_predicate_to_native(builder._partition_filter))
+ if builder._filter is not None:
+ native.with_filter(_predicate_to_native(builder._filter))
Review Comment:
[P2] Preserve Java search exactness when native predicates are folded
The new native dispatch passes the original filter through `_predicate_to_native`, but that bridge uses Rust's predicate builder and can change the search exactness contract before full-text ranking. Java `LikeOptimization` requires a nonempty literal/prefix/suffix/substring, so `LIKE ''`, `LIKE '%'` and `LIKE '%%'` remain `Like` leaves; `FilteredRowIdReader.isExact` excludes their indexed candidates when `global-index.filter.refine-from-data=false`. The new classic helper correctly follows that policy. Rust instead rewrites `LIKE ''` to `Eq('')` and the all-wildcard patterns to `IsNotNull`; empty `contains`/`endsWith` literals are also folded to `IsNotNull`. `GlobalIndexScanner::is_exact_predicate` then accepts those normalized leaves as exact. This option defaults to false, so enabling native reads now changes the result set for a supported REST table. The normalization predates this PR, but the new full-text route exposes it.
Reproduced on Python head `a604c96fe4b5603133f9eb358eaa06a2ab8837ce` with Rust head `3badea107c3a8f434c82aec239a0b89cc3ad8049`: use the existing REST fixture, DE + row tracking + Parquet, FULL text/scalar modes, `content='paimon'` for all rows, and labels `['', None, 'keep', 'drop', '', 'keep', 'drop', None]`. Build the content full-text index and a label bitmap or BTree index, and search with limit 64:
- Refinement disabled, complete scalar coverage: `LIKE '%'` / `LIKE '%%'` / `contains('')` / `endsWith('')` return no rows on classic, but `{0,2,3,4,5,6}` on native. `LIKE ''` returns no rows on classic, but `{0,4}` on native.
- After appending labels `['', None, 'keep', 'drop']` as rows 8-11 without rebuilding the indexes: classic returns only `{8,10,11}` for the all-non-null predicates and `{8}` for `LIKE ''`; native additionally retains the indexed rows above.
- The focused matrix (2 index families x 5 predicates x complete/partial coverage x refinement off/on) gives 20 failures, all with refinement disabled, and 20 passing refinement-enabled controls. Native execution is wrapped with the existing assertion against a Python full-text scan retry.
Please preserve the original Java-aligned candidate-only classification across native conversion, or align the corresponding Rust/FFI predicate normalization for search. Do not relax the classic helper just to match the folded native predicates. Add these empty/all-wildcard cases with both index families and partial coverage; `LIKE '%'` is a normal supported filter, not an invalid input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Delegate local full-text search to Rust core [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #10494:
URL: https://github.com/apache/paimon/pull/10494#discussion_r4237619465
##########
paimon-python/pypaimon/table/source/full_text_read.py:
##########
@@ -91,27 +92,71 @@ def read(self, splits: List[FullTextSearchSplit]) -> GlobalIndexResult:
splits_by_column.setdefault(split.column_name, []).append(split)
live_rows = global_index_live_row_filter.live_rows(
self._table, self._partition_filter)
- if self._filter is not None:
- from pypaimon.table.source.global_index_row_filter import matching_rows
-
- candidates = RoaringBitmap64()
- for split in index_splits:
- candidates.add_range(split.row_range_start, split.row_range_end)
- for row_range in _raw_row_ranges(raw_splits):
- candidates.add_range(row_range.from_, row_range.to)
- # Scalar indexes may prune this read, but incomplete scalar coverage
- # must not exclude rows covered by the full-text search plan.
- table = self._table.copy({"scalar-index.search-mode": "full"})
- matched = matching_rows(table, self._filter, candidates, self._partition_filter)
- live_rows = matched if live_rows is None else RoaringBitmap64.and_(live_rows, matched)
- indexed_result = self._eval_column_query(splits_by_column, live_rows)
+ matched = self._matched_indexed_rows(index_splits, live_rows)
+ indexed_result = self._eval_column_query(splits_by_column, matched if self._filter is not None else live_rows)
raw_ranges = _raw_row_ranges(raw_splits)
if self._filter is None:
raw_result = self._read_raw_search(raw_ranges, _index_type(index_splits))
else:
- raw_result = self._read_raw_search(raw_ranges, _index_type(index_splits), live_rows)
+ candidates = GlobalIndexResult.from_ranges(raw_ranges).results()
+ if live_rows is not None:
+ candidates = RoaringBitmap64.and_(candidates, live_rows)
+ matched = self._matching_rows(candidates)
+ raw_result = self._read_raw_search(raw_ranges, _index_type(index_splits), matched)
return indexed_result.or_(raw_result).top_k(self._limit)
+ def _matching_rows(self, candidates):
+ from pypaimon.table.source.global_index_row_filter import matching_rows
+
+ table = self._table.copy({'scalar-index.search-mode': 'full'})
+ return matching_rows(table, self._filter, candidates, self._partition_filter)
+
+ def _matched_indexed_rows(self, splits, live_rows):
+ if self._filter is None:
+ return live_rows
+ from pypaimon.common.options.core_options import GlobalIndexSearchMode
+ from pypaimon.globalindex.data_evolution_global_index_coverage import DataEvolutionGlobalIndexCoverage
+ from pypaimon.globalindex.data_evolution_global_index_scanner import DataEvolutionGlobalIndexScanner
+ from pypaimon.snapshot.time_travel_util import TimeTravelUtil
+
+ covered = RoaringBitmap64()
+ files = {}
+ for split in splits:
+ covered.add_range(split.row_range_start, split.row_range_end)
+ files.update((file.file_name, file) for file in split.scalar_index_files)
+ if live_rows is not None:
+ covered = RoaringBitmap64.and_(covered, live_rows)
+ if covered.is_empty():
+ return covered
+ snapshot = getattr(self, '_snapshot', None)
+ if snapshot is None:
+ snapshot = TimeTravelUtil.resolve_snapshot(self._table)
+ mode = self._table.options.scalar_index_search_mode()
+ scalar_files = list(files.values())
+ coverage = DataEvolutionGlobalIndexCoverage(self._table, snapshot, self._partition_filter, scalar_files)
+ raw_ranges = coverage.unindexed_ranges(self._table.fields, self._filter, mode)
+ unindexed = RoaringBitmap64.and_(GlobalIndexResult.from_ranges(raw_ranges).results(), covered)
+ decided = RoaringBitmap64.remove_all(covered, unindexed)
+ matched = RoaringBitmap64()
+ if not decided.is_empty():
+ scanner = DataEvolutionGlobalIndexScanner.create(self._table, index_files=scalar_files,
+ snapshot=snapshot, partition_filter=self._partition_filter)
+ evaluation = None
+ if scanner is not None:
+ try:
+ evaluation = scanner.scan_with_coverage(self._filter)
+ finally:
+ scanner.close()
+ if evaluation is not None:
+ candidates = RoaringBitmap64.and_(evaluation.result.results(), decided)
+ if evaluation.result.is_exact():
+ matched = candidates
+ elif self._table.options.global_index_filter_refine_from_data():
Review Comment:
[P2] Migrate the existing full-text tests to the new Java semantics
Honoring candidate refinement and scalar coverage here, together with the no-definition early return in full_text_scan.py, changes the classic/Python route too. The existing full_text_scalar_filter_test.py still asserts the previous behavior, so this PR leaves the normal Python suite red even though the new native tests pass.
With Python 3.10, PyArrow 19, and all native test gates disabled, running full_text_scalar_filter_test.py, full_text_reader_cleanup_test.py, and vector_filter_exactness_test.py gives 71 passed on base 1b49a70cfaf5d4430298c825b5ba9d499bddad01, but 67 passed / 4 failed on this head. The same four failures are confirmed in the PR's Python 3.11 CI job 114208339144 (7,021 passed / 4 failed):
- test_full_text_data_filters_precede_top_k_and_preserve_scores[False-btree]: candidate-only LIKE now returns empty with refinement disabled instead of row 2.
- test_partial_scalar_coverage_does_not_drop_full_text_matches: scalar FAST excludes the uncovered row 4; the test still expects rows 2 and 4.
- test_partition_and_data_predicates_keep_boolean_semantics[pt = 'b' AND label = 'target'-expected0]: the partition has no selected full-text definition, so the result is empty instead of row 4.
- test_partition_without_full_text_index_respects_search_mode[full-expected1]: the same definition requirement conflicts with its expected raw-row result.
Please update these existing fixtures/configurations and assertions to the intended Java contract in this PR: explicitly enable refinement when testing refined LIKE matches, distinguish FAST from FULL scalar coverage, and seed a full-text definition in the selected partition when testing raw fallback. Keep the implementation Java-aligned rather than reverting the new semantics just to satisfy the old assertions, and rerun the existing suite as well as the new REST/native cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Delegate local full-text search to Rust core [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10494:
URL: https://github.com/apache/paimon/pull/10494
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org