You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by ai...@apache.org on 2026/10/22 03:22:20 UTC
(doris) branch master updated: [fix](be) Make SEARCH predicates consistently two-valued (#69347)
This is an automated email from the ASF dual-hosted git repository.
airborne12 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 67f31fa4b80 [fix](be) Make SEARCH predicates consistently two-valued (#69347)
67f31fa4b80 is described below
commit 67f31fa4b808d22ee1b70dc5beda571e4939561b
Author: lihangyu <li...@selectdb.com>
AuthorDate: Thu Oct 22 11:22:11 2026 +0800
[fix](be) Make SEARCH predicates consistently two-valued (#69347)
### What problem does this PR solve?
Related PR: #68758
SEARCH declares a non-nullable result, but UNKNOWN leaves and root-only
NULL masking can omit rows under NOT or change membership across scoring
and cached execution. Evaluate the complete backend query as a
two-valued match set: ordinary terms do not match NULL or proven-absent
fields; AND/OR combine match sets; NOT complements them. Ordinary SQL
expressions and MATCH retain their existing three-valued semantics.
Filter physical NULL postings at SEARCH leaf boundaries with the
existing `NullMaskedQuery`, including NULL ARRAY values with retained
nested payloads. Keep cached result bitmaps independently owned and
preserve source-managed document IDs across collectors.
This change uses generic execution for masked terms. It does not add
`excluded_rows` to term queries, weights, scorers, or grouped reads.
Grouped term opening and term-specific block-max WAND are deferred for
these masked leaves; no performance improvement or benchmark result is
claimed.
A missing iterator does not prove that a Variant path is absent. When a
historical rowset contains the path but has no executable index after
CREATE INDEX, fail explicitly instead of returning incomplete results.
Tests write actual objects with `parse_to_variant`, check their type and
extracted values, then distinguish unavailable historical indexes, newly
indexed rows, and genuinely absent paths. Index backfill and scan
fallback are outside this change. The historical-rowset fixture uses a
non-tokenized index with `enable_add_index_for_new_data=true`, so Cloud
and local mode both leave the historical rowsets unindexed.
**Independent validation:** this PR contains no FE or protocol changes
and was built and tested with FE source identical to master. Backend
field-existence semantics are tested directly through the existing
`TSearchClause` interface, including empty arrays, physical NULL
postings, numeric fields, absent paths, cache, scoring, and multiple
readers. SQL suites use clauses already emitted by master FE. There is
no companion-PR prerequisite or required merge order.
---
be/src/exprs/function/search/function_search.cpp | 132 ++---
.../exprs/function/search/function_search_leaf.cpp | 23 +-
.../exprs/function/search/function_search_leaf.h | 12 +-
be/src/exprs/function/search/search_lowering.cpp | 26 +-
.../function/search/search_predicate_collector.cpp | 5 +
.../search/variant_inverted_index_search.cpp | 59 +-
.../search/variant_inverted_index_search.h | 26 +-
be/src/exprs/vsearch.cpp | 100 +++-
.../inverted/query/engine/null_masked_query.h | 44 +-
.../inverted/reader/collection_statistics.cpp | 42 +-
.../function/search/function_search_null_test.cpp | 327 +++++++++--
.../exprs/function/search/function_search_test.cpp | 638 ++++++++++++++++++---
.../exprs/function/search/search_lowering_test.cpp | 8 +-
be/test/exprs/vsearch_expr_test.cpp | 96 +++-
.../inverted/reader/collection_statistics_test.cpp | 82 ++-
.../storage/segment/inverted_index_array_test.cpp | 180 ++++++
.../variant/variant_column_writer_reader_test.cpp | 76 +++
.../test_cross_field_or_with_null.out | 21 +
.../data/search/test_search_boundary_cases.out | 8 +-
.../data/search/test_search_dsl_syntax.out | 8 +
.../data/search/test_search_not_null_bitmap.out | 17 +-
.../data/search/test_search_null_regression.out | 14 +-
.../data/search/test_search_null_semantics.out | 23 +-
.../search/test_search_segment_document_ids.out | 6 +
.../search/test_search_vs_match_consistency.out | 32 +-
...test_variant_search_historical_rowset_index.out | 157 +++++
.../test_variant_search_missing_doc_path.out | 305 ++++++++++
.../test_complex_or_null_semantics.groovy | 20 +-
.../test_cross_field_or_with_null.groovy | 35 +-
.../test_search_or_null_semantics.groovy | 20 +-
regression-test/suites/search/README.md | 37 +-
.../test_search_document_boolean_contract.groovy | 197 +++++++
.../test_search_null_array_payload_contract.groovy | 88 +++
.../test_search_variant_two_valued_contract.groovy | 153 +++++
.../search/test_search_boundary_cases.groovy | 3 +-
.../suites/search/test_search_dsl_syntax.groovy | 1 +
.../search/test_search_not_null_bitmap.groovy | 28 +-
.../search/test_search_null_regression.groovy | 5 +-
.../search/test_search_null_semantics.groovy | 11 +-
.../search/test_search_vs_match_consistency.groovy | 7 +-
...t_variant_search_historical_rowset_index.groovy | 141 +++++
.../test_variant_search_missing_doc_path.groovy | 106 ++++
42 files changed, 2849 insertions(+), 470 deletions(-)
diff --git a/be/src/exprs/function/search/function_search.cpp b/be/src/exprs/function/search/function_search.cpp
index 5365e004fcf..0dc39328bf5 100644
--- a/be/src/exprs/function/search/function_search.cpp
+++ b/be/src/exprs/function/search/function_search.cpp
@@ -114,41 +114,33 @@ static std::string extract_segment_prefix(
namespace {
-// A SEARCH result is cached as two entries: its TRUE rows, and its UNKNOWN rows under the
-// signature with a "__null" suffix.
+// Keep two-valued SEARCH results separate from older SQL UNKNOWN cache entries.
InvertedIndexQueryCache::CacheKey dsl_cache_key(const std::string& seg_prefix,
const std::string& signature) {
- return {seg_prefix, "__search_dsl__", InvertedIndexQueryType::SEARCH_DSL_QUERY, signature};
+ return {seg_prefix, "__search_dsl_2vl__", InvertedIndexQueryType::SEARCH_DSL_QUERY, signature};
}
-// Either entry can be evicted, or not yet inserted, without the other, and TRUE rows found
-// without their UNKNOWN rows are no result.
bool lookup_dsl_result(InvertedIndexQueryCache* cache, const std::string& seg_prefix,
const std::string& signature, InvertedIndexResultBitmap* result) {
InvertedIndexQueryCacheHandle rows_handle;
- InvertedIndexQueryCacheHandle nulls_handle;
- if (!cache->lookup(dsl_cache_key(seg_prefix, signature), &rows_handle) ||
- !cache->lookup(dsl_cache_key(seg_prefix, signature + "__null"), &nulls_handle)) {
+ if (!cache->lookup(dsl_cache_key(seg_prefix, signature), &rows_handle)) {
return false;
}
auto rows = rows_handle.get_bitmap();
- auto nulls = nulls_handle.get_bitmap();
- if (rows == nullptr || nulls == nullptr) {
+ if (rows == nullptr) {
return false;
}
- *result = InvertedIndexResultBitmap(std::move(rows), std::move(nulls));
+ // SQL bitmap composition can mutate its input. The cache owns an independent copy.
+ *result = InvertedIndexResultBitmap(std::make_shared<roaring::Roaring>(*rows),
+ std::make_shared<roaring::Roaring>());
return true;
}
-// The UNKNOWN rows go in first, so a search that finds the TRUE rows finds them as well.
void insert_dsl_result(InvertedIndexQueryCache* cache, const std::string& seg_prefix,
const std::string& signature, const InvertedIndexResultBitmap& result) {
- auto nulls = result.get_null_bitmap() != nullptr ? result.get_null_bitmap()
- : std::make_shared<roaring::Roaring>();
- InvertedIndexQueryCacheHandle nulls_handle;
- cache->insert(dsl_cache_key(seg_prefix, signature + "__null"), std::move(nulls), &nulls_handle);
InvertedIndexQueryCacheHandle rows_handle;
- cache->insert(dsl_cache_key(seg_prefix, signature), result.get_data_bitmap(), &rows_handle);
+ cache->insert(dsl_cache_key(seg_prefix, signature),
+ std::make_shared<roaring::Roaring>(*result.get_data_bitmap()), &rows_handle);
}
bool is_nested_group_search_supported() {
@@ -172,7 +164,7 @@ public:
if (!binding.is_bound()) {
LOG(INFO) << "search: No inverted index for field '" << field
<< "' in this segment, query_type=" << static_cast<int>(query_type)
- << ", returning UNKNOWN bitmap";
+ << ", returning non-matching bitmap";
*out = logical::FieldProps {};
return Status::OK();
}
@@ -244,28 +236,23 @@ Status compile_node(const logical::Node& node, const SearchLeafContext& ctx,
FieldReaderResolver& resolver, query_v2::QueryPtr* out,
std::string* binding_key);
-// Marks the rows where the field is NULL as UNKNOWN for leaves that would misreport them: an array
-// index can list a NULL array's leftover values, and a term threshold counts its terms two-valued.
-Status mask_null_rows(const FieldReaderBinding& binding, const FieldReaderResolver& resolver,
- const std::string& field, bool threshold, query_v2::QueryPtr* query) {
- const bool array = binding.column_type != nullptr &&
- remove_nullable(binding.column_type)->get_primitive_type() == TYPE_ARRAY;
- if (!array && !threshold) {
- return Status::OK();
- }
+// Physical NULL postings can overlap a NULL ARRAY's leftover values. Exclude them
+// before Boolean composition without introducing SQL UNKNOWN or changing scores.
+Status mask_null_rows(const FieldReaderResolver& resolver, const std::string& field,
+ query_v2::QueryPtr* query) {
IndexIterator* iterator = resolver.get_iterator(field);
- if (iterator == nullptr) {
- return Status::OK();
- }
- auto has_null = iterator->has_null();
- if (!has_null.has_value() || !has_null.value()) {
+ DORIS_CHECK(iterator != nullptr);
+ const bool has_null = DORIS_TRY(iterator->has_null());
+ if (!has_null) {
return Status::OK();
}
InvertedIndexQueryCacheHandle handle;
RETURN_IF_ERROR(iterator->read_null_bitmap(&handle));
auto nulls = handle.get_bitmap();
- if (nulls != nullptr && !nulls->isEmpty()) {
- *query = std::make_shared<query_v2::NullMaskedQuery>(std::move(*query), std::move(nulls));
+ DORIS_CHECK(nulls != nullptr);
+ if (!nulls->isEmpty()) {
+ *query = std::make_shared<query_v2::NullMaskedQuery>(std::move(*query), std::move(nulls),
+ /*nulls_are_false=*/true);
}
return Status::OK();
}
@@ -277,8 +264,7 @@ bool has_required_clause(const Clauses& clauses) {
clauses, [](const auto& clause) { return clause.first == logical::Occur::kMust; });
}
-// Without scoring, optional clauses beside a required one change neither the rows nor their
-// UNKNOWN state, so they are left out.
+// Without scoring, optional clauses beside a required one do not change the match set.
Clauses matching_clauses(const logical::Bool& boolean, bool scoring) {
if (scoring || boolean.op != logical::BoolOp::kOccur || boolean.min_should_match != 0 ||
!has_required_clause(boolean.clauses)) {
@@ -367,8 +353,8 @@ Status compile_node(const logical::Node& node, const SearchLeafContext& ctx,
if (binding_key != nullptr) {
*binding_key = field->binding;
}
- if (node.as<logical::Unknown>() != nullptr) {
- *out = make_unknown_leaf_query(ctx.num_rows);
+ if (node.as<logical::Empty>() != nullptr) {
+ *out = std::make_shared<query_v2::BitSetQuery>(std::make_shared<roaring::Roaring>());
return resolver.map_leaf_query(field->name, out);
}
const FieldReaderBinding* binding = resolver.find_binding(field->binding);
@@ -376,14 +362,21 @@ Status compile_node(const logical::Node& node, const SearchLeafContext& ctx,
return Status::InternalError("search: field '{}' has no compiler for binding '{}'",
field->name, field->binding);
}
+ SearchLeafContext leaf_ctx = ctx;
+ RETURN_IF_ERROR(resolver.leaf_num_rows(field->name, &leaf_ctx.num_rows));
const auto* set = node.as<logical::TermSet>();
const bool threshold = set != nullptr && set->min_should_match > 0;
- if (threshold) {
- RETURN_IF_ERROR(compile_term_threshold(*set, ctx, *binding->leaf_compiler, out));
+ if (node.as<logical::Exists>() != nullptr) {
+ // Exists also works for BKD and uses the NestedGroup leaf's element domain.
+ auto matches = std::make_shared<roaring::Roaring>();
+ matches->addRange(0, leaf_ctx.num_rows);
+ *out = std::make_shared<query_v2::BitSetQuery>(std::move(matches));
+ } else if (threshold) {
+ RETURN_IF_ERROR(compile_term_threshold(*set, leaf_ctx, *binding->leaf_compiler, out));
} else {
- RETURN_IF_ERROR(binding->leaf_compiler->compile(node, ctx, out));
+ RETURN_IF_ERROR(binding->leaf_compiler->compile(node, leaf_ctx, out));
}
- RETURN_IF_ERROR(mask_null_rows(*binding, resolver, field->name, threshold, out));
+ RETURN_IF_ERROR(mask_null_rows(resolver, field->name, out));
return resolver.map_leaf_query(field->name, out);
}
@@ -453,14 +446,6 @@ Status FunctionSearch::_evaluate_search_param(
"NESTED query requires NestedGroup support, which is unavailable in this build");
}
- if (!is_nested_query && (iterators.empty() || data_type_with_names.empty())) {
- LOG(INFO) << "No indexed columns or iterators available, returning empty result, dsl:"
- << search_param.original_dsl;
- bitmap_result = InvertedIndexResultBitmap(std::make_shared<roaring::Roaring>(),
- std::make_shared<roaring::Roaring>());
- return Status::OK();
- }
-
// Track overall query time (equivalent to inverted_index_query_timer in MATCH path).
// Must be declared before the DSL cache lookup so that cache-hit fast paths are
// also covered by the timer.
@@ -468,7 +453,7 @@ Status FunctionSearch::_evaluate_search_param(
OlapReaderStatistics* outer_stats = index_query_context ? index_query_context->stats : nullptr;
SCOPED_RAW_TIMER(outer_stats ? &outer_stats->inverted_index_query_timer : &query_timer_dummy);
- // DSL result cache only stores bitmap/null bitmap. It does not store BM25 scores,
+ // DSL result cache only stores the two-valued match bitmap. It does not store BM25 scores,
// so score() queries must execute scorers to populate CollectionSimilarity.
const bool enable_scoring =
index_query_context != nullptr && index_query_context->collection_similarity != nullptr;
@@ -560,7 +545,7 @@ Status FunctionSearch::_evaluate_search_param(
return Status::OK();
}
- VariantSearchNullBitmapAdapter null_resolver(resolver);
+ SearchNullBitmapResolver null_resolver;
query_v2::QueryExecutionContext exec_ctx =
build_variant_search_query_execution_context(num_rows, resolver, &null_resolver);
@@ -580,13 +565,17 @@ Status FunctionSearch::_evaluate_search_param(
}
std::shared_ptr<roaring::Roaring> roaring = std::make_shared<roaring::Roaring>();
- // A three-valued root reports the rows it leaves UNKNOWN (TRUE OR NULL is TRUE, FALSE OR
- // NULL is NULL); a lucene-style Boolean root is two-valued: every row it does not match is
- // FALSE.
- std::shared_ptr<roaring::Roaring> null_bitmap = std::make_shared<roaring::Roaring>();
- const bool three_valued =
- exec_ctx.null_resolver != nullptr && search_param.root.clause_type != "OCCUR_BOOLEAN";
- const bool top_k_path = enable_scoring && !is_asc && top_k > 0;
+ // Keyword readers can lack BM25 length statistics. Preserve their original
+ // scores and all matches for the SQL sort when pruning cannot order them.
+ const bool can_prune_scores = std::none_of(
+ resolver.binding_cache().begin(), resolver.binding_cache().end(),
+ [](const auto& entry) {
+ const auto& binding = entry.second;
+ DORIS_CHECK(binding.inverted_reader != nullptr);
+ return binding.inverted_reader->type() == InvertedIndexReaderType::STRING_TYPE;
+ });
+ const bool top_k_path = enable_scoring && !is_asc && top_k > 0 && top_k < num_rows &&
+ top_k <= query_v2::TopKCollector::kMaxK && can_prune_scores;
exec_ctx.top_k = top_k_path;
{
int64_t exec_dummy = 0;
@@ -595,33 +584,24 @@ Status FunctionSearch::_evaluate_search_param(
bool use_wand = index_query_context->runtime_state != nullptr &&
index_query_context->runtime_state->query_options()
.enable_inverted_index_wand_query;
- query_v2::collect_multi_segment_top_k(
- weight, exec_ctx, root_binding_key, top_k, roaring,
- index_query_context->collection_similarity, use_wand,
- index_query_context->delete_bitmap, three_valued ? null_bitmap.get() : nullptr);
+ query_v2::collect_multi_segment_top_k(weight, exec_ctx, root_binding_key, top_k,
+ roaring,
+ index_query_context->collection_similarity,
+ use_wand, index_query_context->delete_bitmap);
} else {
- // The scorers listing the rows report the UNKNOWN ones as they go.
query_v2::collect_multi_segment_doc_set(
weight, exec_ctx, root_binding_key, roaring,
index_query_context ? index_query_context->collection_similarity : nullptr,
- enable_scoring, three_valued ? null_bitmap.get() : nullptr);
+ enable_scoring);
}
}
VLOG_DEBUG << "search: Query completed, matched " << roaring->cardinality() << " documents";
- VLOG_TRACE << "search: Before mask - true_bitmap=" << roaring->cardinality()
- << ", null_bitmap=" << null_bitmap->cardinality();
-
- // Create result and mask out NULLs (SQL WHERE clause semantics: only TRUE rows)
- bitmap_result = InvertedIndexResultBitmap(std::move(roaring), std::move(null_bitmap));
- bitmap_result.mask_out_null();
-
- VLOG_TRACE << "search: After mask - result_bitmap="
- << bitmap_result.get_data_bitmap()->cardinality();
+ // SEARCH never returns UNKNOWN; outer SQL composes this non-nullable match set.
+ bitmap_result =
+ InvertedIndexResultBitmap(std::move(roaring), std::make_shared<roaring::Roaring>());
- // The cached result keeps its UNKNOWN rows, so NOT, OR and AND over a cache hit stay
- // three-valued.
if (dsl_cache && cache_usable) {
insert_dsl_result(dsl_cache, seg_prefix, dsl_sig, bitmap_result);
}
diff --git a/be/src/exprs/function/search/function_search_leaf.cpp b/be/src/exprs/function/search/function_search_leaf.cpp
index 4a02d29665e..f9e3de40532 100644
--- a/be/src/exprs/function/search/function_search_leaf.cpp
+++ b/be/src/exprs/function/search/function_search_leaf.cpp
@@ -160,17 +160,14 @@ ScalarLeafCompiler::ScalarLeafCompiler(segment_v2::IndexIterator* iterator, Data
Status ScalarLeafCompiler::compile(const logical::Node& leaf, const SearchLeafContext& ctx,
query_v2::QueryPtr* out) {
const auto* compare = leaf.as<logical::Compare>();
- if (compare == nullptr || _iterator == nullptr) {
- *out = make_unknown_leaf_query(ctx.num_rows);
- return Status::OK();
- }
+ DORIS_CHECK(compare != nullptr);
+ DORIS_CHECK(_iterator != nullptr);
DataTypePtr value_type = unwrap_value_type(_column_type);
Field value;
if (Status parsed = parse_scalar_value(value_type, compare->value, &value); !parsed.ok()) {
LOG(INFO) << "search: scalar leaf value is unsupported, field=" << compare->field.name
<< ", value='" << compare->value << "', reason=" << parsed.to_string();
- *out = make_unknown_leaf_query(ctx.num_rows);
- return Status::OK();
+ return parsed;
}
segment_v2::InvertedIndexParam param;
@@ -182,17 +179,9 @@ Status ScalarLeafCompiler::compile(const logical::Node& leaf, const SearchLeafCo
param.roaring = std::make_shared<roaring::Roaring>();
RETURN_IF_ERROR(_iterator->read_from_index(segment_v2::IndexParam {¶m}));
- auto null_bitmap = std::make_shared<roaring::Roaring>();
- auto has_null = _iterator->has_null();
- if (has_null.has_value() && has_null.value()) {
- segment_v2::InvertedIndexQueryCacheHandle null_bitmap_cache_handle;
- RETURN_IF_ERROR(_iterator->read_null_bitmap(&null_bitmap_cache_handle));
- if (auto bitmap = null_bitmap_cache_handle.get_bitmap(); bitmap != nullptr) {
- null_bitmap = bitmap;
- }
- }
- *out = std::make_shared<query_v2::BitSetQuery>(std::move(param.roaring),
- std::move(null_bitmap));
+ // Physical NULL postings are excluded by the shared SEARCH leaf mask. Never
+ // attach SQL UNKNOWN rows or mutate the reader-owned query cache bitmap here.
+ *out = std::make_shared<query_v2::BitSetQuery>(std::move(param.roaring));
return Status::OK();
}
diff --git a/be/src/exprs/function/search/function_search_leaf.h b/be/src/exprs/function/search/function_search_leaf.h
index 4946e168e12..c4c64b0ab18 100644
--- a/be/src/exprs/function/search/function_search_leaf.h
+++ b/be/src/exprs/function/search/function_search_leaf.h
@@ -51,16 +51,6 @@ public:
segment_v2::inverted_index::query_v2::QueryPtr* out) = 0;
};
-// UNKNOWN for every row: no row matches and every row counts as NULL.
-inline segment_v2::inverted_index::query_v2::QueryPtr make_unknown_leaf_query(uint32_t num_rows) {
- auto null_bitmap = std::make_shared<roaring::Roaring>();
- if (num_rows > 0) {
- null_bitmap->addRange(0, num_rows);
- }
- return std::make_shared<segment_v2::inverted_index::query_v2::BitSetQuery>(
- std::make_shared<roaring::Roaring>(), std::move(null_bitmap));
-}
-
// Compiles a leaf into the lazy query_v2 query the engine evaluates on the field's bound source,
// after the source resolved the leaf's terms in one dictionary batch.
class LazyLeafCompiler final : public SearchLeafCompiler {
@@ -78,7 +68,7 @@ private:
};
// Answers a comparison leaf from a scalar (BKD) index through its iterator.
-// Every other leaf on such an index is UNKNOWN.
+// Unsupported leaves and invalid values fail the query.
class ScalarLeafCompiler final : public SearchLeafCompiler {
public:
ScalarLeafCompiler(segment_v2::IndexIterator* iterator, DataTypePtr column_type,
diff --git a/be/src/exprs/function/search/search_lowering.cpp b/be/src/exprs/function/search/search_lowering.cpp
index 3cf0b952474..5d681588d93 100644
--- a/be/src/exprs/function/search/search_lowering.cpp
+++ b/be/src/exprs/function/search/search_lowering.cpp
@@ -59,15 +59,6 @@ Occur to_occur(TSearchOccur::type occur) {
}
}
-NodePtr lower_direct_index_leaf(const std::string& clause_type, FieldRef field,
- const std::string& value) {
- if (clause_type == "TERM" || clause_type == "EXACT") {
- return make_node(
- Compare {.field = std::move(field), .op = CompareOp::kEqual, .value = value});
- }
- return make_node(Unknown {.field = std::move(field)});
-}
-
// Normalizes the literal text between a wildcard's '*' and '?' operators, so that a normalizer
// never reads them and they stay operators.
Status normalize_wildcard(const FieldProps& props, FieldCatalog& catalog,
@@ -104,10 +95,6 @@ Status lower_pattern_leaf(const std::string& clause_type, const FieldProps& prop
.pattern = "^(" + value + ")$"});
return Status::OK();
}
- if (clause_type == "WILDCARD" && value == "*") {
- *out = make_node(Exists {.field = std::move(field)});
- return Status::OK();
- }
const bool prefix = clause_type == "PREFIX";
// The DSL keeps a PREFIX value's trailing '*'.
std::string pattern =
@@ -172,11 +159,20 @@ Status lower_leaf(const TSearchClause& clause, const LoweringOptions& options,
catalog.resolve(clause.field_name, search_clause_query_type(clause_type), &props));
FieldRef field {.name = clause.field_name, .binding = props.binding};
if (!props.bound) {
- *out = make_node(Unknown {.field = std::move(field)});
+ *out = make_node(Empty {.field = std::move(field)});
+ return Status::OK();
+ }
+ if (clause_type == "WILDCARD" && value == "*") {
+ *out = make_node(Exists {.field = std::move(field)});
return Status::OK();
}
if (props.direct_index) {
- *out = lower_direct_index_leaf(clause_type, std::move(field), value);
+ if (clause_type != "TERM" && clause_type != "EXACT") {
+ return Status::NotSupported("SEARCH clause {} is unsupported for field {}", clause_type,
+ field.name);
+ }
+ *out = make_node(
+ Compare {.field = std::move(field), .op = CompareOp::kEqual, .value = value});
return Status::OK();
}
if (is_pattern_clause(clause_type)) {
diff --git a/be/src/exprs/function/search/search_predicate_collector.cpp b/be/src/exprs/function/search/search_predicate_collector.cpp
index f8b8a8614c4..e25e1ee279d 100644
--- a/be/src/exprs/function/search/search_predicate_collector.cpp
+++ b/be/src/exprs/function/search/search_predicate_collector.cpp
@@ -144,6 +144,11 @@ Status SearchPredicateCollector::collect_from_clause(const TSearchClause& clause
const FieldBindingMap& field_bindings,
CollectInfoMap* collect_infos) {
const std::string& clause_type = clause.clause_type;
+ // Lucene pure NOT injects a fieldless constant-score match-all child.
+ if (clause_type == "MATCH_ALL_DOCS") {
+ return Status::OK();
+ }
+
if (clause_type == "NESTED") {
return Status::Error<ErrorCode::INVERTED_INDEX_NOT_SUPPORTED>(
"Scoring nested search clauses is not supported");
diff --git a/be/src/exprs/function/search/variant_inverted_index_search.cpp b/be/src/exprs/function/search/variant_inverted_index_search.cpp
index a6af0706e3a..14b7daf9fa2 100644
--- a/be/src/exprs/function/search/variant_inverted_index_search.cpp
+++ b/be/src/exprs/function/search/variant_inverted_index_search.cpp
@@ -156,16 +156,6 @@ Status FieldReaderResolver::resolve(const std::string& field_name,
auto iterator_it = _iterators.find(field_name);
if (iterator_it == _iterators.end() || iterator_it->second == nullptr) {
- if (is_variant_sub) {
- add_search_binding_diagnostic(
- _context,
- fmt::format("[VariantSearchBinding] phase=field_resolve result=no_iterator "
- "logical_field={} stored_field={} query_type={} "
- "reason=iterator_not_found",
- field_name, stored_field_name, query_type_to_string(query_type)));
- *binding = FieldReaderBinding();
- return Status::OK();
- }
return Status::Error<ErrorCode::INVERTED_INDEX_FILE_NOT_FOUND>(
"iterator not found for field '{}'", field_name);
}
@@ -376,26 +366,6 @@ Status FieldReaderResolver::analyzer_context_for(const std::string& binding_key,
return Status::OK();
}
-std::shared_ptr<roaring::Roaring> VariantSearchNullBitmapAdapter::bitmap_for(
- const query_v2::Scorer& /*scorer*/, const std::string& logical_field) const {
- if (logical_field.empty()) {
- return nullptr;
- }
- auto* iterator = _resolver.get_iterator(logical_field);
- if (iterator == nullptr) {
- return nullptr;
- }
- auto has_null = iterator->has_null();
- if (!has_null.has_value() || !has_null.value()) {
- return nullptr;
- }
- segment_v2::InvertedIndexQueryCacheHandle cache_handle;
- // A failed read fails the search: read as no NULL rows, the rows would turn TRUE under NOT.
- THROW_IF_ERROR(iterator->read_null_bitmap(&cache_handle));
- auto bitmap = cache_handle.get_bitmap();
- return bitmap == nullptr ? nullptr : std::make_shared<roaring::Roaring>(*bitmap);
-}
-
void populate_variant_search_binding_context(const FieldReaderResolver& resolver,
query_v2::QueryExecutionContext* exec_ctx) {
DCHECK(exec_ctx != nullptr);
@@ -698,9 +668,34 @@ Status VariantNestedSearchEvaluator::evaluate(
return map_variant_nested_leaf_query_to_active_group(mapper_context,
logical_field_name, query);
});
+ resolver.set_leaf_num_rows_resolver([mapper_context](const std::string& logical_field_name,
+ uint32_t* num_rows) {
+ if (!starts_with_root_field(logical_field_name, mapper_context.root_field) ||
+ logical_field_name.size() <= mapper_context.root_field.size()) {
+ return Status::OK();
+ }
+ const auto relative_path = logical_field_name.substr(mapper_context.root_field.size() + 1);
+ auto [leaf_found, leaf_chain, leaf_path] =
+ mapper_context.variant_reader->collect_nested_group_chain(relative_path);
+ if (!leaf_found) {
+ return Status::OK();
+ }
+ DORIS_CHECK(!leaf_chain.empty());
+ uint64_t count = 0;
+ RETURN_IF_ERROR(mapper_context.read_provider->get_total_elements(
+ mapper_context.column_iter_opts, leaf_chain.back(), &count));
+ if (count > std::numeric_limits<uint32_t>::max()) {
+ return Status::InvalidArgument("nested leaf element_count exceeds uint32_t max");
+ }
+ *num_rows = static_cast<uint32_t>(count);
+ return Status::OK();
+ });
struct ScopedLeafMapperReset {
FieldReaderResolver& resolver;
- ~ScopedLeafMapperReset() { resolver.set_leaf_query_mapper(nullptr); }
+ ~ScopedLeafMapperReset() {
+ resolver.set_leaf_query_mapper(nullptr);
+ resolver.set_leaf_num_rows_resolver(nullptr);
+ }
} mapper_reset {resolver};
// The elements map to their parents without scores, so the tree compiles unscored.
RETURN_IF_ERROR(_function_search.build_query_recursive(
@@ -711,7 +706,7 @@ Status VariantNestedSearchEvaluator::evaluate(
return Status::OK();
}
- VariantSearchNullBitmapAdapter null_resolver(resolver);
+ SearchNullBitmapResolver null_resolver;
query_v2::QueryExecutionContext exec_ctx = build_variant_search_query_execution_context(
static_cast<uint32_t>(total_elements), resolver, &null_resolver);
diff --git a/be/src/exprs/function/search/variant_inverted_index_search.h b/be/src/exprs/function/search/variant_inverted_index_search.h
index 59d84c1d775..1edac0b2b0c 100644
--- a/be/src/exprs/function/search/variant_inverted_index_search.h
+++ b/be/src/exprs/function/search/variant_inverted_index_search.h
@@ -143,6 +143,15 @@ public:
return (it != _iterators.end()) ? it->second : nullptr;
}
+ void set_leaf_num_rows_resolver(std::function<Status(const std::string&, uint32_t*)> resolver) {
+ _leaf_num_rows_resolver = std::move(resolver);
+ }
+
+ Status leaf_num_rows(const std::string& field_name, uint32_t* num_rows) const {
+ return _leaf_num_rows_resolver ? _leaf_num_rows_resolver(field_name, num_rows)
+ : Status::OK();
+ }
+
void set_leaf_query_mapper(SearchLeafQueryMapper mapper) {
_leaf_query_mapper = std::move(mapper);
}
@@ -175,18 +184,17 @@ private:
// The indexes the sources read, closed after them and latest first.
std::vector<std::unique_ptr<segment_v2::OpenedIndex>> _opened_indexes;
SearchLeafQueryMapper _leaf_query_mapper;
+ std::function<Status(const std::string&, uint32_t*)> _leaf_num_rows_resolver;
};
-class VariantSearchNullBitmapAdapter final : public inverted_index::query_v2::NullBitmapResolver {
+// Physical NULLs are excluded by SEARCH leaves. Internal Boolean scorers,
+// including NestedGroup element queries, only see matching/non-matching sets.
+class SearchNullBitmapResolver final : public inverted_index::query_v2::NullBitmapResolver {
public:
- explicit VariantSearchNullBitmapAdapter(const FieldReaderResolver& resolver)
- : _resolver(resolver) {}
-
- std::shared_ptr<roaring::Roaring> bitmap_for(const inverted_index::query_v2::Scorer& scorer,
- const std::string& logical_field) const override;
-
-private:
- const FieldReaderResolver& _resolver;
+ std::shared_ptr<roaring::Roaring> bitmap_for(const inverted_index::query_v2::Scorer&,
+ const std::string&) const override {
+ return nullptr;
+ }
};
void populate_variant_search_binding_context(
diff --git a/be/src/exprs/vsearch.cpp b/be/src/exprs/vsearch.cpp
index e74cbb672ad..0b213a33a5e 100644
--- a/be/src/exprs/vsearch.cpp
+++ b/be/src/exprs/vsearch.cpp
@@ -33,6 +33,8 @@
#include "runtime/runtime_state.h"
#include "storage/index/inverted/inverted_index_reader.h"
#include "storage/schema.h"
+#include "storage/segment/segment.h"
+#include "storage/segment/variant/variant_column_reader.h"
namespace doris {
using namespace segment_v2;
@@ -59,6 +61,61 @@ void add_search_binding_diagnostic(const IndexExecContext* index_context,
}
}
+// A missing iterator is not evidence of a missing value: doc/sparse paths and
+// old rowsets may contain values without an executable SEARCH index.
+Status verify_missing_variant_path(const TSearchFieldBinding& binding, int column_index,
+ IndexExecContext* index_context) {
+ auto* segment = index_context->segment();
+ if (segment == nullptr || !binding.__isset.parent_field_name ||
+ !binding.__isset.subcolumn_path) {
+ return Status::Error<ErrorCode::INVERTED_INDEX_NOT_SUPPORTED>(
+ "SEARCH cannot establish missing path '{}'", binding.field_name);
+ }
+ const auto& read_schema = index_context->read_schema();
+ DORIS_CHECK_GE(column_index, 0);
+ DORIS_CHECK_LT(static_cast<size_t>(column_index), read_schema->num_read_columns());
+ const auto& column = *read_schema->column(column_index);
+ const auto& opts = index_context->column_iter_opts();
+ DORIS_CHECK(opts.stats != nullptr);
+ StorageReadOptions read_options(*opts.stats);
+ read_options.io_ctx = opts.io_ctx;
+ std::shared_ptr<VariantColumnReader> reader;
+ Status st = segment->get_variant_root_reader(column, read_options, &reader);
+ if (st.is<ErrorCode::NOT_FOUND>()) {
+ // After DROP v(uid=1), ADD v(uid=2), an old segment cannot contain this scan's root.
+ // Resolve by the current uid so historical values of the same name cannot match.
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(st);
+ DORIS_CHECK(reader != nullptr);
+ RETURN_IF_ERROR(reader->load_external_meta_once(opts.stats, &opts.io_ctx));
+ const PathInData path(binding.subcolumn_path);
+ const auto* stats = reader->get_stats();
+ // Doc statistics enumerate stored paths without the sparse statistics limit. For example,
+ // a segment containing only `string4` can prove `string5` absent even in doc mode.
+ // An opaque root/ancestor value may contain the requested path, so it remains unavailable.
+ bool in_doc = stats->existed_in_doc_value_column("") ||
+ stats->has_prefix_path_in_doc_value_column(path.get_path() + ".");
+ std::string ancestor = path.get_path();
+ while (!in_doc) {
+ in_doc = stats->existed_in_doc_value_column(ancestor);
+ const auto dot = ancestor.rfind('.');
+ if (dot == std::string::npos) {
+ break;
+ }
+ ancestor.resize(dot);
+ }
+ // Truncated sparse statistics cannot prove absence, nor can an existing unindexed path.
+ if (reader->get_subcolumn_meta_by_path(path) != nullptr ||
+ reader->has_prefix_path(path, opts.stats, &opts.io_ctx) ||
+ reader->exist_in_sparse_column(path) || reader->is_exceeded_sparse_column_limit() ||
+ in_doc || stats->has_nested_group) {
+ return Status::Error<ErrorCode::INVERTED_INDEX_NOT_SUPPORTED>(
+ "SEARCH index unavailable for Variant field '{}'", binding.field_name);
+ }
+ return Status::OK();
+}
+
void bind_nested_column(const VSearchExpr& expr, const VSlotRef& slot_ref,
const TSearchFieldBinding* binding, const IndexExecContext& index_context,
SearchInputBundle* bundle) {
@@ -119,6 +176,11 @@ Status collect_slot_search_input(const VSearchExpr& expr, const VSlotRef& slot_r
"reason=slot_iterator_missing",
field_name, column_index));
}
+ if (!is_variant_subcolumn) {
+ return Status::Error<ErrorCode::INVERTED_INDEX_NOT_SUPPORTED>(
+ "SEARCH index unavailable for field '{}'", field_name);
+ }
+ RETURN_IF_ERROR(verify_missing_variant_path(*binding, column_index, index_context));
return Status::OK();
}
@@ -142,6 +204,16 @@ Status collect_slot_search_input(const VSearchExpr& expr, const VSlotRef& slot_r
return Status::OK();
}
+const VSlotRef& variant_root_slot(const VExpr& expr) {
+ const VExpr* root = &expr;
+ while (root->fn().name.function_name == "element_at") {
+ DORIS_CHECK(!root->children().empty());
+ root = root->children().front().get();
+ }
+ DORIS_CHECK(root->is_slot_ref());
+ return *assert_cast<const VSlotRef*>(root);
+}
+
Status collect_search_inputs(const VSearchExpr& expr, VExprContext* context,
SearchInputBundle* bundle) {
DCHECK(bundle != nullptr);
@@ -176,15 +248,10 @@ Status collect_search_inputs(const VSearchExpr& expr, VExprContext* context,
field_bindings[child_index].is_variant_subcolumn) {
// With an unmaterialized path, the ElementAt chain still reads a current root
// slot. Use it only to identify the NESTED owner, not as the leaf's index iterator.
+ const auto& root_slot = variant_root_slot(*child);
if (search_param.root.clause_type == "NESTED") {
- const VExpr* root_expr = child.get();
- while (root_expr->fn().name.function_name == "element_at") {
- DORIS_CHECK(!root_expr->children().empty());
- root_expr = root_expr->children().front().get();
- }
- DORIS_CHECK(root_expr->is_slot_ref());
- bind_nested_column(expr, *assert_cast<const VSlotRef*>(root_expr),
- &field_bindings[child_index], *index_context, bundle);
+ bind_nested_column(expr, root_slot, &field_bindings[child_index],
+ *index_context, bundle);
}
// Variant subcolumn not materialized - skip, will create empty BitSetQuery in function_search
add_search_binding_diagnostic(
@@ -199,6 +266,8 @@ Status collect_search_inputs(const VSearchExpr& expr, VExprContext* context,
field_bindings[child_index].__isset.subcolumn_path
? field_bindings[child_index].subcolumn_path
: ""));
+ RETURN_IF_ERROR(verify_missing_variant_path(
+ field_bindings[child_index], root_slot.column_id(), index_context.get()));
++child_index;
continue;
}
@@ -281,21 +350,6 @@ Status VSearchExpr::evaluate_inverted_index(VExprContext* context, uint32_t segm
VLOG_DEBUG << "VSearchExpr: bundle.iterators.size()=" << bundle.iterators.size();
- const bool is_nested_query = _search_param.root.clause_type == "NESTED";
- if (bundle.iterators.empty() && !is_nested_query) {
- LOG(WARNING) << "VSearchExpr: No indexed columns available for evaluation, DSL: "
- << _original_dsl;
- add_search_binding_diagnostic(
- index_context.get(),
- fmt::format("[VariantSearchBinding] phase=evaluate_search result=no_iterator "
- "dsl={} reason=no_indexed_columns",
- _original_dsl));
- auto empty_bitmap = InvertedIndexResultBitmap(std::make_shared<roaring::Roaring>(),
- std::make_shared<roaring::Roaring>());
- index_context->set_index_result_for_expr(this, std::move(empty_bitmap));
- return Status::OK();
- }
-
auto index_query_context = index_context->get_index_query_context();
auto function = std::make_shared<FunctionSearch>();
diff --git a/be/src/storage/index/inverted/query/engine/null_masked_query.h b/be/src/storage/index/inverted/query/engine/null_masked_query.h
index ad402a76da0..042a8443d82 100644
--- a/be/src/storage/index/inverted/query/engine/null_masked_query.h
+++ b/be/src/storage/index/inverted/query/engine/null_masked_query.h
@@ -33,10 +33,14 @@ namespace doris::segment_v2::inverted_index::query_v2 {
// The rows `inner` matches outside `nulls`. A row in `nulls` is UNKNOWN even where `inner` lists
// it, as an array column's index does for the values a NULL array still holds.
+// SEARCH sets nulls_are_false so those rows are nonmatches before Boolean composition.
class NullMaskedScorer final : public Scorer {
public:
- NullMaskedScorer(ScorerPtr inner, std::shared_ptr<roaring::Roaring> nulls)
- : _inner(std::move(inner)), _nulls(std::move(nulls)) {
+ NullMaskedScorer(ScorerPtr inner, std::shared_ptr<roaring::Roaring> nulls,
+ bool nulls_are_false = false)
+ : _inner(std::move(inner)),
+ _nulls(std::move(nulls)),
+ _nulls_are_false(nulls_are_false) {
if (const auto* truths = _inner->get_true_bitmap(); truths != nullptr) {
_true_rows = *truths - *_nulls;
}
@@ -62,10 +66,15 @@ public:
return _true_rows.has_value() ? &*_true_rows : nullptr;
}
- bool has_null_bitmap(const NullBitmapResolver* /*resolver*/ = nullptr) override { return true; }
+ bool has_null_bitmap(const NullBitmapResolver* /*resolver*/ = nullptr) override {
+ return !_nulls_are_false;
+ }
// The inner rows' own UNKNOWN rows stay UNKNOWN too.
const roaring::Roaring* get_null_bitmap(const NullBitmapResolver* resolver = nullptr) override {
+ if (_nulls_are_false) {
+ return nullptr;
+ }
if (!_null_rows.has_value()) {
const auto* inner_nulls =
_inner->has_null_bitmap(resolver) ? _inner->get_null_bitmap(resolver) : nullptr;
@@ -86,13 +95,19 @@ private:
std::shared_ptr<roaring::Roaring> _nulls;
std::optional<roaring::Roaring> _true_rows;
std::optional<roaring::Roaring> _null_rows;
+ bool _nulls_are_false;
uint32_t _doc = TERMINATED;
};
class NullMaskedWeight final : public Weight {
public:
- NullMaskedWeight(WeightPtr inner, std::shared_ptr<roaring::Roaring> nulls)
- : _inner(std::move(inner)), _nulls(std::move(nulls)) {}
+ NullMaskedWeight(WeightPtr inner, std::shared_ptr<roaring::Roaring> nulls,
+ bool nulls_are_false = false)
+ : _inner(std::move(inner)),
+ _nulls(std::move(nulls)),
+ _nulls_are_false(nulls_are_false) {}
+
+ bool is_two_valued() const override { return _nulls_are_false; }
ScorerPtr scorer(const QueryExecutionContext& context,
const std::string& binding_key) override {
@@ -100,7 +115,7 @@ public:
if (inner == nullptr) {
return nullptr;
}
- return std::make_shared<NullMaskedScorer>(std::move(inner), _nulls);
+ return std::make_shared<NullMaskedScorer>(std::move(inner), _nulls, _nulls_are_false);
}
bool lists_rows(const QueryExecutionContext& context,
@@ -114,27 +129,36 @@ public:
auto rows = _inner->listed_rows(context, binding_key, candidates);
rows.null_rows |= candidates == nullptr ? *_nulls : *_nulls & *candidates;
rows.true_rows -= rows.null_rows;
+ if (_nulls_are_false) {
+ rows.null_rows = roaring::Roaring();
+ }
return rows;
}
private:
WeightPtr _inner;
std::shared_ptr<roaring::Roaring> _nulls;
+ bool _nulls_are_false;
};
-// A leaf whose rows in `nulls` are UNKNOWN whatever it matches there.
+// A leaf whose rows in `nulls` are UNKNOWN (SQL) or FALSE (SEARCH).
class NullMaskedQuery final : public Query {
public:
- NullMaskedQuery(QueryPtr inner, std::shared_ptr<roaring::Roaring> nulls)
- : _inner(std::move(inner)), _nulls(std::move(nulls)) {}
+ NullMaskedQuery(QueryPtr inner, std::shared_ptr<roaring::Roaring> nulls,
+ bool nulls_are_false = false)
+ : _inner(std::move(inner)),
+ _nulls(std::move(nulls)),
+ _nulls_are_false(nulls_are_false) {}
WeightPtr weight(bool enable_scoring) override {
- return std::make_shared<NullMaskedWeight>(_inner->weight(enable_scoring), _nulls);
+ return std::make_shared<NullMaskedWeight>(_inner->weight(enable_scoring), _nulls,
+ _nulls_are_false);
}
private:
QueryPtr _inner;
std::shared_ptr<roaring::Roaring> _nulls;
+ bool _nulls_are_false;
};
} // namespace doris::segment_v2::inverted_index::query_v2
diff --git a/be/src/storage/index/inverted/reader/collection_statistics.cpp b/be/src/storage/index/inverted/reader/collection_statistics.cpp
index 1e691caaf15..4cf3b8d287b 100644
--- a/be/src/storage/index/inverted/reader/collection_statistics.cpp
+++ b/be/src/storage/index/inverted/reader/collection_statistics.cpp
@@ -39,6 +39,28 @@
#include "util/uid_util.h"
namespace doris {
+namespace {
+
+Result<std::shared_ptr<lucene::search::IndexSearcher>> open_optional_clucene_stats_searcher(
+ const segment_v2::IndexFileReader& file_reader, const TabletIndex* index,
+ const io::IOContext* io_ctx) {
+ auto compound = file_reader.open(index, io_ctx);
+ if (!compound.has_value()) {
+ // Variant rowsets can contain only `a` or only `b`. Missing `a` must not discard
+ // this segment's available `b` statistics when scoring SEARCH('a:hit OR b:hit').
+ const auto& status = compound.error();
+ if (status.is<ErrorCode::INVERTED_INDEX_FILE_NOT_FOUND>() ||
+ status.is<ErrorCode::INVERTED_INDEX_BYPASS>()) {
+ return std::shared_ptr<lucene::search::IndexSearcher> {};
+ }
+ return ResultError(status);
+ }
+ auto* reader = lucene::index::IndexReader::open(compound.value().get());
+ return std::make_shared<lucene::search::IndexSearcher>(reader, true);
+}
+
+} // namespace
+
namespace collection_statistics_detail {
Result<SniiScoringSegmentStats> resolve_snii_scoring_segment(uint64_t index_doc_count,
@@ -303,9 +325,11 @@ Status CollectionStatistics::process_segment(const RowsetSharedPtr& rowset,
InvertedIndexCacheHandle inverted_index_cache_handle;
#endif
#ifdef BE_TEST
- auto compound_reader = DORIS_TRY(idx_file_reader->open(collect_info.index_meta, io_ctx));
- auto* reader = lucene::index::IndexReader::open(compound_reader.get());
- owned_index_searcher = std::make_shared<lucene::search::IndexSearcher>(reader, true);
+ owned_index_searcher = DORIS_TRY(open_optional_clucene_stats_searcher(
+ *idx_file_reader, collect_info.index_meta, io_ctx));
+ if (owned_index_searcher == nullptr) {
+ continue; // This field has no index in this segment; collect the other fields.
+ }
index_searcher = owned_index_searcher.get();
index_reader = index_searcher->getReader();
#else
@@ -314,11 +338,13 @@ Status CollectionStatistics::process_segment(const RowsetSharedPtr& rowset,
if (!InvertedIndexSearcherCache::instance()->lookup(searcher_cache_key,
&inverted_index_cache_handle)) {
- auto compound_reader =
- DORIS_TRY(idx_file_reader->open(collect_info.index_meta, io_ctx));
- auto* reader = lucene::index::IndexReader::open(compound_reader.get());
- size_t reader_size = reader->getTermInfosRAMUsed();
- auto searcher_ptr = std::make_shared<lucene::search::IndexSearcher>(reader, true);
+ auto searcher_ptr = DORIS_TRY(open_optional_clucene_stats_searcher(
+ *idx_file_reader, collect_info.index_meta, io_ctx));
+ if (searcher_ptr == nullptr) {
+ continue; // This field has no index in this segment; collect the other fields.
+ }
+ index_reader = searcher_ptr->getReader();
+ size_t reader_size = index_reader->getTermInfosRAMUsed();
auto* cache_value = new CluceneSearcherCacheValue(std::move(searcher_ptr), reader_size,
UnixMillis());
InvertedIndexSearcherCache::instance()->insert(searcher_cache_key, cache_value,
diff --git a/be/test/exprs/function/search/function_search_null_test.cpp b/be/test/exprs/function/search/function_search_null_test.cpp
index ecd6df3c3ef..5fc2f6bba1f 100644
--- a/be/test/exprs/function/search/function_search_null_test.cpp
+++ b/be/test/exprs/function/search/function_search_null_test.cpp
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-// How SEARCH keeps the rows a NULL field leaves UNKNOWN, read through fake text indexes.
+// SEARCH treats NULL fields as nonmatches before Boolean composition.
#include <gen_cpp/Exprs_types.h>
#include <gtest/gtest.h>
@@ -43,6 +43,9 @@
#include "storage/index/inverted/common/inverted_index_cache.h"
#include "storage/index/inverted/inverted_index_iterator.h"
#include "storage/index/inverted/inverted_index_reader.h"
+#include "storage/index/inverted/query/engine/bit_set_query/bit_set_query.h"
+#include "storage/index/inverted/query/engine/boolean_query/boolean_query_builder.h"
+#include "storage/index/inverted/query/engine/null_masked_query.h"
#include "storage/index/inverted/query/scoring/collection_similarity.h"
#include "storage/index/inverted/query/spi/fake_index_source.h"
#include "storage/index/inverted/reader/collection_statistics.h"
@@ -182,6 +185,7 @@ public:
field.meta = make_index(static_cast<int64_t>(_fields.size()));
field.reader = std::make_shared<FakeTextIndexReader>(
&field.meta, std::make_shared<UnopenedIndexFileReader>("/tmp/search_null_idx"));
+ field.reader->source->set_doc_count(_row_count);
field.iterator.add_reader(segment_v2::InvertedIndexReaderType::FULLTEXT, field.reader);
DataTypePtr type = std::make_shared<DataTypeString>();
if (array) {
@@ -214,7 +218,7 @@ public:
}
Status evaluate(const TSearchClause& root, const EvaluateOptions& options,
- InvertedIndexResultBitmap* result) {
+ InvertedIndexResultBitmap* result, CollectionSimilarityPtr* scores = nullptr) {
auto context = std::make_shared<segment_v2::IndexQueryContext>();
context->stats = &_stats;
if (options.scoring) {
@@ -228,13 +232,16 @@ public:
_runtime_state.set_query_options(query_options);
context->runtime_state = &_runtime_state;
}
+ if (scores != nullptr) {
+ *scores = context->collection_similarity;
+ }
FunctionSearch search;
return search.evaluate_inverted_index_with_search_param(
param(root, options), _types, _iterators, _row_count, *result, options.cache,
nullptr, nullptr, context);
}
- // The TRUE and UNKNOWN rows of `root`.
+ // The match set and empty NULL bitmap of `root`.
void expect(const TSearchClause& root, const EvaluateOptions& options,
const roaring::Roaring& true_rows, const roaring::Roaring& null_rows) {
InvertedIndexResultBitmap result;
@@ -327,9 +334,8 @@ private:
} // namespace
-// A term the dictionary lacks matches no row, and the rows where its field is NULL stay
-// UNKNOWN, alone or listed with other terms of the field.
-TEST(FunctionSearchNullTest, AbsentTermKeepsTheFieldNullRows) {
+// An absent term matches no row. Its negation includes every row, including NULL.
+TEST(FunctionSearchNullTest, AbsentTermNegationIncludesTheFieldNullRows) {
for (const bool batches : {false, true}) {
for (const bool scoring : {false, true}) {
SCOPED_TRACE(::testing::Message() << "batches=" << batches << " scoring=" << scoring);
@@ -342,20 +348,19 @@ TEST(FunctionSearchNullTest, AbsentTermKeepsTheFieldNullRows) {
const EvaluateOptions options = scored_if(scoring);
fixture.expect(compound("NOT", {leaf("TERM", "body", "missing")}), options,
- rows({0, 1, 2}), rows({3}));
+ rows({0, 1, 2, 3}), rows({}));
fixture.expect(compound("NOT", {compound("AND", {leaf("TERM", "body", "missing"),
leaf("TERM", "body", "foo")})}),
- options, rows({0, 1, 2}), rows({3}));
+ options, rows({0, 1, 2, 3}), rows({}));
fixture.expect(compound("NOT", {compound("OR", {leaf("TERM", "body", "missing"),
leaf("TERM", "body", "absent")})}),
- options, rows({0, 1, 2}), rows({3}));
+ options, rows({0, 1, 2, 3}), rows({}));
}
}
}
-// A scored conjunction whose terms share no row still leaves the field's NULL rows UNKNOWN, as
-// the unscored one does.
-TEST(FunctionSearchNullTest, EmptyScoredConjunctionKeepsTheFieldNullRows) {
+// A disjoint conjunction has no matches; negation includes NULL with and without scoring.
+TEST(FunctionSearchNullTest, EmptyScoredConjunctionNegationIncludesNullRows) {
for (const bool scoring : {false, true}) {
SCOPED_TRACE(::testing::Message() << "scoring=" << scoring);
NullSearchFixture fixture(3);
@@ -367,14 +372,13 @@ TEST(FunctionSearchNullTest, EmptyScoredConjunctionKeepsTheFieldNullRows) {
EvaluateOptions options = scored_if(scoring);
options.default_operator = "and";
- fixture.expect(compound("NOT", {leaf("TERM", "body", "foo/bar")}), options, rows({0, 1}),
- rows({2}));
+ fixture.expect(compound("NOT", {leaf("TERM", "body", "foo/bar")}), options, rows({0, 1, 2}),
+ rows({}));
}
}
-// A TERM that must hold some of its analyzed terms counts them two-valued; the rows where its
-// field is NULL still stay UNKNOWN.
-TEST(FunctionSearchNullTest, TermThresholdKeepsTheFieldNullRows) {
+// A term threshold excludes NULL payloads and its negation includes NULL rows.
+TEST(FunctionSearchNullTest, TermThresholdUsesTwoValuedMatching) {
for (const bool scoring : {false, true}) {
SCOPED_TRACE(::testing::Message() << "scoring=" << scoring);
NullSearchFixture fixture(4);
@@ -386,15 +390,14 @@ TEST(FunctionSearchNullTest, TermThresholdKeepsTheFieldNullRows) {
EvaluateOptions options = scored_if(scoring);
options.minimum_should_match = 1;
- fixture.expect(leaf("TERM", "body", "foo/bar"), options, rows({0, 1}), rows({3}));
- fixture.expect(compound("NOT", {leaf("TERM", "body", "foo/bar")}), options, rows({2}),
- rows({3}));
+ fixture.expect(leaf("TERM", "body", "foo/bar"), options, rows({0, 1}), rows({}));
+ fixture.expect(compound("NOT", {leaf("TERM", "body", "foo/bar")}), options, rows({2, 3}),
+ rows({}));
}
}
-// A value the analyzer turns into no token matches no row, and the rows where the field is NULL
-// stay UNKNOWN.
-TEST(FunctionSearchNullTest, AnalyzerEmptyLeafKeepsTheFieldNullRows) {
+// An analyzer-empty value matches no rows; its negation includes NULL rows.
+TEST(FunctionSearchNullTest, AnalyzerEmptyLeafNegationIncludesNullRows) {
for (const std::string clause_type : {"TERM", "PHRASE", "MATCH", "ANY", "ALL"}) {
for (const bool scoring : {false, true}) {
SCOPED_TRACE(::testing::Message() << clause_type << " scoring=" << scoring);
@@ -404,14 +407,14 @@ TEST(FunctionSearchNullTest, AnalyzerEmptyLeafKeepsTheFieldNullRows) {
body.nulls = rows({2});
fixture.expect(compound("NOT", {leaf(clause_type, "body", "!!!")}), scored_if(scoring),
- rows({0, 1}), rows({2}));
+ rows({0, 1, 2}), rows({}));
}
}
}
// An array column can keep values in a row whose array is NULL, and its index lists that row
-// under them. The row is UNKNOWN in every Boolean, and FALSE in a two-valued one.
-TEST(FunctionSearchNullTest, ArrayNullRowWithValuesIsUnknown) {
+// under them. The row is FALSE before every Boolean operation.
+TEST(FunctionSearchNullTest, ArrayNullPayloadIsExcludedBeforeBooleanComposition) {
for (const bool scoring : {false, true}) {
SCOPED_TRACE(::testing::Message() << "scoring=" << scoring);
NullSearchFixture fixture(4);
@@ -423,14 +426,14 @@ TEST(FunctionSearchNullTest, ArrayNullRowWithValuesIsUnknown) {
other.source->add("x", {1});
const EvaluateOptions options = scored_if(scoring);
- fixture.expect(leaf("TERM", "arr", "alpha"), options, rows({0}), rows({1}));
+ fixture.expect(leaf("TERM", "arr", "alpha"), options, rows({0}), rows({}));
fixture.expect(
compound("OR", {leaf("TERM", "arr", "alpha"), leaf("TERM", "other", "omega")}),
- options, rows({0, 2}), rows({1}));
+ options, rows({0, 2}), rows({}));
fixture.expect(compound("AND", {leaf("TERM", "arr", "alpha"), leaf("TERM", "other", "x")}),
- options, rows({}), rows({1}));
- fixture.expect(compound("NOT", {leaf("TERM", "arr", "alpha")}), options, rows({2, 3}),
- rows({1}));
+ options, rows({}), rows({}));
+ fixture.expect(compound("NOT", {leaf("TERM", "arr", "alpha")}), options, rows({1, 2, 3}),
+ rows({}));
fixture.expect(compound("OCCUR_BOOLEAN",
{occur(leaf("TERM", "arr", "alpha"), TSearchOccur::SHOULD),
occur(leaf("TERM", "other", "omega"), TSearchOccur::SHOULD)}),
@@ -438,7 +441,7 @@ TEST(FunctionSearchNullTest, ArrayNullRowWithValuesIsUnknown) {
if (scoring) {
fixture.expect(
compound("OR", {leaf("TERM", "arr", "alpha"), leaf("TERM", "other", "omega")}),
- scored_if(true, 2), rows({0, 2}), rows({1}));
+ scored_if(true, 2), rows({0, 2}), rows({}));
}
}
}
@@ -461,9 +464,8 @@ TEST(FunctionSearchNullTest, FailedNullBitmapReadFailsTheSearch) {
}
}
-// The cache keeps a result's TRUE and UNKNOWN rows as two entries; one found without the other
-// is computed again rather than taken as having no UNKNOWN rows.
-TEST(FunctionSearchNullTest, DslCacheEntryWithoutItsNullRowsIsAMiss) {
+// A legacy three-valued cache entry cannot supply a two-valued SEARCH result.
+TEST(FunctionSearchNullTest, OldThreeValuedDslCacheEntryIsAMiss) {
ScopedQueryCache cache;
NullSearchFixture fixture(4);
auto& body = fixture.add_field("body");
@@ -480,15 +482,43 @@ TEST(FunctionSearchNullTest, DslCacheEntryWithoutItsNullRowsIsAMiss) {
InvertedIndexQueryCacheHandle handle;
cache.get()->insert(key, std::make_shared<roaring::Roaring>(rows({1, 2})), &handle);
- fixture.expect(root, options, rows({1, 2}), rows({3}));
+ fixture.expect(root, options, rows({1, 2, 3}), rows({}));
EXPECT_FALSE(body.source->opened.empty());
- // The computed result is cached with its UNKNOWN rows.
+ // The computed two-valued match set is cached in the new namespace.
body.source->opened.clear();
- fixture.expect(root, options, rows({1, 2}), rows({3}));
+ fixture.expect(root, options, rows({1, 2, 3}), rows({}));
EXPECT_TRUE(body.source->opened.empty());
}
+// Construct the backend clause directly: existence includes an empty non-NULL array and
+// excludes NULL arrays even if their nested payload retains a posting. This contract does
+// not depend on how an FE version parses the field-scoped '*' syntax.
+TEST(FunctionSearchNullTest, FieldExistenceUsesBackendClausesIndependentlyOfFrontend) {
+ ScopedQueryCache cache;
+ NullSearchFixture fixture(4);
+ auto& array = fixture.add_field("a", true);
+ array.source->add("hit", {0, 1});
+ array.nulls = rows({0, 3});
+ const auto exists = leaf("WILDCARD", "a", "*");
+ for (const bool scoring : {false, true}) {
+ for (const bool cached : {false, true}) {
+ auto options = scored_if(scoring);
+ options.cache = cached;
+ for (int round = 0; round < 2; ++round) {
+ SCOPED_TRACE(::testing::Message()
+ << "scoring=" << scoring << " cache=" << cached << " round=" << round);
+ fixture.expect(exists, options, rows({1, 2}), rows({}));
+ fixture.expect(compound("NOT", {exists}), options, rows({0, 3}), rows({}));
+ fixture.expect(compound("AND", {exists, leaf("TERM", "a", "hit")}), options,
+ rows({1}), rows({}));
+ fixture.expect(compound("OR", {exists, leaf("TERM", "a", "hit")}), options,
+ rows({1, 2}), rows({}));
+ }
+ }
+ }
+}
+
// A term threshold leaves no row UNKNOWN, so a TopK over it keeps the threshold's block-max WAND.
// Once row 0 holds both terms, neither term alone can beat it, so WAND skips a lagging term's block.
TEST(FunctionSearchNullTest, TopKOverATermThresholdKeepsWand) {
@@ -507,8 +537,7 @@ TEST(FunctionSearchNullTest, TopKOverATermThresholdKeepsWand) {
EXPECT_GT(body.source->shallow_seeks, 0U);
}
-// A TopK takes the UNKNOWN rows from the scorer it ranks with instead of building and reading the
-// query again.
+// A TopK reads each query source once while collecting two-valued matches.
TEST(FunctionSearchNullTest, TopKReadsTheQueryOnce) {
NullSearchFixture fixture(4);
auto& body = fixture.add_field("body");
@@ -519,9 +548,225 @@ TEST(FunctionSearchNullTest, TopKReadsTheQueryOnce) {
other.nulls = rows({0});
fixture.expect(compound("OR", {leaf("TERM", "body", "foo"), leaf("TERM", "other", "bar")}),
- scored_if(true, 3), rows({0, 1, 2}), rows({3}));
+ scored_if(true, 3), rows({0, 1, 2}), rows({}));
EXPECT_EQ(std::vector<std::string> {"foo"}, body.source->opened);
EXPECT_EQ(std::vector<std::string> {"bar"}, other.source->opened);
}
+// Physical NULL rows must not match a conjunction, whether or not they retain postings.
+TEST(FunctionSearchNullTest, NullableConjunctionFiltersPhysicalNullPostings) {
+ for (const bool null_has_postings : {false, true}) {
+ SCOPED_TRACE(null_has_postings);
+ NullSearchFixture fixture(16);
+ auto& body = fixture.add_field("body");
+ body.nulls = rows({0});
+ std::vector<uint32_t> foo = {1, 2, 3, 4, 5, 6, 7};
+ std::vector<uint32_t> bar = {1, 2, 3, 4, 5};
+ if (null_has_postings) {
+ foo.insert(foo.begin(), 0);
+ bar.insert(bar.begin(), 0);
+ }
+ body.source->add("foo", foo);
+ body.source->add("bar", bar);
+ auto& selective = fixture.add_field("selective");
+ selective.source->add("needle", {0, 1});
+ fixture.expect(compound("AND", {leaf("TERM", "body", "foo"), leaf("TERM", "body", "bar"),
+ leaf("TERM", "selective", "needle")}),
+ {}, rows({1}), rows({}));
+ }
+}
+
+// A physical NULL row can retain every term. Check positive and negated results
+// across posting densities, batching capabilities, and scoring modes.
+TEST(FunctionSearchNullTest, NullableConjunctionMatchesAcrossSourceCapabilities) {
+ const std::vector<uint32_t> all {0, 1, 2, 3, 4, 5};
+ for (const size_t narrowing_terms : {0U, 1U, 2U}) {
+ for (const bool batches : {false, true}) {
+ for (const bool scoring : {false, true}) {
+ SCOPED_TRACE(::testing::Message()
+ << "narrowing_terms=" << narrowing_terms << " batches=" << batches
+ << " scoring=" << scoring);
+ NullSearchFixture fixture(6);
+ auto& body = fixture.add_field("body");
+ body.source->batches = batches;
+ body.nulls = rows({0});
+ body.source->add("foo",
+ narrowing_terms < 2 ? all : std::vector<uint32_t> {0, 1, 3, 4});
+ body.source->add("bar",
+ narrowing_terms == 0 ? all : std::vector<uint32_t> {0, 1, 3});
+ const auto root =
+ compound("AND", {leaf("TERM", "body", "foo"), leaf("TERM", "body", "bar")});
+ fixture.expect(root, scored_if(scoring),
+ narrowing_terms == 0 ? rows({1, 2, 3, 4, 5}) : rows({1, 3}),
+ rows({}));
+ fixture.expect(compound("NOT", {root}), scored_if(scoring),
+ narrowing_terms == 0 ? rows({0}) : rows({0, 2, 4, 5}), rows({}));
+ }
+ }
+ }
+}
+
+// A nullable docs-only source has no positions for batched AND scoring. Its individual
+// term scorers must retain both implicit unit frequencies and the SEARCH NULL exclusion.
+TEST(FunctionSearchNullTest, NullableDocsOnlyConjunctionPreservesUnitFrequencyScores) {
+ for (const bool batches : {false, true}) {
+ NullSearchFixture fixture(6);
+ auto& body = fixture.add_field("body");
+ body.source->batches = batches;
+ body.source->has_positions = false;
+ body.nulls = rows({0});
+ body.source->add("foo", std::vector<uint32_t> {0, 1, 3});
+ body.source->add("bar", std::vector<uint32_t> {0, 1, 4});
+ const auto reference =
+ compound("OCCUR_BOOLEAN", {occur(leaf("TERM", "body", "foo"), TSearchOccur::MUST),
+ occur(leaf("TERM", "body", "bar"), TSearchOccur::MUST)});
+ InvertedIndexResultBitmap reference_rows;
+ CollectionSimilarityPtr reference_scores;
+ ASSERT_TRUE(fixture.evaluate(reference, scored_if(true), &reference_rows, &reference_scores)
+ .ok());
+ InvertedIndexResultBitmap result;
+ CollectionSimilarityPtr scores;
+ ASSERT_TRUE(fixture.evaluate(compound("AND", {leaf("TERM", "body", "foo"),
+ leaf("TERM", "body", "bar")}),
+ scored_if(true), &result, &scores)
+ .ok());
+ EXPECT_EQ(rows({1}).toString(), result.get_data_bitmap()->toString());
+ ASSERT_EQ(1U, scores->_bm25_scores.size());
+ EXPECT_GT(scores->_bm25_scores.at(1), 0.0F);
+ EXPECT_FLOAT_EQ(reference_scores->_bm25_scores.at(1), scores->_bm25_scores.at(1));
+ }
+}
+
+// NULL payloads at the beginning, middle and end must not change surviving scores.
+// Compare standard AND/OR with the equivalent occurrence Boolean.
+TEST(FunctionSearchNullTest, NullableBooleanTermsPreserveScores) {
+ for (const std::string op : {"AND", "OR"}) {
+ SCOPED_TRACE(op);
+ NullSearchFixture fixture(6);
+ auto& body = fixture.add_field("body");
+ body.source->batches = true;
+ body.nulls = rows({0, 2, 4});
+ using Posting = index_query::testing::FakeIndexSource::Posting;
+ body.source->add("foo", std::vector<Posting> {{.doc = 0, .positions = {0, 1, 2}},
+ {.doc = 1, .positions = {0, 1}},
+ {.doc = 2, .positions = {0, 1, 2}},
+ {.doc = 3, .positions = {0}},
+ {.doc = 4, .positions = {0, 1, 2}}});
+ body.source->add("bar", std::vector<Posting> {{.doc = 0, .positions = {0, 1, 2}},
+ {.doc = 1, .positions = {0, 1}},
+ {.doc = 2, .positions = {0, 1, 2}},
+ {.doc = 4, .positions = {0, 1, 2}}});
+ const auto occurrence = op == "AND" ? TSearchOccur::MUST : TSearchOccur::SHOULD;
+ const auto reference =
+ compound("OCCUR_BOOLEAN", {occur(leaf("TERM", "body", "foo"), occurrence),
+ occur(leaf("TERM", "body", "bar"), occurrence)});
+ InvertedIndexResultBitmap reference_rows;
+ CollectionSimilarityPtr reference_scores;
+ ASSERT_TRUE(fixture.evaluate(reference, scored_if(true), &reference_rows, &reference_scores)
+ .ok());
+ const auto root = compound(op, {leaf("TERM", "body", "foo"), leaf("TERM", "body", "bar")});
+ InvertedIndexResultBitmap result;
+ CollectionSimilarityPtr scores;
+ ASSERT_TRUE(fixture.evaluate(root, scored_if(true), &result, &scores).ok());
+ EXPECT_EQ((op == "AND" ? rows({1}) : rows({1, 3})).toString(),
+ result.get_data_bitmap()->toString());
+ EXPECT_EQ(reference_rows.get_data_bitmap()->toString(),
+ result.get_data_bitmap()->toString());
+ ASSERT_EQ(reference_scores->_bm25_scores.size(), scores->_bm25_scores.size());
+ for (const auto& [row, score] : reference_scores->_bm25_scores) {
+ EXPECT_FLOAT_EQ(score, scores->_bm25_scores.at(row));
+ }
+ }
+}
+
+// Enabling WAND collection must preserve TopK rows and scores when NULL rows
+// have either no postings or higher-frequency postings than every valid row.
+TEST(FunctionSearchNullTest, NullableShouldTermsPreserveTopKRowsAndScores) {
+ for (const bool null_has_postings : {false, true}) {
+ SCOPED_TRACE(null_has_postings);
+ NullSearchFixture fixture(5);
+ auto& body = fixture.add_field("body");
+ body.nulls = rows({0});
+ using Posting = index_query::testing::FakeIndexSource::Posting;
+ std::vector<Posting> foo = {{.doc = 1, .positions = {0, 1, 2}},
+ {.doc = 3, .positions = {0}}};
+ std::vector<Posting> bar = {{.doc = 1, .positions = {0, 1, 2}},
+ {.doc = 4, .positions = {0}}};
+ if (null_has_postings) {
+ const Posting null_posting = {.doc = 0, .positions = {0, 1, 2, 3, 4, 5}};
+ foo.insert(foo.begin(), null_posting);
+ bar.insert(bar.begin(), null_posting);
+ }
+ body.source->add("foo", foo);
+ body.source->add("bar", bar);
+ const auto root = compound("OCCUR_BOOLEAN",
+ {occur(leaf("TERM", "body", "foo"), TSearchOccur::SHOULD),
+ occur(leaf("TERM", "body", "bar"), TSearchOccur::SHOULD)});
+ InvertedIndexResultBitmap full;
+ CollectionSimilarityPtr full_scores;
+ ASSERT_TRUE(fixture.evaluate(root, scored_if(true), &full, &full_scores).ok());
+ EXPECT_EQ(rows({1, 3, 4}).toString(), full.get_data_bitmap()->toString());
+ EvaluateOptions options = scored_if(true, 1);
+ options.wand = true;
+ InvertedIndexResultBitmap top;
+ CollectionSimilarityPtr top_scores;
+ ASSERT_TRUE(fixture.evaluate(root, options, &top, &top_scores).ok());
+ EXPECT_EQ(rows({1}).toString(), top.get_data_bitmap()->toString());
+ ASSERT_EQ(1U, top_scores->_bm25_scores.size());
+ EXPECT_FLOAT_EQ(full_scores->_bm25_scores.at(1), top_scores->_bm25_scores.at(1));
+ }
+}
+
+// Sharing a physical source does not imply sharing a logical field's NULL mask.
+TEST(FunctionSearchNullTest, SharedSourceKeepsDifferentNullMasksSeparate) {
+ for (const bool scoring : {false, true}) {
+ for (const std::string op : {"AND", "OR"}) {
+ SCOPED_TRACE(scoring);
+ SCOPED_TRACE(op);
+ NullSearchFixture fixture(3);
+ auto& first = fixture.add_field("first");
+ first.source->batches = true;
+ first.source->add("foo", {0, 1, 2});
+ first.source->add("bar", {0, 1, 2});
+ first.nulls = rows({0});
+ auto& second = fixture.add_field("second");
+ second.source = first.source;
+ second.nulls = rows({1});
+ fixture.expect(
+ compound(op, {leaf("TERM", "first", "foo"), leaf("TERM", "second", "bar")}),
+ scored_if(scoring), op == "AND" ? rows({2}) : rows({0, 1, 2}), rows({}));
+ }
+ }
+}
+
+// The shared mask retains SQL UNKNOWN by default. Only SEARCH opts into FALSE,
+// which changes negation while leaving the positive match set and input cache intact.
+TEST(FunctionSearchNullTest, NullMaskKeepsGenericSqlThreeValuedDefault) {
+ namespace query = segment_v2::inverted_index::query_v2;
+ auto postings = std::make_shared<roaring::Roaring>(rows({0, 1}));
+ auto nulls = std::make_shared<roaring::Roaring>(rows({1, 2}));
+ query::QueryExecutionContext context;
+ context.segment_num_rows = 3;
+ for (bool search : {false, true}) {
+ auto leaf = std::make_shared<query::BitSetQuery>(postings);
+ query::QueryPtr masked =
+ search ? std::make_shared<query::NullMaskedQuery>(leaf, nulls, true)
+ : std::make_shared<query::NullMaskedQuery>(leaf, nulls);
+ auto builder = query::create_operator_boolean_query_builder(query::OperatorType::OP_NOT);
+ builder->add(masked);
+ auto negated = builder->build();
+ for (bool scoring : {false, true}) {
+ auto scorer = negated->weight(scoring)->scorer(context);
+ roaring::Roaring matches;
+ for (auto doc = scorer->doc(); doc != query::TERMINATED; doc = scorer->advance()) {
+ matches.add(doc);
+ }
+ EXPECT_EQ((search ? rows({1, 2}) : rows({})).toString(), matches.toString());
+ EXPECT_EQ(!search, scorer->has_null_bitmap());
+ }
+ EXPECT_EQ(rows({0, 1}).toString(), postings->toString());
+ EXPECT_EQ(rows({1, 2}).toString(), nulls->toString());
+ }
+}
+
} // namespace doris
diff --git a/be/test/exprs/function/search/function_search_test.cpp b/be/test/exprs/function/search/function_search_test.cpp
index 6c02c4273e0..ee9cb764885 100644
--- a/be/test/exprs/function/search/function_search_test.cpp
+++ b/be/test/exprs/function/search/function_search_test.cpp
@@ -17,6 +17,25 @@
#include "exprs/function/search/function_search.h"
+#include <CLucene.h>
+// CLucene internal headers use pre-C++11 virtual declarations.
+#ifdef __clang__
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wshadow-field"
+#pragma clang diagnostic ignored "-Woverloaded-virtual"
+#pragma clang diagnostic ignored "-Winconsistent-missing-override"
+#pragma clang diagnostic ignored "-Wreorder-ctor"
+#pragma clang diagnostic ignored "-Wshorten-64-to-32"
+#elif defined(__GNUC__)
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Woverloaded-virtual"
+#endif
+#include <CLucene/index/_MultiSegmentReader.h>
+#ifdef __clang__
+#pragma clang diagnostic pop
+#elif defined(__GNUC__)
+#pragma GCC diagnostic pop
+#endif
#include <gen_cpp/Exprs_types.h>
#include <gtest/gtest.h>
@@ -44,15 +63,19 @@
#include "core/data_type/primitive_type.h"
#include "exprs/function/search/function_search_leaf.h"
#include "exprs/function/search/search_lowering.h"
+#include "io/fs/local_file_system.h"
#include "runtime/exec_env.h"
#include "runtime/index_policy/index_policy_mgr.h"
#include "runtime/runtime_state.h"
#include "storage/index/index_file_reader.h"
#include "storage/index/index_iterator.h"
#include "storage/index/inverted/analysis/analyzer/analyzer.h"
+#include "storage/index/inverted/analysis/analyzer/custom_analyzer.h"
#include "storage/index/inverted/analysis/inverted_index_parser.h"
#include "storage/index/inverted/clucene/clucene_index_reader.h"
+#include "storage/index/inverted/clucene/spi/clucene_index_source.h"
#include "storage/index/inverted/inverted_index_iterator.h"
+#include "storage/index/inverted/query/engine/bit_set_query/bit_set_query.h"
#include "storage/index/inverted/query/engine/collect/doc_set_collector.h"
#include "storage/index/inverted/query/engine/collect/top_k_collector.h"
#include "storage/index/inverted/query/engine/expand_query/expand_query.h"
@@ -184,17 +207,27 @@ public:
DORIS_CHECK(inverted_param != nullptr);
DORIS_CHECK(*inverted_param != nullptr);
DORIS_CHECK((*inverted_param)->roaring != nullptr);
- (*inverted_param)->roaring->add(3);
+ if (cached_result) {
+ (*inverted_param)->roaring = cached_result;
+ } else {
+ (*inverted_param)->roaring->add(3);
+ }
return Status::OK();
}
- Status read_null_bitmap(segment_v2::InvertedIndexQueryCacheHandle* /*cache_handle*/) override {
+ Status read_null_bitmap(segment_v2::InvertedIndexQueryCacheHandle* cache_handle) override {
+ if (cached_result) {
+ return InvertedIndexIterator::read_null_bitmap(cache_handle);
+ }
return Status::OK();
}
- Result<bool> has_null() override { return false; }
+ Result<bool> has_null() override {
+ return cached_result ? InvertedIndexIterator::has_null() : Result<bool>(false);
+ }
int read_calls = 0;
+ std::shared_ptr<roaring::Roaring> cached_result;
};
class DummyInvertedIndexReader final : public segment_v2::InvertedIndexReader {
@@ -552,19 +585,14 @@ static TSearchClause make_leaf_clause(const std::string& clause_type, const std:
static Status insert_search_dsl_cache(
segment_v2::InvertedIndexQueryCache* cache,
const std::shared_ptr<segment_v2::IndexFileReader>& index_file_reader,
- const TSearchParam& search_param, roaring::Roaring bitmap) {
+ const TSearchParam& search_param, roaring::Roaring bitmap,
+ const std::string& cache_namespace = "__search_dsl_2vl__") {
std::string signature = FunctionSearch::dsl_cache_signature(search_param, nullptr);
if (signature.empty()) {
return Status::InternalError("the search param cannot be serialized");
}
- // A cached result is its rows and its UNKNOWN rows, here none.
- segment_v2::InvertedIndexQueryCache::CacheKey null_key {
- index_file_reader->get_index_path_prefix(), "__search_dsl__",
- segment_v2::InvertedIndexQueryType::SEARCH_DSL_QUERY, signature + "__null"};
- segment_v2::InvertedIndexQueryCacheHandle null_handle;
- cache->insert(null_key, std::make_shared<roaring::Roaring>(), &null_handle);
segment_v2::InvertedIndexQueryCache::CacheKey key {
- index_file_reader->get_index_path_prefix(), "__search_dsl__",
+ index_file_reader->get_index_path_prefix(), cache_namespace,
segment_v2::InvertedIndexQueryType::SEARCH_DSL_QUERY, std::move(signature)};
segment_v2::InvertedIndexQueryCacheHandle handle;
cache->insert(key, std::make_shared<roaring::Roaring>(std::move(bitmap)), &handle);
@@ -773,33 +801,14 @@ TEST_F(FunctionSearchTest, TestEvaluateInvertedIndexBasic) {
}
TEST_F(FunctionSearchTest, TestEvaluateInvertedIndexWithSearchParamEmptyInputs) {
- // Test evaluate_inverted_index_with_search_param with empty inputs
- TSearchParam search_param;
- search_param.original_dsl = "title:hello";
-
- TSearchClause rootClause;
- rootClause.clause_type = "TERM";
- rootClause.field_name = "title";
- rootClause.value = "hello";
- search_param.root = rootClause;
-
- std::unordered_map<std::string, IndexFieldNameAndTypePair> empty_data_types;
- std::unordered_map<std::string, IndexIterator*> empty_iterators;
- uint32_t num_rows = 100;
- InvertedIndexResultBitmap bitmap_result;
-
- // Test with empty iterators
- auto status = function_search->evaluate_inverted_index_with_search_param(
- search_param, empty_data_types, empty_iterators, num_rows, bitmap_result);
- EXPECT_TRUE(status.ok()); // Should return OK but with empty result
-
- // Test with empty data types but non-empty iterators - should still return OK
- // because empty data_types will cause early return
- std::unordered_map<std::string, IndexIterator*> non_empty_iterators;
- non_empty_iterators["title"] = nullptr; // Add null iterator
- status = function_search->evaluate_inverted_index_with_search_param(
- search_param, empty_data_types, non_empty_iterators, num_rows, bitmap_result);
- EXPECT_TRUE(status.ok()); // Should return OK due to empty data_types check
+ TSearchParam param;
+ param.original_dsl = "title:hello";
+ param.root = make_leaf_clause("TERM", "hello");
+ param.root.field_name = "title";
+ InvertedIndexResultBitmap result;
+ auto status =
+ function_search->evaluate_inverted_index_with_search_param(param, {}, {}, 100, result);
+ EXPECT_EQ(ErrorCode::INVERTED_INDEX_FILE_NOT_FOUND, status.code());
}
TEST_F(FunctionSearchTest, TestEmptySearchParam) {
@@ -815,7 +824,7 @@ TEST_F(FunctionSearchTest, TestEmptySearchParam) {
auto status = function_search->evaluate_inverted_index_with_search_param(
empty_param, data_types, iterators, num_rows, bitmap_result);
- EXPECT_TRUE(status.ok()); // Should handle gracefully
+ EXPECT_EQ(ErrorCode::INVALID_ARGUMENT, status.code());
}
TEST_F(FunctionSearchTest, TestNullIterators) {
@@ -889,9 +898,15 @@ TEST_F(FunctionSearchTest, TestZeroRowsScenario) {
TSearchClause rootClause;
rootClause.clause_type = "TERM";
- rootClause.field_name = "title";
+ rootClause.__isset.field_name = true;
+ rootClause.__isset.value = true;
+ rootClause.field_name = "v.missing";
rootClause.value = "hello";
search_param.root = rootClause;
+ TSearchFieldBinding binding;
+ binding.field_name = rootClause.field_name;
+ binding.__set_is_variant_subcolumn(true);
+ search_param.field_bindings = {binding};
// Empty data types and iterators - this is a realistic zero-data scenario
std::unordered_map<std::string, IndexFieldNameAndTypePair> data_types;
@@ -902,7 +917,9 @@ TEST_F(FunctionSearchTest, TestZeroRowsScenario) {
auto status = function_search->evaluate_inverted_index_with_search_param(
search_param, data_types, iterators, num_rows, bitmap_result);
- EXPECT_TRUE(status.ok()); // Should handle zero data gracefully and return empty result
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_TRUE(bitmap_result.get_data_bitmap()->isEmpty());
+ EXPECT_TRUE(bitmap_result.get_null_bitmap()->isEmpty());
}
TEST_F(FunctionSearchTest, TestVeryLargeRowCount) {
@@ -912,9 +929,15 @@ TEST_F(FunctionSearchTest, TestVeryLargeRowCount) {
TSearchClause rootClause;
rootClause.clause_type = "TERM";
- rootClause.field_name = "title";
+ rootClause.__isset.field_name = true;
+ rootClause.__isset.value = true;
+ rootClause.field_name = "v.missing";
rootClause.value = "hello";
search_param.root = rootClause;
+ TSearchFieldBinding binding;
+ binding.field_name = rootClause.field_name;
+ binding.__set_is_variant_subcolumn(true);
+ search_param.field_bindings = {binding};
// Empty data types and iterators - this tests the large row count parameter handling
std::unordered_map<std::string, IndexFieldNameAndTypePair> data_types;
@@ -925,7 +948,9 @@ TEST_F(FunctionSearchTest, TestVeryLargeRowCount) {
auto status = function_search->evaluate_inverted_index_with_search_param(
search_param, data_types, iterators, num_rows, bitmap_result);
- EXPECT_TRUE(status.ok()); // Should handle large row counts gracefully and return empty result
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_TRUE(bitmap_result.get_data_bitmap()->isEmpty());
+ EXPECT_TRUE(bitmap_result.get_null_bitmap()->isEmpty());
}
// Tests for FieldReaderResolver::resolve function coverage (lines 74+)
@@ -1453,6 +1478,8 @@ TEST_F(FunctionSearchTest, TestBuildLeafQueryPhrase) {
data_type_with_names.emplace("content", IndexFieldNameAndTypePair {"content", nullptr});
std::unordered_map<std::string, IndexIterator*> iterators;
+ RecordingIndexIterator physical_iterator;
+ iterators["content"] = &physical_iterator;
FieldReaderResolver resolver(data_type_with_names, iterators, context);
FieldReaderBinding binding;
@@ -1518,6 +1545,8 @@ TEST_F(FunctionSearchTest, TestBuildLeafQueryPhraseUsesPlainTerms) {
std::unordered_map<std::string, IndexFieldNameAndTypePair> data_type_with_names;
data_type_with_names.emplace("content", IndexFieldNameAndTypePair {"content", nullptr});
std::unordered_map<std::string, IndexIterator*> iterators;
+ RecordingIndexIterator physical_iterator;
+ iterators["content"] = &physical_iterator;
FieldReaderResolver resolver(data_type_with_names, iterators, context);
FieldReaderBinding binding;
@@ -1552,7 +1581,7 @@ TEST_F(FunctionSearchTest, TestBuildLeafQueryPhraseUsesPlainTerms) {
policy_mgr->apply_policy_changes({}, {tokenizer.id, analyzer.id});
}
-TEST_F(FunctionSearchTest, TestBuildLeafQueryVariantMissingFieldReturnsUnknown) {
+TEST_F(FunctionSearchTest, TestBuildLeafQueryVariantMissingFieldDoesNotMatch) {
TSearchClause clause;
clause.clause_type = "TERM";
clause.field_name = "var.items.missing";
@@ -1597,10 +1626,7 @@ TEST_F(FunctionSearchTest, TestBuildLeafQueryVariantMissingFieldReturnsUnknown)
auto scorer = weight->scorer(exec_ctx);
ASSERT_NE(scorer, nullptr);
EXPECT_EQ(inverted_index::query_v2::TERMINATED, scorer->doc());
- ASSERT_TRUE(scorer->has_null_bitmap());
- const auto* null_bitmap = scorer->get_null_bitmap();
- ASSERT_NE(null_bitmap, nullptr);
- EXPECT_EQ(5u, null_bitmap->cardinality());
+ EXPECT_FALSE(scorer->has_null_bitmap());
}
TEST_F(FunctionSearchTest, TestFieldReaderResolverVariantSubcolumnWithMissingIterator) {
@@ -1622,7 +1648,7 @@ TEST_F(FunctionSearchTest, TestFieldReaderResolverVariantSubcolumnWithMissingIte
auto status =
resolver.resolve("var.items.level", InvertedIndexQueryType::EQUAL_QUERY, &binding);
- ASSERT_TRUE(status.ok());
+ EXPECT_EQ(ErrorCode::INVERTED_INDEX_FILE_NOT_FOUND, status.code());
EXPECT_FALSE(binding.is_bound());
EXPECT_TRUE(resolver.binding_cache().empty());
}
@@ -1900,7 +1926,7 @@ TEST_F(FunctionSearchTest, TestBuildLeafQueryExecutesSelectedSniiWildcardReader)
auto weight = query->weight(true);
ASSERT_NE(nullptr, weight);
- VariantSearchNullBitmapAdapter nulls(resolver);
+ SearchNullBitmapResolver nulls;
auto scorer = weight->scorer(build_variant_search_query_execution_context(4, resolver, &nulls),
binding_key);
ASSERT_NE(nullptr, scorer);
@@ -1914,15 +1940,13 @@ TEST_F(FunctionSearchTest, TestBuildLeafQueryExecutesSelectedSniiWildcardReader)
EXPECT_TRUE(decoy_reader->fake_source->expanded.empty());
// The NULL rows are the column's, so either index answers them, once.
EXPECT_EQ(1, decoy_reader->null_bitmap_calls + selected_reader->null_bitmap_calls);
- ASSERT_TRUE(scorer->has_null_bitmap());
- const auto* null_bitmap = scorer->get_null_bitmap();
- ASSERT_NE(nullptr, null_bitmap);
- expect_bitmap_eq(*null_bitmap, {3});
+ EXPECT_FALSE(scorer->has_null_bitmap());
+ EXPECT_EQ(nullptr, scorer->get_null_bitmap());
scoped_policy_mgr.apply_policy_changes({}, {analyzer.id});
}
-TEST_F(FunctionSearchTest, TestSniiWildcardPreservesThreeValuedBooleanAndFieldExists) {
+TEST_F(FunctionSearchTest, TestSniiWildcardUsesDocumentBooleanAndFieldExists) {
auto context = std::make_shared<IndexQueryContext>();
std::map<std::string, std::string> properties {
{INVERTED_INDEX_PARSER_KEY, INVERTED_INDEX_PARSER_STANDARD}};
@@ -1960,10 +1984,9 @@ TEST_F(FunctionSearchTest, TestSniiWildcardPreservesThreeValuedBooleanAndFieldEx
not_clause.__isset.children = true;
auto exists_clause = make_leaf_clause("WILDCARD", "*");
- VariantSearchNullBitmapAdapter nulls(resolver);
+ SearchNullBitmapResolver nulls;
auto verify_result = [&](const TSearchClause& root,
- std::initializer_list<uint32_t> expected_docs,
- std::initializer_list<uint32_t> expected_nulls) {
+ std::initializer_list<uint32_t> expected_docs) {
inverted_index::query_v2::QueryPtr query;
std::string binding_key;
auto status = function_search->build_query_recursive(root, context, resolver, &query,
@@ -1976,15 +1999,12 @@ TEST_F(FunctionSearchTest, TestSniiWildcardPreservesThreeValuedBooleanAndFieldEx
build_variant_search_query_execution_context(4, resolver, &nulls), binding_key);
ASSERT_NE(nullptr, scorer);
expect_bitmap_eq(collect_docs(scorer), expected_docs);
- ASSERT_TRUE(scorer->has_null_bitmap());
- const auto* null_bitmap = scorer->get_null_bitmap();
- ASSERT_NE(nullptr, null_bitmap);
- expect_bitmap_eq(*null_bitmap, expected_nulls);
+ EXPECT_FALSE(scorer->has_null_bitmap());
};
- verify_result(or_clause, {0, 1}, {3});
- verify_result(not_clause, {1, 2}, {3});
- verify_result(exists_clause, {0, 1, 2}, {3});
+ verify_result(or_clause, {0, 1});
+ verify_result(not_clause, {1, 2, 3});
+ verify_result(exists_clause, {0, 1, 2});
EXPECT_EQ(0, reader->query_calls);
}
@@ -2031,7 +2051,7 @@ TEST_F(FunctionSearchTest, TestSniiNativeTermOnAnalyzedFieldIsAMatchAnyQuery) {
auto weight = query->weight(false);
ASSERT_NE(nullptr, weight);
- VariantSearchNullBitmapAdapter nulls(resolver);
+ SearchNullBitmapResolver nulls;
auto scorer = weight->scorer(build_variant_search_query_execution_context(4, resolver, &nulls),
binding_key);
ASSERT_NE(nullptr, scorer);
@@ -2072,7 +2092,7 @@ TEST_F(FunctionSearchTest, TestSniiNativeTermOnKeywordFieldIsAnEqualQuery) {
reader->fake_source->prepared);
auto weight = query->weight(false);
ASSERT_NE(nullptr, weight);
- VariantSearchNullBitmapAdapter nulls(resolver);
+ SearchNullBitmapResolver nulls;
auto scorer = weight->scorer(build_variant_search_query_execution_context(4, resolver, &nulls),
binding_key);
ASSERT_NE(nullptr, scorer);
@@ -2187,7 +2207,7 @@ TEST_F(FunctionSearchTest, TestSniiNativeTermDefaultOperatorAndMapsToMatchAllQue
auto weight = query->weight(false);
ASSERT_NE(nullptr, weight);
- VariantSearchNullBitmapAdapter nulls(resolver);
+ SearchNullBitmapResolver nulls;
auto scorer = weight->scorer(build_variant_search_query_execution_context(4, resolver, &nulls),
binding_key);
ASSERT_NE(nullptr, scorer);
@@ -2232,7 +2252,7 @@ TEST_F(FunctionSearchTest, TestSniiNativeTermSingleTokenAllowsMinimumShouldMatch
auto weight = query->weight(false);
ASSERT_NE(nullptr, weight);
- VariantSearchNullBitmapAdapter nulls(resolver);
+ SearchNullBitmapResolver nulls;
auto scorer = weight->scorer(build_variant_search_query_execution_context(4, resolver, &nulls),
binding_key);
ASSERT_NE(nullptr, scorer);
@@ -2274,7 +2294,7 @@ TEST_F(FunctionSearchTest, TestSniiNativeTermZeroTokenMinimumShouldMatchReturnsE
auto weight = query->weight(false);
ASSERT_NE(nullptr, weight);
- VariantSearchNullBitmapAdapter nulls(resolver);
+ SearchNullBitmapResolver nulls;
auto scorer = weight->scorer(build_variant_search_query_execution_context(4, resolver, &nulls),
binding_key);
ASSERT_NE(nullptr, scorer);
@@ -2319,7 +2339,7 @@ TEST_F(FunctionSearchTest, TestSniiNativeKeywordPrefixIsALiteralStem) {
auto weight = query->weight(false);
ASSERT_NE(nullptr, weight);
- VariantSearchNullBitmapAdapter nulls(resolver);
+ SearchNullBitmapResolver nulls;
auto scorer = weight->scorer(build_variant_search_query_execution_context(4, resolver, &nulls),
binding_key);
ASSERT_NE(nullptr, scorer);
@@ -2400,7 +2420,7 @@ TEST_F(FunctionSearchTest, TestSniiNativeRegexpMatchesWholeTerms) {
// SEARCH's REGEXP matches whole terms, so "alphabet" stays out.
auto weight = query->weight(false);
ASSERT_NE(nullptr, weight);
- VariantSearchNullBitmapAdapter nulls(resolver);
+ SearchNullBitmapResolver nulls;
auto scorer = weight->scorer(build_variant_search_query_execution_context(4, resolver, &nulls),
binding_key);
ASSERT_NE(nullptr, scorer);
@@ -2469,7 +2489,7 @@ TEST_F(FunctionSearchTest, TestSniiNativeCustomKeywordPrefixStripsTheDslSuffix)
auto weight = query->weight(false);
ASSERT_NE(nullptr, weight);
- VariantSearchNullBitmapAdapter nulls(resolver);
+ SearchNullBitmapResolver nulls;
auto scorer = weight->scorer(build_variant_search_query_execution_context(4, resolver, &nulls),
binding_key);
ASSERT_NE(nullptr, scorer);
@@ -2583,6 +2603,8 @@ TEST_F(FunctionSearchTest, TestBuildLeafQueryClucenePrefixOfAStopwordStaysAPrefi
std::unordered_map<std::string, IndexFieldNameAndTypePair> data_type_with_names;
data_type_with_names.emplace("body", IndexFieldNameAndTypePair {"body", nullptr});
std::unordered_map<std::string, IndexIterator*> iterators;
+ RecordingIndexIterator physical_iterator;
+ iterators["body"] = &physical_iterator;
FieldReaderResolver resolver(data_type_with_names, iterators, context);
FieldReaderBinding binding;
@@ -3192,7 +3214,7 @@ TEST_F(FunctionSearchTest, TestSearchDslCacheRemainsEnabledForUnreferencedSniiFi
EXPECT_EQ(0, number_iterator.read_calls);
}
-TEST_F(FunctionSearchTest, TestBuildLeafQueryDirectUnknownClauseUsesLeafMapper) {
+TEST_F(FunctionSearchTest, TestBuildLeafQueryRejectsUnsupportedDirectClause) {
TSearchClause clause;
clause.clause_type = "PHRASE";
clause.field_name = "var.items.active";
@@ -3245,23 +3267,9 @@ TEST_F(FunctionSearchTest, TestBuildLeafQueryDirectUnknownClauseUsesLeafMapper)
std::string out_binding_key;
Status st = function_search->build_query_recursive(clause, context, resolver, &out,
&out_binding_key, "OR", 0, 4);
- ASSERT_TRUE(st.ok());
- ASSERT_NE(out, nullptr);
- EXPECT_TRUE(mapper_called);
- EXPECT_EQ(key, out_binding_key);
- EXPECT_TRUE(iterator.last_column_name.empty());
-
- auto weight = out->weight(false);
- ASSERT_NE(weight, nullptr);
- inverted_index::query_v2::QueryExecutionContext exec_ctx;
- exec_ctx.segment_num_rows = 4;
- auto scorer = weight->scorer(exec_ctx);
- ASSERT_NE(scorer, nullptr);
- EXPECT_EQ(inverted_index::query_v2::TERMINATED, scorer->doc());
- ASSERT_TRUE(scorer->has_null_bitmap());
- const auto* null_bitmap = scorer->get_null_bitmap();
- ASSERT_NE(null_bitmap, nullptr);
- EXPECT_EQ(4u, null_bitmap->cardinality());
+ EXPECT_FALSE(st.ok());
+ EXPECT_FALSE(mapper_called);
+ EXPECT_EQ(nullptr, out);
}
TEST_F(FunctionSearchTest, TestBuildLeafQueryVariantBoolUsesDirectIndexReader) {
@@ -4046,4 +4054,452 @@ TEST_F(FunctionSearchTest, TestSniiNativeWildcardNormalizesOnlyItsLiterals) {
scoped_policy_mgr.apply_policy_changes({}, {char_filter.id, analyzer.id});
}
+TEST_F(FunctionSearchTest, DocumentBooleanContractPreservesScoringMatchSets) {
+ auto index_meta = make_test_inverted_index(
+ 101, {{INVERTED_INDEX_PARSER_KEY, INVERTED_INDEX_PARSER_NONE}});
+ auto file_reader = std::make_shared<RejectingCluceneIndexFileReader>();
+ std::vector<std::unique_ptr<InvertedIndexIterator>> owned_iterators;
+ std::unordered_map<std::string, IndexFieldNameAndTypePair> types;
+ std::unordered_map<std::string, IndexIterator*> iterators;
+ const std::vector<roaring::Roaring> hits = {make_bitmap({0, 1, 2}), make_bitmap({0, 3, 6}),
+ make_bitmap({2, 7})};
+ const std::vector<roaring::Roaring> nulls = {make_bitmap({6, 7, 8}), make_bitmap({2, 5, 8}),
+ make_bitmap({0, 5, 8})};
+ for (size_t i = 0; i < hits.size(); ++i) {
+ const std::string field(1, static_cast<char>('a' + i));
+ auto reader =
+ std::make_shared<RecordingNativeInvertedIndexReader>(&index_meta, file_reader);
+ std::vector<uint32_t> hit_rows(hits[i].begin(), hits[i].end());
+ reader->fake_source->add("hit", hit_rows);
+ reader->set_null_bitmap(nulls[i]);
+ auto iterator = std::make_unique<InvertedIndexIterator>();
+ iterator->add_reader(InvertedIndexReaderType::FULLTEXT, reader);
+ types.emplace(field, IndexFieldNameAndTypePair {field, std::make_shared<DataTypeString>()});
+ iterators[field] = iterator.get();
+ owned_iterators.emplace_back(std::move(iterator));
+ }
+ auto leaf = [](const std::string& field) {
+ auto clause = make_leaf_clause("TERM", "hit");
+ clause.field_name = field;
+ return clause;
+ };
+ auto boolean = [](const std::string& op, const std::vector<TSearchClause>& children) {
+ TSearchClause clause;
+ clause.clause_type = op;
+ clause.__set_children(children);
+ return clause;
+ };
+ const auto a = leaf("a");
+ const auto b = leaf("b");
+ const auto c = leaf("c");
+ const auto conjunction = boolean("AND", {a, b});
+ const auto disjunction = boolean("OR", {a, b});
+ auto must = a;
+ must.__set_occur(TSearchOccur::MUST);
+ auto not_b = b;
+ not_b.__set_occur(TSearchOccur::MUST_NOT);
+ auto not_c = c;
+ not_c.__set_occur(TSearchOccur::MUST_NOT);
+ const auto exclusions = boolean("OCCUR_BOOLEAN", {must, not_b, not_c});
+ const std::vector<std::pair<TSearchClause, roaring::Roaring>> cases = {
+ {a, make_bitmap({0, 1, 2})},
+ {conjunction, make_bitmap({0})},
+ {disjunction, make_bitmap({0, 1, 2, 3, 6})},
+ {boolean("NOT", {conjunction}), make_bitmap({1, 2, 3, 4, 5, 6, 7, 8})},
+ {boolean("NOT", {disjunction}), make_bitmap({4, 5, 7, 8})},
+ {boolean("NOT", {a}), make_bitmap({3, 4, 5, 6, 7, 8})},
+ {boolean("AND", {a, boolean("NOT", {b})}), make_bitmap({1, 2})},
+ {exclusions, make_bitmap({1})},
+ {boolean("NOT", {exclusions}), make_bitmap({0, 2, 3, 4, 5, 6, 7, 8})},
+ {boolean("OR", {a, boolean("NOT", {a})}), make_bitmap({0, 1, 2, 3, 4, 5, 6, 7, 8})},
+ {boolean("AND", {a, boolean("NOT", {a})}), make_bitmap({})},
+ {boolean("NOT", {boolean("NOT", {a})}), make_bitmap({0, 1, 2})}};
+ for (bool scoring : {false, true}) {
+ for (const auto& [root, expected] : cases) {
+ TSearchParam param;
+ param.original_dsl = "document Boolean contract";
+ param.root = root;
+ auto context = std::make_shared<IndexQueryContext>();
+ if (scoring) {
+ context->collection_similarity = std::make_shared<CollectionSimilarity>();
+ context->collection_statistics = std::make_shared<FixedSearchStatistics>();
+ }
+ InvertedIndexResultBitmap result;
+ ASSERT_TRUE(function_search
+ ->evaluate_inverted_index_with_search_param(param, types, iterators,
+ 9, result, false,
+ nullptr, {}, context)
+ .ok());
+ ASSERT_NE(nullptr, result.get_data_bitmap());
+ EXPECT_TRUE(*result.get_data_bitmap() == expected);
+ ASSERT_NE(nullptr, result.get_null_bitmap());
+ EXPECT_TRUE(result.get_null_bitmap()->isEmpty());
+ }
+ }
+}
+
+TEST_F(FunctionSearchTest, MissingFieldsStillEvaluateTheCompleteBooleanTree) {
+ auto missing = make_leaf_clause("TERM", "hit");
+ missing.field_name = "v.absent";
+ TSearchFieldBinding binding;
+ binding.field_name = missing.field_name;
+ binding.__set_is_variant_subcolumn(true);
+ // The production VSearchExpr collection boundary verifies storage absence
+ // before omitting the field from the resolver's metadata map.
+ TSearchClause negated;
+ negated.clause_type = "NOT";
+ negated.__set_children({missing});
+ TSearchClause twice;
+ twice.clause_type = "NOT";
+ twice.__set_children({negated});
+ auto exists = missing;
+ exists.clause_type = "WILDCARD";
+ exists.value = "*";
+ auto not_exists = negated;
+ not_exists.__set_children({exists});
+ for (const auto& [root, expected] : std::vector<std::pair<TSearchClause, roaring::Roaring>> {
+ {missing, make_bitmap({})},
+ {negated, make_bitmap({0, 1, 2})},
+ {twice, make_bitmap({})},
+ {exists, make_bitmap({})},
+ {not_exists, make_bitmap({0, 1, 2})}}) {
+ TSearchParam param;
+ param.original_dsl = "missing field Boolean contract";
+ param.root = root;
+ param.field_bindings = {binding};
+ for (bool scoring : {false, true}) {
+ auto context = std::make_shared<IndexQueryContext>();
+ if (scoring) {
+ context->collection_similarity = std::make_shared<CollectionSimilarity>();
+ }
+ InvertedIndexResultBitmap result;
+ auto status = function_search->evaluate_inverted_index_with_search_param(
+ param, {}, {}, 3, result, false, nullptr, {}, context);
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_TRUE(*result.get_data_bitmap() == expected);
+ EXPECT_TRUE(result.get_null_bitmap()->isEmpty());
+ }
+ }
+}
+
+// A numeric existence clause uses the physical NULL bitmap, not a term lookup for "*".
+// Supplying the backend clause directly keeps this test independent of FE parser changes.
+TEST_F(FunctionSearchTest, NumericExistenceUsesBackendClausesIndependentlyOfFrontend) {
+ auto index_meta = make_test_inverted_index(105);
+ auto file_reader = std::make_shared<RejectingCluceneIndexFileReader>();
+ auto reader = std::make_shared<RecordingNativeInvertedIndexReader>(
+ &index_meta, file_reader, InvertedIndexReaderType::BKD);
+ reader->set_null_bitmap(make_bitmap({0, 3}));
+ InvertedIndexIterator iterator;
+ iterator.add_reader(InvertedIndexReaderType::BKD, reader);
+ auto context = std::make_shared<IndexQueryContext>();
+ std::unordered_map<std::string, IndexFieldNameAndTypePair> types {
+ {"age", {"age", std::make_shared<DataTypeInt32>()}}};
+ std::unordered_map<std::string, IndexIterator*> iterators {{"age", &iterator}};
+ FieldReaderResolver resolver(types, iterators, context);
+ auto exists = make_leaf_clause("WILDCARD", "*");
+ exists.field_name = "age";
+ TSearchClause not_exists;
+ not_exists.clause_type = "NOT";
+ not_exists.__set_children({exists});
+ SearchNullBitmapResolver null_resolver;
+ for (const bool scoring : {false, true}) {
+ for (const auto& [clause, expected] :
+ std::vector<std::pair<TSearchClause, roaring::Roaring>> {
+ {exists, make_bitmap({1, 2})}, {not_exists, make_bitmap({0, 3})}}) {
+ inverted_index::query_v2::QueryPtr query;
+ std::string binding_key;
+ ASSERT_TRUE(function_search
+ ->build_query_recursive(clause, context, resolver, &query,
+ &binding_key, "or", -1, 4)
+ .ok());
+ auto scorer = query->weight(scoring)->scorer(
+ build_variant_search_query_execution_context(4, resolver, &null_resolver),
+ binding_key);
+ EXPECT_EQ(expected, collect_docs(scorer));
+ EXPECT_FALSE(scorer->has_null_bitmap());
+ }
+ }
+ EXPECT_EQ(0, reader->raw_query_calls);
+}
+
+TEST_F(FunctionSearchTest, DirectSearchOwnsTheReaderCachedMatchBitmap) {
+ auto index_meta = make_test_inverted_index(104);
+ auto file_reader = std::make_shared<RejectingCluceneIndexFileReader>();
+ auto reader = std::make_shared<RecordingNativeInvertedIndexReader>(
+ &index_meta, file_reader, InvertedIndexReaderType::BKD);
+ reader->set_null_bitmap(make_bitmap({1}));
+ RecordingDirectInvertedIndexIterator iterator;
+ iterator.add_reader(InvertedIndexReaderType::BKD, reader);
+ // Model the real BKD reader's cache alias, including a NULL payload hit.
+ // The leaf must remove it from its own result without modifying the cache.
+ iterator.cached_result = std::make_shared<roaring::Roaring>(make_bitmap({0, 1, 3}));
+ auto context = std::make_shared<IndexQueryContext>();
+ std::unordered_map<std::string, IndexFieldNameAndTypePair> types {
+ {"age", {"age", std::make_shared<DataTypeInt32>()}}};
+ std::unordered_map<std::string, IndexIterator*> iterators {{"age", &iterator}};
+ FieldReaderResolver resolver(types, iterators, context);
+ auto clause = make_leaf_clause("TERM", "42");
+ clause.field_name = "age";
+ for (int repetition = 0; repetition < 2; ++repetition) {
+ inverted_index::query_v2::QueryPtr query;
+ std::string binding_key;
+ ASSERT_TRUE(function_search
+ ->build_query_recursive(clause, context, resolver, &query, &binding_key,
+ "OR", -1, 4)
+ .ok());
+ auto scorer = query->weight(false)->scorer(
+ build_variant_search_query_execution_context(4, resolver, nullptr), binding_key);
+ expect_bitmap_eq(collect_docs(scorer), {0, 3});
+ EXPECT_FALSE(scorer->has_null_bitmap());
+ expect_bitmap_eq(*iterator.cached_result, {0, 1, 3});
+ }
+ EXPECT_EQ(2, iterator.read_calls);
+}
+
+TEST_F(FunctionSearchTest, SearchCacheVersionsAndOwnsTheCompleteMatchSet) {
+ ScopedInvertedIndexQueryCache cache_guard;
+ auto text_meta = make_test_inverted_index(
+ 102, {{INVERTED_INDEX_PARSER_KEY, INVERTED_INDEX_PARSER_NONE}});
+ auto number_meta = make_test_inverted_index(103);
+ auto file_reader = std::make_shared<RejectingCluceneIndexFileReader>(
+ InvertedIndexStorageFormatPB::V2, "/tmp/search_2vl_cache_idx");
+ auto text_reader = std::make_shared<DummyInvertedIndexReader>(
+ &text_meta, file_reader, InvertedIndexReaderType::STRING_TYPE);
+ auto number_reader = std::make_shared<RecordingNativeInvertedIndexReader>(
+ &number_meta, file_reader, InvertedIndexReaderType::BKD);
+ InvertedIndexIterator text_iterator;
+ text_iterator.add_reader(InvertedIndexReaderType::STRING_TYPE, text_reader);
+ RecordingDirectInvertedIndexIterator number_iterator;
+ number_iterator.add_reader(InvertedIndexReaderType::BKD, number_reader);
+ std::unordered_map<std::string, IndexFieldNameAndTypePair> types {
+ {"unused", {"unused", std::make_shared<DataTypeString>()}},
+ {"age", {"age", std::make_shared<DataTypeInt32>()}}};
+ std::unordered_map<std::string, IndexIterator*> iterators {{"unused", &text_iterator},
+ {"age", &number_iterator}};
+ TSearchParam param;
+ param.original_dsl = "age:42";
+ param.root = make_leaf_clause("TERM", "42");
+ param.root.field_name = "age";
+ // Seed the old namespace with a deliberately wrong result. It cannot be
+ // reused under the document Boolean contract, with or without its NULL key.
+ ASSERT_TRUE(insert_search_dsl_cache(cache_guard.get(), file_reader, param, make_bitmap({1}),
+ "__search_dsl__")
+ .ok());
+ OlapReaderStatistics stats;
+ auto context = std::make_shared<IndexQueryContext>();
+ context->stats = &stats;
+ InvertedIndexResultBitmap cold;
+ ASSERT_TRUE(function_search
+ ->evaluate_inverted_index_with_search_param(
+ param, types, iterators, 4, cold, true, nullptr, {}, context)
+ .ok());
+ expect_bitmap_eq(*cold.get_data_bitmap(), {3});
+ EXPECT_TRUE(cold.get_null_bitmap()->isEmpty());
+ EXPECT_EQ(1, number_iterator.read_calls);
+ // Mutating the caller's bitmap must not alter the cached entry.
+ *cold.get_data_bitmap() = roaring::Roaring();
+ for (int i = 0; i < 2; ++i) {
+ InvertedIndexResultBitmap warm;
+ ASSERT_TRUE(function_search
+ ->evaluate_inverted_index_with_search_param(
+ param, types, iterators, 4, warm, true, nullptr, {}, context)
+ .ok());
+ expect_bitmap_eq(*warm.get_data_bitmap(), {3});
+ EXPECT_TRUE(warm.get_null_bitmap()->isEmpty());
+ *warm.get_data_bitmap() = roaring::Roaring();
+ }
+ EXPECT_EQ(1, number_iterator.read_calls);
+ EXPECT_EQ(2, stats.inverted_index_query_cache_hit);
+ EXPECT_EQ(1, stats.inverted_index_query_cache_miss);
+}
+
+// Keep the NULL-layout / score / collector matrix together against one physical
+// reader so every execution path uses the same independently specified oracle.
+// NOLINTNEXTLINE(readability-function-cognitive-complexity)
+TEST_F(FunctionSearchTest, SearchExistsUsesLocalDocIdsForEachCluceneSubReader) {
+ using namespace inverted_index::query_v2;
+ const std::string path = "./ut_dir/search_exists_multi_segment";
+ ASSERT_TRUE(io::global_local_filesystem()->delete_directory(path).ok());
+ ASSERT_TRUE(io::global_local_filesystem()->create_directory(path).ok());
+ DEFER(EXPECT_TRUE(io::global_local_filesystem()->delete_directory(path).ok()));
+ inverted_index::CustomAnalyzerConfig::Builder analyzer_builder;
+ analyzer_builder.with_tokenizer_config("standard", {});
+ auto analyzer = inverted_index::CustomAnalyzer::build_custom_analyzer(analyzer_builder.build());
+ auto* writer = _CLNEW lucene::index::IndexWriter(path.c_str(), analyzer.get(), true);
+ writer->setMaxBufferedDocs(2);
+ writer->setRAMBufferSizeMB(-1);
+ writer->setMergeFactor(1000000000);
+ writer->setUseCompoundFile(false);
+ auto string_reader = std::make_shared<lucene::util::SStringReader<char>>();
+ auto* document = _CLNEW lucene::document::Document();
+ auto* field = _CLNEW lucene::document::Field(
+ L"a", int32_t(lucene::document::Field::STORE_NO) |
+ int32_t(lucene::document::Field::INDEX_NONORMS) |
+ int32_t(lucene::document::Field::INDEX_TOKENIZED));
+ document->add(*field);
+ for (const std::string value : {"hit", "hit", "miss", "hit"}) {
+ string_reader->init(value.data(), value.size(), false);
+ field->setValue(analyzer->reusableTokenStream(field->name(), string_reader));
+ writer->addDocument(document);
+ }
+ writer->close();
+ _CLDELETE(writer);
+ _CLDELETE(document);
+ auto reader = std::shared_ptr<lucene::index::IndexReader>(
+ lucene::index::IndexReader::open(path.c_str()), [](lucene::index::IndexReader* ptr) {
+ ptr->close();
+ _CLDELETE(ptr);
+ });
+ ASSERT_EQ(4, reader->maxDoc());
+ const auto* multi_reader = dynamic_cast<lucene::index::MultiSegmentReader*>(reader.get());
+ ASSERT_NE(nullptr, multi_reader);
+ const auto* segments = multi_reader->getSubReaders();
+ ASSERT_NE(nullptr, segments);
+ ASSERT_GT(segments->length, 1);
+
+ TabletIndex index_meta;
+ TabletIndexPB index_pb;
+ index_pb.set_index_id(910901);
+ index_meta.init_from_pb(index_pb);
+ auto index_file = std::make_shared<RejectingCluceneIndexFileReader>();
+ auto physical_reader =
+ std::make_shared<RecordingNativeInvertedIndexReader>(&index_meta, index_file);
+ auto context = std::make_shared<IndexQueryContext>();
+ context->collection_statistics = std::make_shared<CollectionStatistics>();
+ context->collection_statistics->_total_num_docs = 4;
+ context->collection_statistics->_total_num_tokens[L"a"] = 4;
+ context->collection_statistics->_term_doc_freqs[L"a"][L"hit"] = 3;
+ std::unordered_map<std::string, IndexFieldNameAndTypePair> types {
+ {"a", {"a", std::make_shared<DataTypeString>()}}};
+ InvertedIndexIterator iterator;
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, physical_reader);
+ std::unordered_map<std::string, IndexIterator*> iterators {{"a", &iterator}};
+ FieldReaderResolver resolver(types, iterators, context);
+ for (const auto type :
+ {InvertedIndexQueryType::MATCH_ANY_QUERY, InvertedIndexQueryType::WILDCARD_QUERY}) {
+ FieldReaderBinding binding;
+ binding.logical_field_name = "a";
+ binding.stored_field_name = "a";
+ binding.stored_field_wstr = L"a";
+ binding.index_source = clucene_index_source(reader, L"a", nullptr);
+ binding.leaf_compiler = std::make_shared<LazyLeafCompiler>(
+ L"a", resolver.binding_key_for("a", type), binding.index_source);
+ binding.inverted_reader = physical_reader;
+ binding.binding_key = resolver.binding_key_for("a", type);
+ resolver._cache[binding.binding_key] = binding;
+ resolver._binding_sources[binding.binding_key] = binding.index_source;
+ }
+ auto exists = make_leaf_clause("WILDCARD", "*");
+ exists.field_name = "a";
+ auto term = make_leaf_clause("TERM", "hit");
+ term.field_name = "a";
+ auto boolean = [](const std::string& type, std::vector<TSearchClause> children) {
+ TSearchClause clause;
+ clause.clause_type = type;
+ clause.__set_children(children);
+ return clause;
+ };
+ SearchNullBitmapResolver null_resolver;
+ for (const auto& nulls : {make_bitmap({}), make_bitmap({1, 2})}) {
+ physical_reader->set_null_bitmap(nulls);
+ roaring::Roaring expected_exists = make_bitmap({0, 1, 2, 3}) - nulls;
+ roaring::Roaring expected_term = make_bitmap({0, 1, 3}) - nulls;
+ for (const auto& [clause, expected] :
+ std::vector<std::pair<TSearchClause, roaring::Roaring>> {
+ {exists, expected_exists},
+ {boolean("NOT", {exists}), nulls},
+ {boolean("AND", {exists, term}), expected_term},
+ {boolean("OR", {exists, term}), expected_exists}}) {
+ for (bool scoring : {false, true}) {
+ QueryPtr query;
+ std::string binding_key;
+ auto status = function_search->build_query_recursive(
+ clause, context, resolver, &query, &binding_key, "or", -1, 4);
+ ASSERT_TRUE(status.ok()) << status;
+ auto exec_ctx =
+ build_variant_search_query_execution_context(4, resolver, &null_resolver);
+ auto weight = query->weight(scoring);
+ auto actual = std::make_shared<roaring::Roaring>();
+ auto similarity = std::make_shared<CollectionSimilarity>();
+ collect_multi_segment_doc_set(weight, exec_ctx, binding_key, actual, similarity,
+ scoring);
+ EXPECT_TRUE(*actual == expected) << clause.clause_type;
+ for (auto doc : *actual) {
+ EXPECT_LT(doc, 4);
+ }
+ if (scoring && clause.clause_type == "WILDCARD") {
+ const auto& scores = similarity->_bm25_scores;
+ EXPECT_EQ(expected.cardinality(), scores.size());
+ for (auto doc : expected) {
+ EXPECT_FLOAT_EQ(1.0F, scores.at(doc));
+ }
+ }
+ for (bool wand : {false, true}) {
+ actual = std::make_shared<roaring::Roaring>();
+ collect_multi_segment_top_k(weight, exec_ctx, binding_key, 20, actual, nullptr,
+ wand);
+ EXPECT_TRUE(*actual == expected);
+ }
+ }
+ }
+ }
+
+ // A nested leaf's element domain can be larger than the active parent domain.
+ // Preserve all four child ordinals before the existing mapper runs.
+ resolver.set_leaf_num_rows_resolver([](const std::string&, uint32_t* count) {
+ *count = 4;
+ return Status::OK();
+ });
+ physical_reader->set_null_bitmap(make_bitmap({}));
+ resolver.set_leaf_query_mapper([&](const std::string&, QueryPtr* child) {
+ QueryExecutionContext child_ctx;
+ child_ctx.segment_num_rows = 2;
+ auto sc = (*child)->weight(false)->scorer(child_ctx);
+ auto parents = std::make_shared<roaring::Roaring>();
+ for (auto doc = sc->doc(); doc != TERMINATED; doc = sc->advance()) {
+ parents->add(doc / 2);
+ }
+ *child = std::make_shared<BitSetQuery>(parents);
+ return Status::OK();
+ });
+ QueryPtr nested;
+ std::string nested_key;
+ ASSERT_TRUE(function_search
+ ->build_query_recursive(exists, context, resolver, &nested, &nested_key,
+ "or", -1, 2)
+ .ok());
+ QueryExecutionContext parent_ctx;
+ parent_ctx.segment_num_rows = 2;
+ auto sc = nested->weight(false)->scorer(parent_ctx);
+ EXPECT_EQ(0, sc->doc());
+ EXPECT_EQ(1, sc->advance());
+ EXPECT_EQ(TERMINATED, sc->advance());
+}
+
+TEST_F(FunctionSearchTest, SearchTopKAboveCollectorCapacityPreservesAllMatches) {
+ SniiScoringFixture fixture(910902, 30000);
+ roaring::Roaring matches;
+ matches.addRange(0, 15000);
+ fixture.reader->fake_source->add("alpha",
+ std::vector<uint32_t>(matches.begin(), matches.end()));
+ fixture.context->collection_statistics = std::make_shared<FixedSearchStatistics>();
+ fixture.context->query_limit = 20000;
+ fixture.context->is_asc = false;
+ InvertedIndexIterator iterator;
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, fixture.reader);
+ std::unordered_map<std::string, IndexFieldNameAndTypePair> types {
+ {"body", {"body", std::make_shared<DataTypeString>()}}};
+ std::unordered_map<std::string, IndexIterator*> iterators {{"body", &iterator}};
+ TSearchParam param;
+ param.root = make_leaf_clause("MATCH", "alpha");
+ InvertedIndexResultBitmap result;
+ auto status = function_search->evaluate_inverted_index_with_search_param(
+ param, types, iterators, fixture.num_rows, result, false, nullptr, {}, fixture.context);
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_NE(nullptr, result.get_data_bitmap());
+ EXPECT_TRUE(*result.get_data_bitmap() == matches);
+ EXPECT_EQ(15000, fixture.context->collection_similarity->_bm25_scores.size());
+}
+
} // namespace doris
diff --git a/be/test/exprs/function/search/search_lowering_test.cpp b/be/test/exprs/function/search/search_lowering_test.cpp
index e42a26b2d6e..34f57229096 100644
--- a/be/test/exprs/function/search/search_lowering_test.cpp
+++ b/be/test/exprs/function/search/search_lowering_test.cpp
@@ -186,10 +186,10 @@ TEST(SearchLoweringTest, LeafPassesItsQueryTypeHintToTheCatalog) {
EXPECT_EQ(catalog.resolved_query_types[kKeyword], InvertedIndexQueryType::EQUAL_QUERY);
}
-TEST(SearchLoweringTest, UnboundFieldLowersToUnknown) {
+TEST(SearchLoweringTest, UnboundFieldLowersToEmpty) {
FakeCatalog catalog;
auto node = lower(leaf("TERM", kUnbound, "x"), catalog);
- EXPECT_NE(node->as<Unknown>(), nullptr);
+ EXPECT_NE(node->as<Empty>(), nullptr);
ASSERT_NE(node->field(), nullptr);
EXPECT_EQ(node->field()->name, kUnbound);
EXPECT_TRUE(node->field()->binding.empty());
@@ -449,9 +449,11 @@ TEST(SearchLoweringTest, ScalarIndexLowersTermAndExactToCompare) {
EXPECT_EQ(compare->field.binding, "price#eq");
}
for (const char* clause_type : {"PHRASE", "PREFIX", "WILDCARD", "REGEXP", "RANGE", "LIST"}) {
- EXPECT_NE(lower(leaf(clause_type, kNumber, "42"), catalog)->as<Unknown>(), nullptr)
+ NodePtr node;
+ EXPECT_FALSE(lower_search_clause(leaf(clause_type, kNumber, "42"), {}, catalog, &node).ok())
<< clause_type;
}
+ EXPECT_NE(lower(leaf("WILDCARD", kNumber, "*"), catalog)->as<Exists>(), nullptr);
EXPECT_TRUE(catalog.analyzed_values.empty());
}
diff --git a/be/test/exprs/vsearch_expr_test.cpp b/be/test/exprs/vsearch_expr_test.cpp
index 332076ad1bc..4e1b3edf5cc 100644
--- a/be/test/exprs/vsearch_expr_test.cpp
+++ b/be/test/exprs/vsearch_expr_test.cpp
@@ -40,15 +40,14 @@
#include "storage/index/inverted/inverted_index_iterator.h"
#include "storage/index/inverted/inverted_index_reader.h"
#include "storage/schema.h"
+#include "storage/segment/segment.h"
#include "storage/segment/variant/nested_group_provider.h"
-#include "storage/tablet/tablet_schema.h"
#if defined(__clang__)
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wkeyword-macro"
#endif
#include "exprs/vslot_ref.h"
-#include "storage/segment/segment.h"
#if defined(__clang__)
#pragma clang diagnostic pop
#endif
@@ -112,22 +111,6 @@ std::shared_ptr<IndexExecContext> make_inverted_context(
std::make_shared<ReadSchema>(std::vector<TabletColumnPtr> {}));
}
-std::shared_ptr<segment_v2::Segment> make_segment_with_variant_parent() {
- TabletSchemaPB schema_pb;
- schema_pb.set_keys_type(KeysType::DUP_KEYS);
- auto* parent = schema_pb.add_column();
- parent->set_unique_id(0);
- parent->set_name("data");
- parent->set_type("VARIANT");
- parent->set_is_key(false);
- parent->set_is_nullable(true);
-
- auto tablet_schema = std::make_shared<TabletSchema>();
- tablet_schema->init_from_pb(schema_pb);
- return std::make_shared<segment_v2::Segment>(0, RowsetId(), tablet_schema,
- InvertedIndexFileInfo());
-}
-
} // namespace
class VSearchExprTest : public testing::Test {
@@ -1409,7 +1392,8 @@ TEST_F(VSearchExprTest, EvaluateInvertedIndexHandlesMissingIterators) {
context->set_index_context(inverted_ctx);
auto status = expr->evaluate_inverted_index(context.get(), 32);
- EXPECT_TRUE(status.ok());
+ EXPECT_EQ(ErrorCode::INVERTED_INDEX_NOT_SUPPORTED, status.code());
+ EXPECT_FALSE(inverted_ctx->has_index_result_for_expr(expr.get()));
EXPECT_FALSE(status_map[0][expr.get()]);
}
@@ -1439,22 +1423,86 @@ TEST_F(VSearchExprTest, MissingVariantChildIteratorDoesNotUseParentIterator) {
status_map[0][expr.get()] = false;
status_map[1][expr.get()] = false;
- // Keep the parent in the segment schema so a parent-rebinding implementation would find it.
+ // Without storage evidence, a missing child iterator is unavailable rather than missing.
segment_v2::ColumnIteratorOptions column_iter_opts;
- auto segment = make_segment_with_variant_parent();
auto inverted_ctx = std::make_shared<IndexExecContext>(
- index_iterators, storage_types, status_map, nullptr, segment.get(), column_iter_opts,
+ index_iterators, storage_types, status_map, nullptr, nullptr, column_iter_opts,
std::make_shared<ReadSchema>(std::vector<TabletColumnPtr> {}));
auto context = std::make_shared<VExprContext>(expr);
context->set_index_context(inverted_ctx);
auto status = expr->evaluate_inverted_index(context.get(), 32);
- EXPECT_TRUE(status.ok()) << status;
- EXPECT_TRUE(inverted_ctx->has_index_result_for_expr(expr.get()));
+ EXPECT_FALSE(status.ok()) << status;
+ EXPECT_FALSE(inverted_ctx->has_index_result_for_expr(expr.get()));
EXPECT_FALSE(status_map[0][expr.get()]);
EXPECT_FALSE(status_map[1][expr.get()]);
}
+TEST_F(VSearchExprTest, MissingVariantRootUsesCurrentReadSchemaUid) {
+ for (bool historical_same_name : {false, true}) {
+ TabletSchemaPB schema_pb;
+ schema_pb.set_keys_type(KeysType::DUP_KEYS);
+ if (historical_same_name) {
+ auto* parent = schema_pb.add_column();
+ parent->set_unique_id(1);
+ parent->set_name("data");
+ parent->set_type("VARIANT");
+ parent->set_is_key(false);
+ parent->set_is_nullable(true);
+ }
+ auto tablet_schema = std::make_shared<TabletSchema>();
+ tablet_schema->init_from_pb(schema_pb);
+ auto segment = std::make_shared<segment_v2::Segment>(0, RowsetId(), tablet_schema,
+ InvertedIndexFileInfo());
+ ColumnPB leaf_pb;
+ leaf_pb.set_unique_id(26);
+ leaf_pb.set_parent_unique_id(25);
+ leaf_pb.set_name("data.items.message");
+ leaf_pb.set_type("STRING");
+ leaf_pb.set_is_key(false);
+ leaf_pb.set_is_nullable(true);
+ auto read_schema = std::make_shared<ReadSchema>(
+ std::vector<TabletColumnPtr> {std::make_shared<TabletColumn>(leaf_pb)});
+ for (bool negated : {false, true}) {
+ TExprNode node = test_node;
+ node.search_param.__set_original_dsl(negated ? "NOT data.items.message:hello"
+ : "data.items.message:hello");
+ node.search_param.root.__set_field_name("data.items.message");
+ auto& binding = node.search_param.field_bindings.front();
+ binding.__set_field_name("data.items.message");
+ binding.__set_is_variant_subcolumn(true);
+ binding.__set_parent_field_name("data");
+ binding.__set_subcolumn_path("items.message");
+ if (negated) {
+ TSearchClause complement;
+ complement.__set_clause_type("NOT");
+ complement.__set_children({node.search_param.root});
+ node.search_param.__set_root(complement);
+ }
+ auto expr = VSearchExpr::create_shared(node);
+ expr->add_child(create_slot_ref(0, "data.items.message"));
+ std::vector<std::unique_ptr<segment_v2::IndexIterator>> iterators;
+ iterators.emplace_back(nullptr);
+ std::vector<IndexFieldNameAndTypePair> types {
+ {"25.data.items.message", std::make_shared<DataTypeString>()}};
+ std::unordered_map<ColumnId, std::unordered_map<const VExpr*, bool>> status_map;
+ status_map[0][expr.get()] = false;
+ OlapReaderStatistics stats;
+ segment_v2::ColumnIteratorOptions opts;
+ opts.stats = &stats;
+ auto index_context = std::make_shared<IndexExecContext>(
+ iterators, types, status_map, nullptr, segment.get(), opts, read_schema);
+ auto context = std::make_shared<VExprContext>(expr);
+ context->set_index_context(index_context);
+ ASSERT_TRUE(expr->evaluate_inverted_index(context.get(), 4).ok());
+ const auto* result = index_context->get_index_result_for_expr(expr.get());
+ ASSERT_NE(nullptr, result);
+ EXPECT_EQ(negated ? 4 : 0, result->get_data_bitmap()->cardinality());
+ EXPECT_TRUE(result->get_null_bitmap()->isEmpty());
+ }
+ }
+}
+
TEST_F(VSearchExprTest, EvaluateInvertedIndexNestedFallbackReturnsNotSupportedInCE) {
TExprNode nested_node = test_node;
nested_node.num_children = 0;
diff --git a/be/test/storage/index/inverted/reader/collection_statistics_test.cpp b/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
index 02e04d046db..2316c05ff86 100644
--- a/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
+++ b/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
@@ -442,7 +442,7 @@ protected:
}
Status write_legacy_v3_segment(const TabletSchemaSPtr& tablet_schema,
- const std::string& segment_path) {
+ const std::string& segment_path, size_t column_ordinal = 0) {
std::vector<StorePath> paths;
paths.emplace_back(test_dir_, 1024);
auto tmp_file_dirs = std::make_unique<segment_v2::TmpFileDirs>(paths);
@@ -461,13 +461,14 @@ protected:
segment_v2::IndexFileWriter file_writer(fs, index_path_prefix, "legacy_v3", 0,
InvertedIndexStorageFormatPB::V3,
std::move(compound_file));
- const auto index_metas = tablet_schema->inverted_indexs(1);
+ const auto& column = tablet_schema->column(column_ordinal);
+ const auto index_metas = tablet_schema->inverted_indexs(column.unique_id());
DORIS_CHECK(index_metas.size() == 1);
std::unique_ptr<segment_v2::IndexColumnWriter> column_writer;
- RETURN_IF_ERROR(segment_v2::IndexColumnWriter::create(
- &tablet_schema->column(0), &column_writer, &file_writer, index_metas[0]));
+ RETURN_IF_ERROR(segment_v2::IndexColumnWriter::create(&column, &column_writer, &file_writer,
+ index_metas[0]));
std::vector<Slice> values {Slice("alpha beta")};
- RETURN_IF_ERROR(column_writer->add_values("content", values.data(), values.size()));
+ RETURN_IF_ERROR(column_writer->add_values(column.name(), values.data(), values.size()));
RETURN_IF_ERROR(column_writer->finish());
RETURN_IF_ERROR(file_writer.begin_close());
return file_writer.finish_close();
@@ -887,6 +888,48 @@ TEST_F(CollectionStatisticsTest, LegacyV3SegmentsUsePhysicalScoringStatistics) {
expect_collected_term(L"1", L"alpha", 2);
}
+TEST_F(CollectionStatisticsTest, MissingFieldIndexPreservesOtherFieldStatistics) {
+ auto schema = create_legacy_v3_schema();
+ TabletColumn other = schema->column(0);
+ other.set_unique_id(2);
+ other.set_name("other");
+ schema->append_column(other);
+ TabletIndex other_index = *schema->inverted_indexs(1).front();
+ other_index._index_id = 2;
+ other_index._col_unique_ids = {2};
+ schema->append_index(std::move(other_index));
+ const std::string first = test_dir_ + "/only_content.dat";
+ const std::string second = test_dir_ + "/only_other.dat";
+ ASSERT_TRUE(write_legacy_v3_segment(schema, first, 0).ok());
+ ASSERT_TRUE(write_legacy_v3_segment(schema, second, 1).ok());
+
+ auto meta = std::make_shared<collection_statistics::MockRowsetMeta>();
+ auto rowset = std::make_shared<collection_statistics::MockRowset>(schema, meta);
+ rowset->set_num_segments(2);
+ rowset->set_segment_path(0, first);
+ rowset->set_segment_path(1, second);
+ auto reader = std::make_shared<collection_statistics::MockRowsetReader>(rowset);
+ std::vector<RowSetSplits> splits {RowSetSplits(reader)};
+ TSearchClause content;
+ content.__set_clause_type("TERM");
+ content.__set_field_name("content");
+ content.__set_value("alpha");
+ TSearchClause other_clause = content;
+ other_clause.__set_field_name("other");
+ TSearchClause root;
+ root.__set_clause_type("OR");
+ root.__set_children({content, other_clause});
+ ASSERT_TRUE(stats_->collect(runtime_state_.get(), splits, schema, create_search_contexts(root),
+ nullptr)
+ .ok());
+ // Neither segment contains both indexes. Both existing fields must contribute exactly once,
+ // regardless of which missing index the unordered field iteration encounters first.
+ expect_collected_stats(L"1", 2, 2);
+ expect_collected_stats(L"2", 2, 2);
+ expect_collected_term(L"1", L"alpha", 1);
+ expect_collected_term(L"2", L"alpha", 1);
+}
+
TEST_F(CollectionStatisticsTest, LegacyV3SkipsMissingSegmentAfterCollectingAvailableStatistics) {
auto tablet_schema = create_legacy_v3_schema();
const std::string first_segment_path = test_dir_ + "/legacy_v3_0.dat";
@@ -2832,4 +2875,33 @@ TEST_F(CollectionStatisticsTest, ScoringAnalyzerContextReportsWrongFamilyCompone
EXPECT_TRUE(analyzer_context_from_properties(valid).has_value());
}
+TEST(SearchPredicateCollectorTest, MatchAllDoesNotRequireFieldStatistics) {
+ TSearchClause clause;
+ clause.clause_type = "MATCH_ALL_DOCS";
+ SearchPredicateCollector collector;
+ CollectInfoMap infos;
+ auto status = collector.collect_from_clause(clause, nullptr, nullptr, {}, &infos);
+ EXPECT_TRUE(status.ok()) << status;
+ EXPECT_TRUE(infos.empty());
+}
+
+TEST(SearchPredicateCollectorTest, LucenePureExclusionAllowsFieldlessMatchAll) {
+ TSearchClause all;
+ all.clause_type = "MATCH_ALL_DOCS";
+ all.__set_occur(TSearchOccur::SHOULD);
+ TSearchClause excluded;
+ excluded.clause_type = "WILDCARD";
+ excluded.__set_field_name("body");
+ excluded.__set_value("hit*");
+ excluded.__set_occur(TSearchOccur::MUST_NOT);
+ TSearchClause root;
+ root.clause_type = "OCCUR_BOOLEAN";
+ root.__set_children({all, excluded});
+ SearchPredicateCollector collector;
+ CollectInfoMap infos;
+ auto status = collector.collect_from_clause(root, nullptr, nullptr, {}, &infos);
+ EXPECT_TRUE(status.ok()) << status;
+ EXPECT_TRUE(infos.empty());
+}
+
} // namespace doris
diff --git a/be/test/storage/segment/inverted_index_array_test.cpp b/be/test/storage/segment/inverted_index_array_test.cpp
index 7c5fcbca7ee..6354c5a774b 100644
--- a/be/test/storage/segment/inverted_index_array_test.cpp
+++ b/be/test/storage/segment/inverted_index_array_test.cpp
@@ -24,9 +24,12 @@
#include <gtest/gtest.h>
#include <string.h>
+#include <cmath>
#include <map>
#include <memory>
#include <string>
+#include <unordered_map>
+#include <vector>
#include "core/block/block.h"
#include "core/column/column_array.h"
@@ -38,21 +41,27 @@
#include "core/field.h"
#include "core/pod_array_fwd.h"
#include "core/types.h"
+#include "exprs/function/search/function_search.h"
#include "gtest/gtest_pred_impl.h"
#include "io/fs/file_writer.h"
#include "io/fs/local_file_system.h"
#include "runtime/exec_env.h"
+#include "runtime/runtime_state.h"
#include "storage/index/index_compound_reader.h"
#include "storage/index/index_file_reader.h"
#include "storage/index/index_file_writer.h"
#include "storage/index/index_fs_directory.h"
+#include "storage/index/inverted/clucene/clucene_index_reader.h"
#include "storage/index/inverted/clucene/inverted_index_writer.h"
#include "storage/index/inverted/common/inverted_index_desc.h"
+#include "storage/index/inverted/query/scoring/collection_similarity.h"
+#include "storage/index/inverted/reader/collection_statistics.h"
#include "storage/index/zone_map/zone_map_index.h"
#include "storage/iterator/olap_data_convertor.h"
#include "storage/tablet/tablet_schema.h"
#include "storage/tablet/tablet_schema_helper.h"
#include "storage/types.h"
+#include "util/defer_op.h"
#include "util/faststring.h"
#include "util/slice.h"
@@ -1090,6 +1099,177 @@ TEST_F(InvertedIndexArrayTest, MultiBlockWrite) {
test_multi_block_write("multi_block", 0, field);
}
+TEST_F(InvertedIndexArrayTest, SearchExcludesOuterNullArrayPayloadPostings) {
+ auto* env = ExecEnv::GetInstance();
+ auto* previous_query_cache = env->get_inverted_index_query_cache();
+ auto* previous_searcher_cache = env->get_inverted_index_searcher_cache();
+ std::unique_ptr<InvertedIndexQueryCache> query_cache(
+ InvertedIndexQueryCache::create_global_cache(1024 * 1024, 1));
+ std::unique_ptr<InvertedIndexSearcherCache> searcher_cache(
+ InvertedIndexSearcherCache::create_global_instance(1024 * 1024, 1));
+ env->set_inverted_index_query_cache(query_cache.get());
+ env->set_inverted_index_searcher_cache(searcher_cache.get());
+ DEFER({
+ env->set_inverted_index_query_cache(previous_query_cache);
+ env->set_inverted_index_searcher_cache(previous_searcher_cache);
+ });
+
+ TabletColumn column;
+ column.set_unique_id(0);
+ column.set_name("arr1");
+ column.set_type(FieldType::OLAP_FIELD_TYPE_ARRAY);
+ column.set_is_nullable(true);
+ TabletColumn element;
+ element.set_type(FieldType::OLAP_FIELD_TYPE_STRING);
+ column.add_sub_column(element);
+ TabletIndexPB index_pb;
+ index_pb.set_index_type(IndexType::INVERTED);
+ index_pb.set_index_id(26034);
+ index_pb.set_index_name("search_nullable_array");
+ index_pb.add_col_unique_id(0);
+ (*index_pb.mutable_properties())["parser"] = "english";
+ (*index_pb.mutable_properties())["support_phrase"] = "true";
+ TabletIndex index_meta;
+ index_meta.init_from_pb(index_pb);
+ auto fs = io::global_local_filesystem();
+ const std::string prefix(InvertedIndexDescriptor::get_index_file_path_prefix(
+ local_segment_path(kTestDir, "search_null_array", 0)));
+ io::FileWriterPtr file_writer;
+ ASSERT_TRUE(
+ fs->create_file(InvertedIndexDescriptor::get_index_file_path_v2(prefix), &file_writer)
+ .ok());
+ auto index_writer = std::make_unique<IndexFileWriter>(fs, prefix, "search_null_array", 0,
+ InvertedIndexStorageFormatPB::V2,
+ std::move(file_writer));
+ std::unique_ptr<IndexColumnWriter> writer;
+ ASSERT_TRUE(IndexColumnWriter::create(&column, &writer, index_writer.get(), &index_meta).ok());
+ // The ARRAY writer consumes payloads before the outer null map. A NULL
+ // ARRAY can therefore have genuine CLucene term/position postings.
+ std::vector<Slice> values {Slice("hit payload"), Slice("hit payload"), Slice("miss value")};
+ const uint64_t offsets[] = {0, 1, 2, 3};
+ const uint8_t null_map[] = {1, 0, 0};
+ ASSERT_TRUE(writer->add_array_values(sizeof(Slice), values.data(), nullptr,
+ reinterpret_cast<const uint8_t*>(offsets), 3)
+ .ok());
+ ASSERT_TRUE(writer->add_array_nulls(null_map, 3).ok());
+ ASSERT_TRUE(writer->finish().ok());
+ ASSERT_TRUE(index_writer->begin_close().ok());
+ ASSERT_TRUE(index_writer->finish_close().ok());
+
+ auto file_reader =
+ std::make_shared<IndexFileReader>(fs, prefix, InvertedIndexStorageFormatPB::V2);
+ ASSERT_TRUE(file_reader->init().ok());
+ auto reader = FullTextIndexReader::create_shared(&index_meta, file_reader);
+ InvertedIndexIterator iterator;
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, reader);
+ OlapReaderStatistics stats;
+ RuntimeState runtime_state;
+ TQueryOptions query_options;
+ query_options.enable_inverted_index_query_cache = true;
+ query_options.enable_inverted_index_searcher_cache = true;
+ runtime_state.set_query_options(query_options);
+ auto context = std::make_shared<IndexQueryContext>();
+ context->runtime_state = &runtime_state;
+ context->stats = &stats;
+ context->collection_statistics = std::make_shared<CollectionStatistics>();
+ // Scores are only used to verify that filtering preserves the remaining
+ // scorer values. Keep collection statistics deterministic for this fixture.
+ context->collection_statistics->_total_num_docs = 3;
+ context->collection_statistics->_total_num_tokens[L"0"] = 6;
+ context->collection_statistics->_term_doc_freqs[L"0"][L"hit"] = 2;
+ context->collection_statistics->_term_doc_freqs[L"0"][L"payload"] = 2;
+ iterator.set_context(context);
+ auto type = make_nullable(std::make_shared<DataTypeArray>(std::make_shared<DataTypeString>()));
+ std::unordered_map<std::string, IndexFieldNameAndTypePair> types {{"arr1", {"0", type}}};
+ std::unordered_map<std::string, IndexIterator*> iterators {{"arr1", &iterator}};
+
+ // Prove this is not an empty-payload NULL fixture: the real reader has a
+ // hit posting for doc 0 and the independently stored NULL bitmap contains it.
+ FieldReaderResolver resolver(types, iterators, context);
+ FieldReaderBinding binding;
+ ASSERT_TRUE(resolver.resolve("arr1", InvertedIndexQueryType::MATCH_ANY_QUERY, &binding).ok());
+ std::unique_ptr<index_query::PostingsCursor> postings;
+ ASSERT_TRUE(binding.index_source->open_term("hit", false, false, &postings).ok());
+ ASSERT_NE(nullptr, postings);
+ roaring::Roaring raw_hits;
+ index_query::PostingsBlock block;
+ bool eof = false;
+ for (;;) {
+ ASSERT_TRUE(postings->next_block(&block, &eof).ok());
+ if (eof) {
+ break;
+ }
+ for (uint64_t ordinal = 0; ordinal < block.size(); ++ordinal) {
+ raw_hits.add(block.doc_at(ordinal));
+ }
+ }
+ ASSERT_TRUE(raw_hits.contains(0));
+ ASSERT_TRUE(raw_hits.contains(1));
+ InvertedIndexQueryCacheHandle null_handle;
+ ASSERT_TRUE(reader->read_null_bitmap(context, &null_handle).ok());
+ ASSERT_NE(nullptr, null_handle.get_bitmap());
+ auto intersection = raw_hits & *null_handle.get_bitmap();
+ ASSERT_EQ(1, intersection.cardinality());
+ ASSERT_TRUE(intersection.contains(0));
+
+ FunctionSearch search;
+ for (bool scoring : {false, true}) {
+ for (const auto& [clause_type, value] :
+ std::vector<std::pair<std::string, std::string>> {{"TERM", "hit"},
+ {"PHRASE", "hit payload"},
+ {"WILDCARD", "h*"},
+ {"WILDCARD", "*"}}) {
+ TSearchClause leaf;
+ leaf.clause_type = clause_type;
+ leaf.__set_field_name("arr1");
+ leaf.__set_value(value);
+ for (bool negate : {false, true}) {
+ TSearchParam param;
+ param.original_dsl = "nullable array payload contract";
+ param.root = leaf;
+ if (negate) {
+ param.root.clause_type = "NOT";
+ param.root.__set_children({leaf});
+ }
+ roaring::Roaring expected;
+ if (value == "*") {
+ expected.add(negate ? 0 : 1);
+ if (!negate) {
+ expected.add(2);
+ }
+ } else {
+ expected.add(negate ? 0 : 1);
+ if (negate) {
+ expected.add(2);
+ }
+ }
+ // Repeated non-scoring calls exercise cold/warm DSL execution;
+ // scoring bypasses that cache and runs the real CLucene scorer.
+ for (int repetition = 0; repetition < 2; ++repetition) {
+ context->collection_similarity =
+ scoring ? std::make_shared<CollectionSimilarity>() : nullptr;
+ InvertedIndexResultBitmap result;
+ auto status = search.evaluate_inverted_index_with_search_param(
+ param, types, iterators, 3, result, true, nullptr, {}, context);
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_TRUE(*result.get_data_bitmap() == expected)
+ << clause_type << "/" << value << "/not=" << negate
+ << "/score=" << scoring;
+ EXPECT_TRUE(result.get_null_bitmap()->isEmpty());
+ if (scoring) {
+ const auto& scores = context->collection_similarity->_bm25_scores;
+ EXPECT_EQ(expected.cardinality(), scores.size());
+ for (const auto& [doc, score] : scores) {
+ EXPECT_TRUE(expected.contains(doc));
+ EXPECT_TRUE(std::isfinite(score));
+ }
+ }
+ }
+ }
+ }
+ }
+}
+
TEST_F(InvertedIndexArrayTest, ArrayInt) {
TabletColumn arrayTabletColumn;
arrayTabletColumn.set_unique_id(0);
diff --git a/be/test/storage/variant/variant_column_writer_reader_test.cpp b/be/test/storage/variant/variant_column_writer_reader_test.cpp
index e0b8ec1c6ff..87c8da0c3fd 100644
--- a/be/test/storage/variant/variant_column_writer_reader_test.cpp
+++ b/be/test/storage/variant/variant_column_writer_reader_test.cpp
@@ -8604,6 +8604,82 @@ TEST_F(VariantColumnWriterReaderTest, test_streaming_write_plan_skips_rowsets_wi
}
}
+TEST_F(VariantColumnWriterReaderTest, test_search_distinguishes_unindexed_historical_variant_path) {
+ init_variant_tablet(41005);
+ auto rowset = create_variant_rowset({{R"({"missing":"hit"})", R"({})"}}, 1);
+ std::vector<segment_v2::SegmentSharedPtr> segments;
+ ASSERT_TRUE(std::static_pointer_cast<BetaRowset>(rowset)->load_segments(&segments).ok());
+ ASSERT_EQ(1, segments.size());
+ auto& segment = segments.front();
+ const auto& parent = segment->tablet_schema()->column(0);
+ ASSERT_TRUE(segment->tablet_schema()->inverted_indexs(parent.unique_id()).empty());
+
+ for (const std::string path : {"missing", "absent"}) {
+ for (const bool negated : {false, true}) {
+ SCOPED_TRACE(path);
+ SCOPED_TRACE(negated);
+ auto leaf = std::make_shared<TabletColumn>();
+ leaf->set_name(parent.name() + "." + path);
+ leaf->set_type(FieldType::OLAP_FIELD_TYPE_STRING);
+ leaf->set_unique_id(-1);
+ leaf->set_parent_unique_id(parent.unique_id());
+ leaf->set_is_nullable(true);
+ leaf->set_path_info(PathInData(leaf->name()));
+ auto schema = std::make_shared<ReadSchema>(std::vector<TabletColumnPtr> {leaf});
+ TSearchFieldBinding binding;
+ binding.__set_field_name(leaf->name());
+ binding.__set_is_variant_subcolumn(true);
+ binding.__set_parent_field_name(parent.name());
+ binding.__set_subcolumn_path(path);
+ TSearchClause term;
+ term.__set_clause_type("TERM");
+ term.__set_field_name(leaf->name());
+ term.__set_value("hit");
+ TSearchParam param;
+ param.__set_original_dsl((negated ? "NOT " : "") + leaf->name() + ":hit");
+ param.__set_root(term);
+ param.__set_field_bindings({binding});
+ if (negated) {
+ TSearchClause complement;
+ complement.__set_clause_type("NOT");
+ complement.__set_children({term});
+ param.__set_root(complement);
+ }
+ TExprNode node;
+ node.__set_node_type(TExprNodeType::SEARCH_EXPR);
+ node.__set_type(TSlotDescriptorBuilder().type(TYPE_BOOLEAN).build().slotType);
+ node.__set_search_param(param);
+ auto expr = VSearchExpr::create_shared(node);
+ auto slot = std::make_shared<VSlotRef>();
+ slot->set_node_type(TExprNodeType::SLOT_REF);
+ slot->set_column_id(0);
+ slot->_column_name = &leaf->name();
+ expr->add_child(slot);
+ std::vector<std::unique_ptr<segment_v2::IndexIterator>> no_indexes(1);
+ std::vector<IndexFieldNameAndTypePair> types {
+ {leaf->name(), std::make_shared<DataTypeString>()}};
+ std::unordered_map<ColumnId, std::unordered_map<const VExpr*, bool>> status_map;
+ OlapReaderStatistics stats;
+ segment_v2::ColumnIteratorOptions opts;
+ opts.stats = &stats;
+ auto scan = std::make_shared<IndexExecContext>(no_indexes, types, status_map, nullptr,
+ segment.get(), opts, schema);
+ VExprContext context(expr);
+ context.set_index_context(scan);
+ const auto status = expr->evaluate_inverted_index(&context, 2);
+ if (path == "missing") {
+ EXPECT_TRUE(status.is<ErrorCode::INVERTED_INDEX_NOT_SUPPORTED>()) << status;
+ EXPECT_FALSE(scan->has_index_result_for_expr(expr.get()));
+ } else {
+ ASSERT_TRUE(status.ok()) << status;
+ const auto* result = scan->get_index_result_for_expr(expr.get());
+ ASSERT_NE(nullptr, result);
+ EXPECT_EQ(negated ? 2 : 0, result->get_data_bitmap()->cardinality());
+ }
+ }
+ }
+}
+
TEST_F(VariantColumnWriterReaderTest, test_nested_search_uses_current_parent_uid) {
init_variant_tablet(41003);
auto rowset = create_variant_rowset({{R"({"items":[{"msg":"old"}]})"}}, 1);
diff --git a/regression-test/data/inverted_index_p0/test_cross_field_or_with_null.out b/regression-test/data/inverted_index_p0/test_cross_field_or_with_null.out
new file mode 100644
index 00000000000..698949a9dbd
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_cross_field_or_with_null.out
@@ -0,0 +1,21 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !standard_not_ids --
+17
+18
+19
+20
+
+-- !lucene_not_ids --
+17
+18
+19
+20
+
+-- !external_not_search_ids --
+17
+18
+19
+20
+
+-- !sql_not_match_ids --
+
diff --git a/regression-test/data/search/test_search_boundary_cases.out b/regression-test/data/search/test_search_boundary_cases.out
index b77079ede54..fea8fdaf564 100644
--- a/regression-test/data/search/test_search_boundary_cases.out
+++ b/regression-test/data/search/test_search_boundary_cases.out
@@ -12,10 +12,10 @@
3
-- !boundary_3_not_null_field --
-11
+17
-- !boundary_3_external_not_null --
-11
+17
-- !boundary_4_empty_string_search --
@@ -46,10 +46,10 @@
17
-- !boundary_9_multiple_not --
-9
+18
-- !boundary_9_external_multiple_not --
-9
+18
-- !boundary_10_performance --
3
diff --git a/regression-test/data/search/test_search_dsl_syntax.out b/regression-test/data/search/test_search_dsl_syntax.out
index e8f6c627121..abe6e0c552a 100644
--- a/regression-test/data/search/test_search_dsl_syntax.out
+++ b/regression-test/data/search/test_search_dsl_syntax.out
@@ -70,6 +70,7 @@
8 Cloud Computing AWS Amazon Web Services deployment and scaling strategies
9 Kubernetes Orchestration Container orchestration with Kubernetes and Docker
10 Microservices Architecture Design patterns for scalable microservices systems
+11 Test with null content \N
12 Test with null title This content has a title but title field is null
13 Test with null category Content with null category field
14 Test with null tags Content with null tags field
@@ -154,6 +155,7 @@
8 Cloud Computing AWS
9 Kubernetes Orchestration
10 Microservices Architecture
+11 Test with null content
12 Test with null title
13 Test with null category
14 Test with null tags
@@ -175,6 +177,7 @@
8 Cloud Computing AWS
9 Kubernetes Orchestration
10 Microservices Architecture
+11 Test with null content
12 Test with null title
13 Test with null category
14 Test with null tags
@@ -204,6 +207,7 @@
8 Cloud Computing AWS Amazon Web Services deployment and scaling strategies
9 Kubernetes Orchestration Container orchestration with Kubernetes and Docker
10 Microservices Architecture Design patterns for scalable microservices systems
+11 Test with null content \N
17 Message about success This is a success message for testing
18 Error message details This is an error message for testing
19 Warning message content This is a warning message for testing
@@ -227,6 +231,7 @@
8 Cloud Computing AWS Cloud
9 Kubernetes Orchestration DevOps
10 Microservices Architecture Architecture
+13 Test with null category \N
17 Message about success Message
18 Error message details Message
19 Warning message content Message
@@ -250,6 +255,7 @@
8 Cloud Computing AWS cloud-computing, AWS, deployment
9 Kubernetes Orchestration kubernetes, docker, containers
10 Microservices Architecture microservices, scalability, patterns
+14 Test with null tags \N
17 Message about success success, message
18 Error message details error, message
19 Warning message content warning, message
@@ -272,6 +278,7 @@
12 Test with null title Test Author
13 Test with null category Test Author
14 Test with null tags Test Author
+15 Test with null author \N
16 Test with null status Test Author
17 Message about success System
18 Error message details System
@@ -296,6 +303,7 @@
8 Cloud Computing AWS published
9 Kubernetes Orchestration published
10 Microservices Architecture published
+16 Test with null status \N
17 Message about success published
18 Error message details published
19 Warning message content published
diff --git a/regression-test/data/search/test_search_not_null_bitmap.out b/regression-test/data/search/test_search_not_null_bitmap.out
index b23fae03cb6..c01b67bcc8a 100644
--- a/regression-test/data/search/test_search_not_null_bitmap.out
+++ b/regression-test/data/search/test_search_not_null_bitmap.out
@@ -5,6 +5,7 @@
4
-- !not_external_ids --
+1
3
4
@@ -12,12 +13,19 @@
3
-- !not_external_count --
-2
+3
+
+-- !not_match_ids --
+3
+4
-- !all_null_internal --
3
-- !all_null_external --
+3
+
+-- !all_null_not_match_count --
0
-- !mixed_not_title_search --
@@ -27,6 +35,8 @@
6
-- !mixed_not_title_external --
+2
+4
5
6
@@ -39,6 +49,8 @@
-- !mixed_not_content_external --
2
+3
+4
5
6
@@ -50,6 +62,7 @@
-- !mixed_complex_external --
1
+2
3
6
@@ -59,5 +72,7 @@
6
-- !multi_must_not_external --
+2
+4
6
diff --git a/regression-test/data/search/test_search_null_regression.out b/regression-test/data/search/test_search_null_regression.out
index 1f9ad79813e..6e9fe781e66 100644
--- a/regression-test/data/search/test_search_null_regression.out
+++ b/regression-test/data/search/test_search_null_regression.out
@@ -16,24 +16,30 @@
5 Biography The life of Selma Blair, a talented actress
-- !regression_2_internal_not --
-5
+8
-- !regression_2_external_not --
-5
+8
-- !regression_2_internal_not_rows --
1 Ronald Reagan Biography of the 40th president
2 Movie Review Selma Blair gives an outstanding performance
+4 Selma Montgomery \N
5 Biography The life of Selma Blair, a talented actress
+10 Clean Article \N
11 Simple Article No special keywords in this content
13 Non-empty title
+14 \N \N
-- !regression_2_external_not_rows --
1 Ronald Reagan Biography of the 40th president
2 Movie Review Selma Blair gives an outstanding performance
+4 Selma Montgomery \N
5 Biography The life of Selma Blair, a talented actress
+10 Clean Article \N
11 Simple Article No special keywords in this content
13 Non-empty title
+14 \N \N
-- !regression_3_null_or --
3
@@ -60,10 +66,10 @@
7
-- !regression_8_multiple_not --
-10
+15
-- !regression_8_external_multiple_not --
-10
+15
-- !regression_9_empty_string --
7
diff --git a/regression-test/data/search/test_search_null_semantics.out b/regression-test/data/search/test_search_null_semantics.out
index 237bd14bc1b..95c9b41cc30 100644
--- a/regression-test/data/search/test_search_null_semantics.out
+++ b/regression-test/data/search/test_search_null_semantics.out
@@ -6,16 +6,18 @@
3
-- !test_case_2_internal_not --
-4
+6
-- !test_case_2_external_not --
-4
+6
-- !test_case_2_phrase_not --
1
2
3
+4
5
+6
7
9
10
@@ -35,13 +37,13 @@
2
-- !test_case_6_not_title_search --
-6
+8
-- !test_case_6_not_title_external --
-6
+8
-- !test_case_7_mixed --
-2
+3
-- !test_case_8_all_null --
0
@@ -80,13 +82,13 @@
15
-- !nested_3_search --
-14
+28
-- !nested_3_match --
14
-- !nested_4_search --
-14
+20
-- !nested_4_match --
14
@@ -132,10 +134,10 @@
1
-- !ternary_3_search --
-14
+18
-- !ternary_3_external --
-14
+18
-- !ternary_4_search --
6
@@ -150,10 +152,11 @@
2
-- !ternary_6_search --
-14
+18
-- !ternary_6_match --
14
-- !ternary_7_all_null --
0
+
diff --git a/regression-test/data/search/test_search_segment_document_ids.out b/regression-test/data/search/test_search_segment_document_ids.out
index 32220079906..11d6a8df449 100644
--- a/regression-test/data/search/test_search_segment_document_ids.out
+++ b/regression-test/data/search/test_search_segment_document_ids.out
@@ -8,6 +8,7 @@
-- !V2_not_or --
1
+2
-- !V2_missing_or --
0
@@ -16,8 +17,10 @@
-- !V2_missing_and --
-- !V2_missing_not_and --
+0
1
2
+3
-- !V2_phrase --
3
@@ -34,6 +37,7 @@
-- !SNII_not_or --
1
+2
-- !SNII_missing_or --
0
@@ -42,8 +46,10 @@
-- !SNII_missing_and --
-- !SNII_missing_not_and --
+0
1
2
+3
-- !SNII_phrase --
3
diff --git a/regression-test/data/search/test_search_vs_match_consistency.out b/regression-test/data/search/test_search_vs_match_consistency.out
index 51a32500b6d..24ff816b490 100644
--- a/regression-test/data/search/test_search_vs_match_consistency.out
+++ b/regression-test/data/search/test_search_vs_match_consistency.out
@@ -24,22 +24,22 @@ Rainbowman Page
6
-- !test_2_1_internal_not --
-5
+8
-- !test_2_1_external_not --
-5
+8
-- !test_2_2_internal_not --
-9
+12
-- !test_2_2_external_not --
-9
+12
-- !test_2_3_internal_not --
-11
+14
-- !test_2_3_external_not --
-11
+14
-- !test_3_1_or_with_null --
1 Ronald Reagan Biography Detailed biography of Ronald Reagan
@@ -73,7 +73,7 @@ Rainbowman Page
11
-- !man_pat_1_search --
-11
+14
-- !man_pat_1_match --
11
@@ -81,14 +81,17 @@ Rainbowman Page
-- !man_pat_1_search_rows --
1 Mandy Patinkin Career Biography of the talented actor Mandy Patinkin NOT_NULL NOT_NULL
3 \N Article about Mandy Patinkin Broadway performances NULL NOT_NULL
+4 Music Industry \N NOT_NULL NULL
6 Mandy Biography Complete story of Mandy Patinkin life and career NOT_NULL NOT_NULL
8 Theater Arts Mandy Patinkin theatrical performances analysis NOT_NULL NOT_NULL
+10 \N \N NULL NULL
12 Empty title with Mandy Patinkin content NOT_NULL NOT_NULL
13 Valid Title NOT_NULL NOT_NULL
14 Mandy Kesha Collaboration Fictional collaboration between Mandy Patinkin and Kesha NOT_NULL NOT_NULL
15 \N Content with both Mandy Patinkin and Kesha references NULL NOT_NULL
16 Comprehensive Bio Mandy Patinkin full biography with career details NOT_NULL NOT_NULL
18 \N Theater review mentioning Mandy Patinkin outstanding performance NULL NOT_NULL
+19 Pop Culture \N NOT_NULL NULL
20 Celebrity News Latest news about Kesha tour and Mandy Patinkin Broadway show NOT_NULL NOT_NULL
-- !man_pat_1_match_rows --
@@ -143,7 +146,7 @@ Rainbowman Page
1
-- !man_pat_4_search_not --
-8
+11
-- !man_pat_4_match_not --
8
@@ -155,7 +158,7 @@ Rainbowman Page
5
-- !fred_1_search --
-9
+12
-- !fred_1_match --
9
@@ -164,11 +167,14 @@ Rainbowman Page
1 Fred Smith Article about Rahul Gandhi politics
2 John Doe Technology article about software
4 Fred Wilson \N
+5 Mary Johnson \N
+6 \N \N
7 Another Fred Some other content without the target name
9 Fred Thompson Mixed article about Rahul Gandhi and others
10 Clean article without target names
12 \N Technology review without political content
13 Fred Frederick Biography of Rahul Gandhi family
+14 News Reporter \N
15
-- !fred_1_match_rows --
@@ -189,22 +195,22 @@ Rainbowman Page
6
-- !fred_2_not_content --
-5
+9
-- !fred_3_or_not --
-11
+14
-- !fred_3_or_not_match --
11
-- !fred_4_and_not --
-1
+2
-- !fred_4_and_not_match --
1
-- !fred_5_nested --
-9
+12
-- !fred_5_nested_match --
9
diff --git a/regression-test/data/variant_p0/test_variant_search_historical_rowset_index.out b/regression-test/data/variant_p0/test_variant_search_historical_rowset_index.out
new file mode 100644
index 00000000000..4edeee182bf
--- /dev/null
+++ b/regression-test/data/variant_p0/test_variant_search_historical_rowset_index.out
@@ -0,0 +1,157 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !historical_values --
+1 object hit
+2 object \N
+3 object miss
+4 object hit
+
+-- !historical_row_matches --
+1
+4
+
+-- !historical_title --
+2
+4
+
+-- !mixed_values --
+1 object hit
+2 object \N
+3 object miss
+4 object hit
+5 object hit
+6 object miss
+
+-- !mixed_row_matches --
+1
+2
+4
+5
+
+-- !standard_false_cold_new_rows --
+5
+
+-- !standard_false_cold_new_rows_scored --
+5
+
+-- !standard_false_cold_absent --
+
+-- !standard_false_cold_not_absent --
+1
+2
+3
+4
+5
+6
+
+-- !standard_false_repeat_new_rows --
+5
+
+-- !standard_false_repeat_new_rows_scored --
+5
+
+-- !standard_false_repeat_absent --
+
+-- !standard_false_repeat_not_absent --
+1
+2
+3
+4
+5
+6
+
+-- !standard_true_cold_new_rows --
+5
+
+-- !standard_true_cold_new_rows_scored --
+5
+
+-- !standard_true_cold_absent --
+
+-- !standard_true_cold_not_absent --
+1
+2
+3
+4
+5
+6
+
+-- !standard_true_repeat_new_rows --
+5
+
+-- !standard_true_repeat_new_rows_scored --
+5
+
+-- !standard_true_repeat_absent --
+
+-- !standard_true_repeat_not_absent --
+1
+2
+3
+4
+5
+6
+
+-- !lucene_false_cold_new_rows --
+5
+
+-- !lucene_false_cold_new_rows_scored --
+5
+
+-- !lucene_false_cold_absent --
+
+-- !lucene_false_cold_not_absent --
+1
+2
+3
+4
+5
+6
+
+-- !lucene_false_repeat_new_rows --
+5
+
+-- !lucene_false_repeat_new_rows_scored --
+5
+
+-- !lucene_false_repeat_absent --
+
+-- !lucene_false_repeat_not_absent --
+1
+2
+3
+4
+5
+6
+
+-- !lucene_true_cold_new_rows --
+5
+
+-- !lucene_true_cold_new_rows_scored --
+5
+
+-- !lucene_true_cold_absent --
+
+-- !lucene_true_cold_not_absent --
+1
+2
+3
+4
+5
+6
+
+-- !lucene_true_repeat_new_rows --
+5
+
+-- !lucene_true_repeat_new_rows_scored --
+5
+
+-- !lucene_true_repeat_absent --
+
+-- !lucene_true_repeat_not_absent --
+1
+2
+3
+4
+5
+6
+
diff --git a/regression-test/data/variant_p0/test_variant_search_missing_doc_path.out b/regression-test/data/variant_p0/test_variant_search_missing_doc_path.out
new file mode 100644
index 00000000000..c01ad99a627
--- /dev/null
+++ b/regression-test/data/variant_p0/test_variant_search_missing_doc_path.out
@@ -0,0 +1,305 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !standard_false_first_a --
+1
+
+-- !standard_false_first_b --
+4
+
+-- !standard_false_first_or --
+1
+4
+
+-- !standard_false_first_and_not --
+1
+
+-- !standard_false_first_not_b --
+1
+2
+3
+
+-- !standard_false_first_missing --
+
+-- !standard_false_first_not_missing --
+1
+2
+3
+4
+
+-- !standard_false_first_prefix_b --
+4
+
+-- !standard_false_first_scored_or --
+1 0.5405593
+4 0.5405593
+
+-- !standard_false_first_external_not_b --
+1
+2
+3
+
+-- !standard_false_repeat_a --
+1
+
+-- !standard_false_repeat_b --
+4
+
+-- !standard_false_repeat_or --
+1
+4
+
+-- !standard_false_repeat_and_not --
+1
+
+-- !standard_false_repeat_not_b --
+1
+2
+3
+
+-- !standard_false_repeat_missing --
+
+-- !standard_false_repeat_not_missing --
+1
+2
+3
+4
+
+-- !standard_false_repeat_prefix_b --
+4
+
+-- !standard_false_repeat_scored_or --
+1 0.5405593
+4 0.5405593
+
+-- !standard_false_repeat_external_not_b --
+1
+2
+3
+
+-- !standard_true_first_a --
+1
+
+-- !standard_true_first_b --
+4
+
+-- !standard_true_first_or --
+1
+4
+
+-- !standard_true_first_and_not --
+1
+
+-- !standard_true_first_not_b --
+1
+2
+3
+
+-- !standard_true_first_missing --
+
+-- !standard_true_first_not_missing --
+1
+2
+3
+4
+
+-- !standard_true_first_prefix_b --
+4
+
+-- !standard_true_first_scored_or --
+1 0.5405593
+4 0.5405593
+
+-- !standard_true_first_external_not_b --
+1
+2
+3
+
+-- !standard_true_repeat_a --
+1
+
+-- !standard_true_repeat_b --
+4
+
+-- !standard_true_repeat_or --
+1
+4
+
+-- !standard_true_repeat_and_not --
+1
+
+-- !standard_true_repeat_not_b --
+1
+2
+3
+
+-- !standard_true_repeat_missing --
+
+-- !standard_true_repeat_not_missing --
+1
+2
+3
+4
+
+-- !standard_true_repeat_prefix_b --
+4
+
+-- !standard_true_repeat_scored_or --
+1 0.5405593
+4 0.5405593
+
+-- !standard_true_repeat_external_not_b --
+1
+2
+3
+
+-- !lucene_false_first_a --
+1
+
+-- !lucene_false_first_b --
+4
+
+-- !lucene_false_first_or --
+1
+4
+
+-- !lucene_false_first_and_not --
+1
+
+-- !lucene_false_first_not_b --
+1
+2
+3
+
+-- !lucene_false_first_missing --
+
+-- !lucene_false_first_not_missing --
+1
+2
+3
+4
+
+-- !lucene_false_first_prefix_b --
+4
+
+-- !lucene_false_first_scored_or --
+1 0.5405593
+4 0.5405593
+
+-- !lucene_false_first_external_not_b --
+1
+2
+3
+
+-- !lucene_false_repeat_a --
+1
+
+-- !lucene_false_repeat_b --
+4
+
+-- !lucene_false_repeat_or --
+1
+4
+
+-- !lucene_false_repeat_and_not --
+1
+
+-- !lucene_false_repeat_not_b --
+1
+2
+3
+
+-- !lucene_false_repeat_missing --
+
+-- !lucene_false_repeat_not_missing --
+1
+2
+3
+4
+
+-- !lucene_false_repeat_prefix_b --
+4
+
+-- !lucene_false_repeat_scored_or --
+1 0.5405593
+4 0.5405593
+
+-- !lucene_false_repeat_external_not_b --
+1
+2
+3
+
+-- !lucene_true_first_a --
+1
+
+-- !lucene_true_first_b --
+4
+
+-- !lucene_true_first_or --
+1
+4
+
+-- !lucene_true_first_and_not --
+1
+
+-- !lucene_true_first_not_b --
+1
+2
+3
+
+-- !lucene_true_first_missing --
+
+-- !lucene_true_first_not_missing --
+1
+2
+3
+4
+
+-- !lucene_true_first_prefix_b --
+4
+
+-- !lucene_true_first_scored_or --
+1 0.5405593
+4 0.5405593
+
+-- !lucene_true_first_external_not_b --
+1
+2
+3
+
+-- !lucene_true_repeat_a --
+1
+
+-- !lucene_true_repeat_b --
+4
+
+-- !lucene_true_repeat_or --
+1
+4
+
+-- !lucene_true_repeat_and_not --
+1
+
+-- !lucene_true_repeat_not_b --
+1
+2
+3
+
+-- !lucene_true_repeat_missing --
+
+-- !lucene_true_repeat_not_missing --
+1
+2
+3
+4
+
+-- !lucene_true_repeat_prefix_b --
+4
+
+-- !lucene_true_repeat_scored_or --
+1 0.5405593
+4 0.5405593
+
+-- !lucene_true_repeat_external_not_b --
+1
+2
+3
+
diff --git a/regression-test/suites/inverted_index_p0/test_complex_or_null_semantics.groovy b/regression-test/suites/inverted_index_p0/test_complex_or_null_semantics.groovy
index 1ad1942afa7..9d4f7427a46 100644
--- a/regression-test/suites/inverted_index_p0/test_complex_or_null_semantics.groovy
+++ b/regression-test/suites/inverted_index_p0/test_complex_or_null_semantics.groovy
@@ -102,8 +102,8 @@ suite("test_complex_or_null_semantics") {
logger.info("Test 1 PASSED: Cross-field OR with NULL - MATCH and SEARCH consistent")
// Test 2: Complex nested query similar to original bug report
- // Standard mode should match MATCH behavior (SQL three-valued logic)
- // Lucene mode uses two-valued logic (different NULL handling)
+ // SEARCH is two-valued in both modes; MATCH retains three-valued logic.
+ // This fixture matches every row except id=19 in either semantic layer.
def test2_match = sql """
SELECT COUNT(*) FROM ${tableName}
WHERE content MATCH_ANY 'President of the United States'
@@ -136,15 +136,15 @@ suite("test_complex_or_null_semantics") {
', '{"mode":"standard"}')
"""
- assertEquals(test2_match[0][0], test2_search_standard[0][0],
- "Standard mode: Complex nested query MATCH and SEARCH should return same count")
+ assertEquals(29, test2_match[0][0], "MATCH keeps its existing NULL semantics")
+ assertEquals(29, test2_search_standard[0][0], "Standard SEARCH excludes only the Braveheart row")
- // Lucene mode: different NULL semantics, just verify it returns a valid result
+ // Group NOT as an OR child so Lucene builds the equivalent Boolean query.
def test2_search_lucene = sql """
SELECT COUNT(*) FROM ${tableName}
WHERE SEARCH('
content:ANY("President of the United States")
- OR NOT (
+ OR (NOT (
content:Braveheart
AND (
NOT content:ALL("List of presidents of India")
@@ -153,12 +153,12 @@ suite("test_complex_or_null_semantics") {
OR content:ALL("Disney+ Hotstar")
)
)
- )
- ')
+ ))
+ ', '{"mode":"lucene"}')
"""
- assertTrue(test2_search_lucene[0][0] >= 0, "Lucene mode: Complex nested query should return valid result")
+ assertEquals(29, test2_search_lucene[0][0], "Grouped Lucene SEARCH matches the same Boolean set")
- logger.info("Test 2 PASSED: Complex nested query - standard mode consistent with MATCH (count: ${test2_match[0][0]}), lucene mode: ${test2_search_lucene[0][0]}")
+ logger.info("Test 2 PASSED: Complex nested query - equivalent SEARCH modes (count: ${test2_match[0][0]}), lucene mode: ${test2_search_lucene[0][0]}")
// Test 3: Verify the 15 critical rows are included
def test3 = sql """
diff --git a/regression-test/suites/inverted_index_p0/test_cross_field_or_with_null.groovy b/regression-test/suites/inverted_index_p0/test_cross_field_or_with_null.groovy
index 410f5019ae7..22b0600dd21 100644
--- a/regression-test/suites/inverted_index_p0/test_cross_field_or_with_null.groovy
+++ b/regression-test/suites/inverted_index_p0/test_cross_field_or_with_null.groovy
@@ -118,14 +118,8 @@ suite("test_cross_field_or_with_null") {
// Test 4: NOT with cross-field OR
// NOT (title MATCH "Philosophy" OR content MATCH "Disney+ Hotstar")
- // Standard mode (SQL three-valued logic):
- // Rows 1-16: OR = TRUE -> NOT TRUE = FALSE (excluded)
- // Rows 17-20: FALSE OR NULL = NULL -> NOT NULL = NULL (excluded)
- // Expected: 0 rows
- // Lucene mode (two-valued logic):
- // Rows 1-16: OR = TRUE -> NOT TRUE = FALSE (excluded)
- // Rows 17-20: no match -> NOT no_match = TRUE (included)
- // Expected: 4 rows
+ // Both SEARCH modes complement the match set: rows 17-20 match NOT.
+ // SQL NOT MATCH retains UNKNOWN and excludes those NULL content rows.
def result4_match = sql """
SELECT COUNT(*) FROM ${tableName}
WHERE NOT (title MATCH_ALL 'Philosophy' OR content MATCH_ALL 'Disney+ Hotstar')
@@ -137,13 +131,33 @@ suite("test_cross_field_or_with_null") {
"""
assertEquals(0, result4_match[0][0]) // All rows excluded due to NULL semantics
- assertEquals(result4_match[0][0], result4_search_standard[0][0])
+ assertEquals(4, result4_search_standard[0][0], "Standard SEARCH includes non-matching NULL rows")
def result4_search_lucene = sql """
SELECT COUNT(*) FROM ${tableName}
- WHERE SEARCH('NOT (title:ALL(Philosophy) OR content:ALL("Disney+ Hotstar"))')
+ WHERE SEARCH('NOT (title:ALL(Philosophy) OR content:ALL("Disney+ Hotstar"))', '{"mode":"lucene"}')
"""
assertEquals(4, result4_search_lucene[0][0]) // Lucene mode: rows 17-20 included
+ qt_standard_not_ids """
+ SELECT id FROM ${tableName}
+ WHERE SEARCH('NOT (title:ALL(Philosophy) OR content:ALL("Disney+ Hotstar"))', '{"mode":"standard"}')
+ ORDER BY id
+ """
+ qt_lucene_not_ids """
+ SELECT id FROM ${tableName}
+ WHERE SEARCH('NOT (title:ALL(Philosophy) OR content:ALL("Disney+ Hotstar"))', '{"mode":"lucene"}')
+ ORDER BY id
+ """
+ qt_external_not_search_ids """
+ SELECT id FROM ${tableName}
+ WHERE NOT SEARCH('title:ALL(Philosophy) OR content:ALL("Disney+ Hotstar")', '{"mode":"standard"}')
+ ORDER BY id
+ """
+ qt_sql_not_match_ids """
+ SELECT id FROM ${tableName}
+ WHERE NOT (title MATCH_ALL 'Philosophy' OR content MATCH_ALL 'Disney+ Hotstar')
+ ORDER BY id
+ """
logger.info("Test 4 passed: NOT with cross-field OR works correctly in both modes")
// Test 5: AND with cross-field OR containing NULL
@@ -199,5 +213,4 @@ suite("test_cross_field_or_with_null") {
assertEquals(19, result8_content_null[0][0]) // Rows 1-15, 17-20
logger.info("Test 8 passed: Individual field queries return expected counts")
- sql "DROP TABLE IF EXISTS ${tableName}"
}
diff --git a/regression-test/suites/inverted_index_p0/test_search_or_null_semantics.groovy b/regression-test/suites/inverted_index_p0/test_search_or_null_semantics.groovy
index 81450a8b379..14004ac5940 100644
--- a/regression-test/suites/inverted_index_p0/test_search_or_null_semantics.groovy
+++ b/regression-test/suites/inverted_index_p0/test_search_or_null_semantics.groovy
@@ -113,7 +113,7 @@ suite("test_search_or_null_semantics") {
"""
def test4_count = test4[0][0]
- assertTrue(test4_count >= 6, "Should return at least 6 rows (rows with Education category and title match)")
+ assertEquals(6, test4_count, "Education category and title match includes exactly six rows")
logger.info("Test 4 PASSED: OR within AND with NULL (returned ${test4_count} rows)")
// Test 5: Deeply nested OR with multiple NULL fields
@@ -126,7 +126,7 @@ suite("test_search_or_null_semantics") {
"""
def test5_count = test5[0][0]
- assertTrue(test5_count >= 16, "Deeply nested OR should return at least 16 rows")
+ assertEquals(17, test5_count, "Deeply nested OR includes exactly rows 1-17")
logger.info("Test 5 PASSED: Deeply nested OR with multiple NULL fields (returned ${test5_count} rows)")
// Test 6: Verify SQL three-valued logic truth table for OR
@@ -145,6 +145,7 @@ suite("test_search_or_null_semantics") {
assertEquals(15, test6b[0][0], "TRUE OR NULL should be TRUE (15 rows included)")
logger.info("Test 6 PASSED: SQL three-valued logic - TRUE OR NULL = TRUE")
+ // Standard mode makes OR NOT a Boolean disjunction rather than a Lucene modifier.
// Test 7: Complex nested query similar to original bug report
def test7 = sql """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ COUNT(*) FROM ${tableName}
@@ -160,19 +161,15 @@ suite("test_search_or_null_semantics") {
)
)
)
- ')
+ ', '{"mode":"standard"}')
"""
def test7_count = test7[0][0]
- assertTrue(test7_count >= 15, "Complex nested query should include the 15 Philosophy rows")
+ assertEquals(20, test7_count, "NOT of the non-matching internal AND includes every row")
logger.info("Test 7 PASSED: Complex nested query (returned ${test7_count} rows)")
// Test 8: NOT with OR and NULL
- // Standard mode (SQL three-valued logic):
- // Rows 1-16: OR = TRUE -> NOT TRUE = FALSE (excluded)
- // Rows 17-20: OR = NULL -> NOT NULL = NULL (excluded)
- // Result: 0 rows
- // Lucene mode (two-valued logic):
+ // Both modes use document Boolean semantics inside SEARCH:
// Rows 1-16: OR = TRUE -> NOT TRUE = FALSE (excluded)
// Rows 17-20: no match -> NOT no_match = TRUE (included)
// Result: 4 rows
@@ -180,11 +177,11 @@ suite("test_search_or_null_semantics") {
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ COUNT(*) FROM ${tableName}
WHERE SEARCH('NOT (title:ALL(Philosophy) OR content:ALL("Disney+ Hotstar"))', '{"mode":"standard"}')
"""
- assertEquals(0, test8_standard[0][0], "Standard mode: NOT OR should exclude all rows due to NULL semantics")
+ assertEquals(4, test8_standard[0][0], "Standard mode: NOT OR includes rows 17-20 whose terms do not match")
def test8_lucene = sql """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ COUNT(*) FROM ${tableName}
- WHERE SEARCH('NOT (title:ALL(Philosophy) OR content:ALL("Disney+ Hotstar"))')
+ WHERE SEARCH('NOT (title:ALL(Philosophy) OR content:ALL("Disney+ Hotstar"))', '{"mode":"lucene"}')
"""
assertEquals(4, test8_lucene[0][0], "Lucene mode: NOT OR returns 4 rows (rows 17-20, no match treated as FALSE)")
logger.info("Test 8 PASSED: NOT OR with NULL works correctly in both modes")
@@ -200,7 +197,6 @@ suite("test_search_or_null_semantics") {
assertEquals(16, test9[0][0], "SEARCH should return 16 rows even without pushdown")
logger.info("Test 9 PASSED: SEARCH works correctly without pushdown")
- sql "DROP TABLE IF EXISTS ${tableName}"
logger.info("All tests PASSED: SEARCH OR NULL semantics work correctly")
}
diff --git a/regression-test/suites/search/README.md b/regression-test/suites/search/README.md
index a819fa331ed..41e37b441f4 100644
--- a/regression-test/suites/search/README.md
+++ b/regression-test/suites/search/README.md
@@ -1,30 +1,23 @@
-# Search Function NULL Semantics Tests
+# SEARCH NULL semantics tests
-This directory contains regression tests for search function NULL value handling fixes.
+SEARCH evaluates a two-valued matching set:
-## Background
+- An ordinary term does not match NULL or a field proven absent from storage.
+- Internal NOT complements that set, so NULL rows can match a negated term.
+- SQL NOT SEARCH complements the same non-nullable result.
+- Ordinary SQL comparisons and MATCH keep three-valued logic. NOT MATCH therefore excludes UNKNOWN rows and can differ from NOT SEARCH.
-Fixed two major issues:
-1. **OR Query Inconsistency**: search() vs match functions returned different results
-2. **NOT Query Inconsistency**: Internal NOT vs external NOT returned different results
+Positive term combinations also compare SEARCH with MATCH. Negated queries keep separate expectations for the two semantic layers.
-## Test Files
+The `tdd_search_2vl` suites additionally cover both parser modes, cold/warm cache, complete scored results, missing VARIANT paths, and nullable ARRAY payloads.
-- `test_search_null_semantics.groovy`: Basic NULL semantics tests
-- `test_search_vs_match_consistency.groovy`: search() vs match() consistency tests
-- `test_search_null_regression.groovy`: Specific regression tests for original bugs
-- `test_search_boundary_cases.groovy`: Edge cases and boundary conditions
+The backend SQL suites use clauses already emitted by the master FE. Backend field-existence
+semantics are checked directly with `TSearchClause` unit tests, including empty arrays, residual
+NULL postings, numeric fields, absent paths, scoring, cache reads, and multiple CLucene readers.
+The FE field-scope regression is maintained separately with its minimal backend binding fix.
-## Expected Behavior
-
-After the fix:
-- OR queries: `search('A or B')` consistent with `A match ... or B match ...`
-- NOT queries: `search('not A')` consistent with `not search('A')`
-- Proper SQL three-value logic for NULL handling
-
-## Running Tests
+Run this directory with the repository wrapper, passing the worktree configuration before `--run`:
```bash
-cd regression-test
-./run-regression-test.sh --suite search
-```
\ No newline at end of file
+./run-regression-test.sh --conf tmp/regression-conf.auto.groovy --run -d search -parallel 1
+```
diff --git a/regression-test/suites/search/tdd_search_2vl/test_search_document_boolean_contract.groovy b/regression-test/suites/search/tdd_search_2vl/test_search_document_boolean_contract.groovy
new file mode 100644
index 00000000000..11b4052c3f6
--- /dev/null
+++ b/regression-test/suites/search/tdd_search_2vl/test_search_document_boolean_contract.groovy
@@ -0,0 +1,197 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Target contract: SEARCH is a two-valued document predicate. A NULL source
+// value does not match a leaf; NOT complements the matching document set.
+// Ordinary SQL comparison and MATCH predicates retain SQL three-valued logic.
+// These are target assertions: do not accept either the old or new result.
+suite("test_search_document_boolean_contract", "p0") {
+ def allIds = [1, 2, 3, 4, 5, 6, 7, 8, 9]
+ def variableNames = [
+ "enable_inverted_index_query",
+ "enable_inverted_index_query_cache",
+ "enable_segment_limit_pushdown",
+ "experimental_enable_virtual_slot_for_cse"
+ ]
+ def savedVariables = [:]
+ variableNames.each { name ->
+ def row = sql("SHOW VARIABLES LIKE '${name}'").find {
+ it[0].toString().equalsIgnoreCase(name)
+ }
+ assertTrue(row != null, "Required session variable is missing: ${name}")
+ savedVariables[name] = row[1].toString()
+ }
+
+ // Queries only contain fixed test strings. Still escape SQL literals at
+ // one boundary so edits cannot silently change the DSL through SQL quoting.
+ def sqlLiteral = { String value ->
+ "'" + value.replace("\\", "\\\\").replace("'", "''") + "'"
+ }
+ def searchPredicate = { String dsl, String mode ->
+ def options = '{"mode":"' + mode + '","default_operator":"OR"}'
+ "SEARCH(${sqlLiteral(dsl)}, ${sqlLiteral(options)})"
+ }
+ def queryIds = { String statement ->
+ sql(statement).collect { ((Number) it[0]).intValue() }.sort()
+ }
+ def assertIds = { List<Integer> expected, String statement, String label ->
+ def actual = queryIds(statement)
+ assertEquals(expected, actual, "${label}\nSQL: ${statement}")
+ }
+
+ try {
+ sql "SET enable_inverted_index_query = true"
+ // Disable SQL CSE here to isolate SEARCH's document Boolean contract.
+ sql "SET experimental_enable_virtual_slot_for_cse = false"
+ sql "DROP TABLE IF EXISTS test_search_document_boolean_contract"
+ sql """
+ CREATE TABLE test_search_document_boolean_contract (
+ id INT,
+ a TEXT,
+ b TEXT,
+ c TEXT,
+ INDEX idx_a(a) USING INVERTED PROPERTIES("parser"="english"),
+ INDEX idx_b(b) USING INVERTED PROPERTIES("parser"="english"),
+ INDEX idx_c(c) USING INVERTED PROPERTIES("parser"="english")
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES(
+ "replication_num"="1",
+ "inverted_index_storage_format"="V2"
+ )
+ """
+ // a/b exhaust all nine hit/miss/NULL combinations. c supplies the
+ // crossed TRUE/NULL exclusions from DORIS-28633: row 1 is excluded by
+ // b despite c=NULL, and row 3 is excluded by c despite b=NULL.
+ sql """
+ INSERT INTO test_search_document_boolean_contract VALUES
+ (1, 'hit', 'hit', NULL),
+ (2, 'hit', 'miss', 'miss'),
+ (3, 'hit', NULL, 'hit'),
+ (4, 'miss', 'hit', 'miss'),
+ (5, 'miss', 'miss', 'miss'),
+ (6, 'miss', NULL, NULL),
+ (7, NULL, 'hit', 'miss'),
+ (8, NULL, 'miss', 'hit'),
+ (9, NULL, NULL, NULL)
+ """
+ sql "SYNC"
+ assertEquals(allIds, queryIds("SELECT id FROM test_search_document_boolean_contract"))
+
+ // Both modes use explicit operators and grouping. In particular,
+ // Lucene A OR NOT A is NOT the ordinary Boolean OR of two predicates:
+ // NOT is a MUST_NOT modifier at that level. A OR (NOT A) makes the
+ // complement a grouped child, giving equivalent Boolean trees here.
+ def cases = [
+ [name: "leaf", dsl: "a:hit", ids: [1, 2, 3]],
+ [name: "and", dsl: "a:hit AND b:hit", ids: [1]],
+ [name: "or", dsl: "a:hit OR b:hit", ids: [1, 2, 3, 4, 7]],
+ [name: "not_and", dsl: "NOT (a:hit AND b:hit)", ids: [2, 3, 4, 5, 6, 7, 8, 9]],
+ [name: "not_or", dsl: "NOT (a:hit OR b:hit)", ids: [5, 6, 8, 9]],
+ [name: "not_leaf", dsl: "NOT a:hit", ids: [4, 5, 6, 7, 8, 9]],
+ [name: "and_not", dsl: "a:hit AND NOT b:hit", ids: [2, 3]],
+ [name: "multi_must_not", dsl: "a:hit AND NOT b:hit AND NOT c:hit", ids: [2]],
+ [name: "not_multi_must_not", dsl: "NOT (a:hit AND NOT b:hit AND NOT c:hit)", ids: [1, 3, 4, 5, 6, 7, 8, 9]],
+ [name: "tautology", dsl: "a:hit OR (NOT a:hit)", ids: allIds],
+ [name: "contradiction", dsl: "a:hit AND NOT a:hit", ids: []],
+ [name: "double_not", dsl: "NOT (NOT a:hit)", ids: [1, 2, 3]],
+ [name: "empty_hit_stream", dsl: "a:unmatchabletoken", ids: []],
+ [name: "not_empty_hit_stream", dsl: "NOT a:unmatchabletoken", ids: allIds]
+ ]
+
+ // DORIS-28632 formerly expected a NULL field to remain UNKNOWN; under
+ // this selected contract, a:hit AND b:hit is FALSE on row 3, and its
+ // complement includes row 3. DORIS-28631 must return that same set
+ // with and without score(). Do not preserve the old standard 3VL
+ // result by accepting a second list of expected IDs.
+ def configurations = [
+ [cache: false, segmentLimit: true],
+ [cache: true, segmentLimit: true]
+ ]
+ // SEARCH remains in its known supported scan path for this first
+ // contract suite. An enable_segment_limit_pushdown=false comparison
+ // belongs in a separate capability test after its accepted plan shape
+ // is verified; never override such a comparison with a SET_VAR hint.
+ configurations.each { configuration ->
+ sql "SET enable_inverted_index_query_cache = ${configuration.cache}"
+ sql "SET enable_segment_limit_pushdown = ${configuration.segmentLimit}"
+ ["standard", "lucene"].each { mode ->
+ cases.each { tc ->
+ def predicate = searchPredicate(tc.dsl, mode)
+ def complement = allIds.findAll { !tc.ids.contains(it) }
+ // A repeat covers warmed execution without pretending
+ // SQL results alone prove that the DSL cache was hit.
+ // FunctionSearchTest verifies actual cache hits and entry ownership.
+ (1..(configuration.cache ? 2 : 1)).each { repetition ->
+ def label = "${mode}/${tc.name}/${configuration}/run${repetition}"
+ assertIds(tc.ids,
+ "SELECT id FROM test_search_document_boolean_contract WHERE ${predicate} ORDER BY id",
+ label)
+ assertIds(complement,
+ "SELECT id FROM test_search_document_boolean_contract WHERE NOT (${predicate}) ORDER BY id",
+ label + "/sql_not")
+ }
+ }
+ }
+ }
+
+ // Keep the score usage in an existing supported shape: one root
+ // SEARCH filter, score() in SELECT/ORDER BY, and a LIMIT. There is no
+ // SEARCH projection, and no claim that a score-ranked result has id
+ // order. LIMIT exceeds the entire fixture, so compare exact ID sets.
+ sql "SET enable_inverted_index_query_cache = false"
+ sql "SET enable_segment_limit_pushdown = true"
+ def scoreCases = cases.findAll {
+ it.name in ["and", "or", "not_and", "not_or", "and_not",
+ "multi_must_not", "not_multi_must_not", "field_exists",
+ "field_missing", "empty_hit_stream", "not_empty_hit_stream"]
+ }
+ ["standard", "lucene"].each { mode ->
+ scoreCases.each { tc ->
+ def predicate = searchPredicate(tc.dsl, mode)
+ ["ASC", "DESC"].each { direction ->
+ def statement = """
+ SELECT id, score() AS s FROM test_search_document_boolean_contract
+ WHERE ${predicate}
+ ORDER BY s ${direction} LIMIT 20
+ """
+ assertIds(tc.ids, statement, "${mode}/${tc.name}/score_${direction}")
+ }
+ }
+ }
+
+ // Protect the outer SQL boundary. NULLs from b='hit' continue to
+ // participate in SQL 3VL even though SEARCH(a:hit) itself is two-valued.
+ ["standard", "lucene"].each { mode ->
+ def predicate = searchPredicate("a:hit", mode)
+ assertIds([2, 4, 5, 6, 7, 8, 9],
+ "SELECT id FROM test_search_document_boolean_contract WHERE NOT (${predicate} AND b='hit') ORDER BY id",
+ "${mode}/mixed_sql_three_valued_boundary")
+ }
+ assertIds([2, 4, 5, 6, 8],
+ "SELECT id FROM test_search_document_boolean_contract WHERE NOT (a='hit' AND b='hit') ORDER BY id",
+ "ordinary_sql_comparisons_remain_three_valued")
+ assertIds([2, 4, 5, 6, 8],
+ "SELECT id FROM test_search_document_boolean_contract WHERE NOT ((a MATCH_ANY 'hit') AND (b MATCH_ANY 'hit')) ORDER BY id",
+ "ordinary_match_remains_three_valued")
+ } finally {
+ savedVariables.each { name, value ->
+ sql "SET ${name} = ${sqlLiteral(value)}"
+ }
+ }
+}
diff --git a/regression-test/suites/search/tdd_search_2vl/test_search_null_array_payload_contract.groovy b/regression-test/suites/search/tdd_search_2vl/test_search_null_array_payload_contract.groovy
new file mode 100644
index 00000000000..713162accfe
--- /dev/null
+++ b/regression-test/suites/search/tdd_search_2vl/test_search_null_array_payload_contract.groovy
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// A NULL ARRAY produced from real columns may retain nested values and index postings.
+// SEARCH must filter those postings before Boolean NOT, while SQL MATCH stays nullable.
+suite("test_search_null_array_payload_contract", "p0") {
+ sql "DROP TABLE IF EXISTS test_search_null_array_payload_source"
+ sql "DROP TABLE IF EXISTS test_search_null_array_payload_target"
+ sql """
+ CREATE TABLE test_search_null_array_payload_source (
+ id INT, lhs ARRAY<STRING>, rhs ARRAY<STRING>
+ ) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num"="1")
+ """
+ sql """
+ CREATE TABLE test_search_null_array_payload_target (
+ id INT, a ARRAY<STRING>,
+ INDEX idx_a(a) USING INVERTED
+ ) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num"="1", "inverted_index_storage_format"="V2")
+ """
+ sql """
+ INSERT INTO test_search_null_array_payload_source VALUES
+ (1, ['hit'], NULL),
+ (2, ['hit'], ['tail']),
+ (3, ['miss'], [])
+ """
+ sql """
+ INSERT INTO test_search_null_array_payload_target
+ SELECT id, array_concat(lhs, rhs) FROM test_search_null_array_payload_source
+ """
+ sql "SYNC"
+ def ids = { String query -> sql(query).collect { ((Number)it[0]).intValue() }.sort() }
+ assertEquals([1], ids("SELECT id FROM test_search_null_array_payload_target WHERE a IS NULL ORDER BY id"))
+ def cases = [
+ [dsl: 'a:hit', matches: [2], complement: [1,3]],
+ [dsl: 'a:"hit"', matches: [2], complement: [1,3]],
+ [dsl: 'a:h*', matches: [2], complement: [1,3]]
+ ]
+ def variable = "enable_inverted_index_query_cache"
+ def saved = sql("SHOW VARIABLES LIKE '${variable}'")[0][1]
+ try {
+ ["standard", "lucene"].each { mode ->
+ [false, true].each { cache ->
+ sql "SET ${variable} = ${cache}"
+ cases.each { tc ->
+ def options = '{"mode":"' + mode + '"}'
+ def positive = "SEARCH('${tc.dsl}', '${options}')"
+ def negative = "SEARCH('NOT (${tc.dsl})', '${options}')"
+ (0..<2).each { repetition ->
+ assertEquals(tc.matches, ids("SELECT id FROM test_search_null_array_payload_target WHERE ${positive} ORDER BY id"), "${mode}/${cache}/${tc.dsl}/${repetition}")
+ assertEquals(tc.complement, ids("SELECT id FROM test_search_null_array_payload_target WHERE ${negative} ORDER BY id"))
+ assertEquals(tc.complement, ids("SELECT id FROM test_search_null_array_payload_target WHERE NOT (${positive}) ORDER BY id"))
+ }
+ ["ASC", "DESC"].each { direction ->
+ assertEquals(tc.matches, ids("SELECT id, score() AS s FROM test_search_null_array_payload_target WHERE ${positive} ORDER BY s ${direction} LIMIT 20"))
+ // A single matching row must survive even when K is smaller
+ // than the segment and the untokenized score is NaN.
+ if (tc.matches == [2]) {
+ assertEquals([2], ids("SELECT id, score() AS s FROM test_search_null_array_payload_target WHERE ${positive} ORDER BY s ${direction} LIMIT 1"))
+ }
+ assertEquals(tc.complement, ids("SELECT id, score() AS s FROM test_search_null_array_payload_target WHERE ${negative} ORDER BY s ${direction} LIMIT 20"))
+ }
+ }
+ }
+ }
+ assertEquals([2], ids("SELECT id FROM test_search_null_array_payload_target WHERE a MATCH_ANY 'hit' ORDER BY id"))
+ assertEquals([3], ids("SELECT id FROM test_search_null_array_payload_target WHERE NOT (a MATCH_ANY 'hit') ORDER BY id"))
+ } finally {
+ sql "SET ${variable} = ${saved}"
+ }
+}
diff --git a/regression-test/suites/search/tdd_search_2vl/test_search_variant_two_valued_contract.groovy b/regression-test/suites/search/tdd_search_2vl/test_search_variant_two_valued_contract.groovy
new file mode 100644
index 00000000000..77a39bf09a2
--- /dev/null
+++ b/regression-test/suites/search/tdd_search_2vl/test_search_variant_two_valued_contract.groovy
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Target: all SEARCH modes are two-valued document predicates, including
+// a Variant path PROVEN absent in the segment. Index unavailability is a
+// different state and is covered by the BE unavailable-binding tests.
+// This fixture only fixes Boolean/NULL semantics. It does not choose the
+// ignore_above contract or promise Elasticsearch indexed-value exists parity.
+suite("test_search_variant_two_valued_contract", "p0") {
+ def allIds = [1, 2, 3, 4]
+ def variableNames = [
+ "enable_inverted_index_query",
+ "enable_inverted_index_query_cache",
+ "enable_segment_limit_pushdown",
+ "experimental_enable_virtual_slot_for_cse"
+ ]
+ def savedVariables = [:]
+ variableNames.each { name ->
+ def row = sql("SHOW VARIABLES LIKE '${name}'").find {
+ it[0].toString().equalsIgnoreCase(name)
+ }
+ assertTrue(row != null)
+ savedVariables[name] = row[1].toString()
+ }
+ def sqlLiteral = { String value ->
+ "'" + value.replace("\\", "\\\\").replace("'", "''") + "'"
+ }
+ def searchPredicate = { String dsl, String mode ->
+ def options = '{"mode":"' + mode + '","default_operator":"OR"}'
+ "SEARCH(${sqlLiteral(dsl)}, ${sqlLiteral(options)})"
+ }
+ def assertIds = { String label, String predicate, List<Integer> expected ->
+ def statement = "SELECT id FROM test_search_variant_two_valued_contract WHERE ${predicate} ORDER BY id"
+ def actual = sql(statement).collect { it[0].toString().toInteger() }
+ logger.info("${label}: expected=${expected}, actual=${actual}, sql=${statement}")
+ assertEquals(expected, actual)
+ }
+
+ try {
+ sql "SET enable_inverted_index_query = true"
+ sql "SET enable_inverted_index_query_cache = false"
+ sql "SET enable_segment_limit_pushdown = true"
+ sql "SET experimental_enable_virtual_slot_for_cse = false"
+ sql "DROP TABLE IF EXISTS test_search_variant_two_valued_contract"
+ sql """
+ CREATE TABLE test_search_variant_two_valued_contract (
+ id INT,
+ s VARCHAR(16),
+ n INT,
+ v VARIANT<PROPERTIES(
+ "variant_enable_doc_mode"="false"
+ )>,
+ INDEX idx_s(s) USING INVERTED PROPERTIES("parser"="none"),
+ INDEX idx_n(n) USING INVERTED,
+ INDEX idx_v(v) USING INVERTED PROPERTIES("parser"="none")
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES(
+ "replication_num"="1",
+ "inverted_index_storage_format"="V2",
+ "disable_auto_compaction"="true"
+ )
+ """
+ // Doc mode is disabled, and the single present path fits the default
+ // materialization budget. This is a known-absent-path fixture rather
+ // than a sparse/doc fallback fixture.
+ // One small INSERT avoids old/new index generations and compaction.
+ // Values are short ASCII, with no analyzer or ignore_above ambiguity.
+ sql """
+ INSERT INTO test_search_variant_two_valued_contract VALUES
+ (1, 'hit', NULL, parse_to_variant('{"present":"hit"}')),
+ (2, 'miss', 1, parse_to_variant('{"present":"miss"}')),
+ (3, NULL, 1, parse_to_variant('{}')),
+ (4, 'hit', 2, NULL)
+ """
+
+ sql "SYNC"
+
+ // Source-value checks make the fixture independently reviewable.
+ // They do not claim to prove a per-segment binding state; the BE
+ // tests separately check binding failures and complete missing-field AST evaluation.
+ assertIds("fixture/all_rows", "TRUE", allIds)
+ assertIds("fixture/scalar_null", "s IS NULL", [3])
+ assertIds("fixture/absent_path", "v['absent'] IS NULL", allIds)
+ assertIds("fixture/present_path_null", "v['present'] IS NULL", [3, 4])
+ assertIds("fixture/present_path_hit", "CAST(v['present'] AS STRING) = 'hit'", [1])
+
+ def cases = [
+ // A wholly empty iterator bundle must still evaluate the AST.
+ [name: "known_missing_leaf", dsl: "v.absent:hit", ids: []],
+ [name: "known_missing_not", dsl: "NOT v.absent:hit", ids: allIds],
+ [name: "known_missing_double_not", dsl: "NOT (NOT v.absent:hit)", ids: []],
+ [name: "all_missing_nested_ast",
+ dsl: "NOT (v.absent:hit OR v.also_absent:hit)", ids: allIds],
+
+ // A known-missing child is FALSE even when another child has a
+ // usable reader. It must not become a full UNKNOWN bitmap.
+ [name: "bound_and_missing", dsl: "s:hit AND v.absent:hit", ids: []],
+ [name: "not_bound_and_missing", dsl: "NOT (s:hit AND v.absent:hit)", ids: allIds],
+ [name: "bound_or_missing", dsl: "s:hit OR v.absent:hit", ids: [1, 4]],
+ [name: "not_bound_or_missing", dsl: "NOT (s:hit OR v.absent:hit)", ids: [2, 3]],
+ [name: "bound_leaf", dsl: "s:hit", ids: [1, 4]],
+ [name: "bound_not_leaf", dsl: "NOT s:hit", ids: [2, 3]],
+
+ // Numeric terms use the same two-valued Boolean contract.
+ [name: "numeric_leaf", dsl: "n:1", ids: [2, 3]],
+ [name: "numeric_not_leaf", dsl: "NOT n:1", ids: [1, 4]]
+ ]
+
+ ["standard", "lucene"].each { mode ->
+ cases.each { tc ->
+ def predicate = searchPredicate(tc.dsl, mode)
+ assertIds("${mode}/${tc.name}", predicate, tc.ids)
+ // SEARCH itself is non-nullable, so SQL NOT must yield the
+ // exact complement of the complete SEARCH result as well.
+ def complement = allIds.findAll { !tc.ids.contains(it) }
+ assertIds("${mode}/${tc.name}/sql_not", "NOT (${predicate})", complement)
+ }
+
+ def predicate = searchPredicate("s:hit", mode)
+ // Row 1 remains UNKNOWN because n is NULL in an OUTER SQL
+ // comparison. The SEARCH leaf being two-valued does not change
+ // SQL AND/NOT semantics outside SEARCH.
+ assertIds("${mode}/outer_sql_three_valued_boundary",
+ "NOT (n = 1 AND ${predicate})", [2, 3, 4])
+
+ // These WHERE-only assertions also protect FE nullability and
+ // constant folding. They may fold before reaching BE, so they
+ // are not a substitute for direct scorer/binding unit tests.
+ assertIds("${mode}/search_is_never_null", "(${predicate}) IS NULL", [])
+ assertIds("${mode}/search_is_always_not_null", "(${predicate}) IS NOT NULL", allIds)
+ }
+ } finally {
+ savedVariables.each { name, value ->
+ sql "SET ${name} = ${sqlLiteral(value)}"
+ }
+ }
+}
diff --git a/regression-test/suites/search/test_search_boundary_cases.groovy b/regression-test/suites/search/test_search_boundary_cases.groovy
index 64b1520f9a5..e2195907c4d 100644
--- a/regression-test/suites/search/test_search_boundary_cases.groovy
+++ b/regression-test/suites/search/test_search_boundary_cases.groovy
@@ -16,6 +16,7 @@
// under the License.
suite("test_search_boundary_cases", "p0") {
+ // SEARCH uses two-valued matching; SQL comparisons and MATCH retain three-valued logic.
def tableName = "search_boundary_test"
// Pin enable_segment_limit_pushdown to prevent CI flakiness from fuzzy testing.
@@ -196,4 +197,4 @@ suite("test_search_boundary_cases", "p0") {
(field2:test OR field2:target OR field2:keyword) AND
NOT (field3:nonexistent OR field4:nonexistent OR field5:nonexistent)', '{"mode":"standard"}')
"""
-}
\ No newline at end of file
+}
diff --git a/regression-test/suites/search/test_search_dsl_syntax.groovy b/regression-test/suites/search/test_search_dsl_syntax.groovy
index d5135367689..5a4d9d4729a 100644
--- a/regression-test/suites/search/test_search_dsl_syntax.groovy
+++ b/regression-test/suites/search/test_search_dsl_syntax.groovy
@@ -16,6 +16,7 @@
// under the License.
suite("test_search_dsl_syntax", "p0") {
+ // SEARCH uses two-valued matching; SQL comparisons and MATCH retain three-valued logic.
def tableName = "search_dsl_test_table"
sql "DROP TABLE IF EXISTS ${tableName}"
diff --git a/regression-test/suites/search/test_search_not_null_bitmap.groovy b/regression-test/suites/search/test_search_not_null_bitmap.groovy
index 9f60d28b270..6f32c066ace 100644
--- a/regression-test/suites/search/test_search_not_null_bitmap.groovy
+++ b/regression-test/suites/search/test_search_not_null_bitmap.groovy
@@ -16,8 +16,11 @@
// under the License.
suite("test_search_not_null_bitmap", "p0") {
- // A NOT clause inside search() keeps rows whose field is NULL, while SQL NOT around search() drops them.
-// Each query is paired with its SQL NOT form.
+ // SEARCH uses two-valued matching; SQL comparisons and MATCH retain three-valued logic.
+ // Regression test for DORIS-24681:
+ // A NULL field is non-matching inside SEARCH. Internal NOT and SQL
+ // NOT SEARCH complement that two-valued result and include NULL rows.
+ // NOT MATCH still excludes UNKNOWN and is covered separately below.
def tableName = "search_not_null_bitmap"
@@ -49,7 +52,7 @@ suite("test_search_not_null_bitmap", "p0") {
Thread.sleep(5000)
// ---------------------------------------------------------------
- // NOT over a field that is NULL in one row
+ // Internal and external NOT SEARCH include NULL rows
// ---------------------------------------------------------------
// NOT in the DSL keeps the NULL row.
@@ -59,7 +62,7 @@ suite("test_search_not_null_bitmap", "p0") {
ORDER BY id
"""
- // A SQL NOT drops it.
+ // SQL NOT consumes SEARCH's non-nullable match result
qt_not_external_ids """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ id FROM ${tableName}
WHERE NOT search('msg:omega')
@@ -76,6 +79,12 @@ suite("test_search_not_null_bitmap", "p0") {
WHERE NOT search('msg:omega')
"""
+ qt_not_match_ids """
+ SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ id FROM ${tableName}
+ WHERE NOT (msg MATCH_ANY 'omega')
+ ORDER BY id
+ """
+
// ---------------------------------------------------------------
// Test with all rows NULL for the searched field
// ---------------------------------------------------------------
@@ -106,7 +115,7 @@ suite("test_search_not_null_bitmap", "p0") {
Thread.sleep(5000)
- // NOT in the DSL keeps every NULL row; a SQL NOT drops them.
+ // All NULL rows are non-matching leaves, so NOT SEARCH includes every row
qt_all_null_internal """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ count(*) FROM ${allNullTable}
WHERE search('NOT msg:anything')
@@ -117,6 +126,11 @@ suite("test_search_not_null_bitmap", "p0") {
WHERE NOT search('msg:anything')
"""
+ qt_all_null_not_match_count """
+ SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ count(*) FROM ${allNullTable}
+ WHERE NOT (msg MATCH_ANY 'anything')
+ """
+
// ---------------------------------------------------------------
// Test with mixed NULL and matching values (multi-field)
// ---------------------------------------------------------------
@@ -153,7 +167,7 @@ suite("test_search_not_null_bitmap", "p0") {
Thread.sleep(5000)
- // NOT on title: the DSL keeps the NULL title rows (id=2,4), a SQL NOT drops them.
+ // NOT on title includes the NULL title rows (id=2,4)
qt_mixed_not_title_search """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ id FROM ${mixedTable}
WHERE search('NOT title:hello')
@@ -166,7 +180,7 @@ suite("test_search_not_null_bitmap", "p0") {
ORDER BY id
"""
- // NOT on content: the DSL keeps the NULL content rows (id=3,4), a SQL NOT drops them.
+ // NOT on content includes the NULL content rows (id=3,4)
qt_mixed_not_content_search """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ id FROM ${mixedTable}
WHERE search('NOT content:morning')
diff --git a/regression-test/suites/search/test_search_null_regression.groovy b/regression-test/suites/search/test_search_null_regression.groovy
index 0ce97ac7c78..35b1a439de7 100644
--- a/regression-test/suites/search/test_search_null_regression.groovy
+++ b/regression-test/suites/search/test_search_null_regression.groovy
@@ -16,6 +16,7 @@
// under the License.
suite("test_search_null_regression", "p0") {
+ // SEARCH uses two-valued matching; SQL comparisons and MATCH retain three-valued logic.
def tableName = "search_null_regression_test"
// Pin enable_segment_limit_pushdown to prevent CI flakiness from fuzzy testing.
@@ -119,7 +120,7 @@ suite("test_search_null_regression", "p0") {
"""
// Regression Test 3: NULL Handling in OR Queries
- // Verify that OR queries properly handle NULL values according to SQL semantics
+ // NULL terms are non-matches inside SEARCH; positive MATCH has the same WHERE hits
qt_regression_3_null_or """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ id FROM ${tableName}
WHERE search('title:NonExistent OR content:Ronald', '{"mode":"standard"}')
@@ -190,4 +191,4 @@ suite("test_search_null_regression", "p0") {
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ count(*) FROM ${tableName}
WHERE search('(title:Ronald OR title:Selma OR content:Round) AND NOT (title:NonExistent AND content:NonExistent)', '{"mode":"standard"}')
"""
-}
\ No newline at end of file
+}
diff --git a/regression-test/suites/search/test_search_null_semantics.groovy b/regression-test/suites/search/test_search_null_semantics.groovy
index 98426816c4d..d9b025a8d08 100644
--- a/regression-test/suites/search/test_search_null_semantics.groovy
+++ b/regression-test/suites/search/test_search_null_semantics.groovy
@@ -16,6 +16,7 @@
// under the License.
suite("test_search_null_semantics", "p0") {
+ // SEARCH uses two-valued matching; SQL comparisons and MATCH retain three-valued logic.
def tableName = "search_null_test"
// Pin enable_segment_limit_pushdown to prevent CI flakiness from fuzzy testing.
@@ -84,7 +85,7 @@ suite("test_search_null_semantics", "p0") {
WHERE not search('content:Round', '{"mode":"standard"}')
"""
- // Test Case 2b: Phrase NOT queries must treat NULL rows as UNKNOWN
+ // Test Case 2b: Phrase NOT complements non-matches, including NULL rows
qt_test_case_2_phrase_not """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ id FROM ${tableName}
WHERE NOT search('content:"Selma Blair"', '{"mode":"standard"}')
@@ -92,7 +93,7 @@ suite("test_search_null_semantics", "p0") {
"""
// Test Case 3: NULL handling in OR queries
- // Verify that NULL OR TRUE = TRUE logic works
+ // A non-matching NULL term OR a matching term is a match
qt_test_case_3_or_with_null """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ id, title, content FROM ${tableName}
WHERE search('title:Ronald OR content:biography', '{"mode":"standard"}')
@@ -100,7 +101,7 @@ suite("test_search_null_semantics", "p0") {
"""
// Test Case 4: NULL handling in AND queries
- // Verify that NULL AND TRUE = NULL logic works
+ // A non-matching NULL term makes SEARCH AND non-matching
qt_test_case_4_and_with_null """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ id, title, content FROM ${tableName}
WHERE search('title:Ronald AND content:biography', '{"mode":"standard"}')
@@ -142,7 +143,7 @@ suite("test_search_null_semantics", "p0") {
"""
// ------------------------------------------------------------------
- // Additional coverage: nested boolean logic with explicit NULL checks
+ // Additional coverage: nested Boolean SEARCH versus SQL/MATCH NULL handling
// ------------------------------------------------------------------
def nestedTable = "search_nested_null_logic"
@@ -274,7 +275,7 @@ suite("test_search_null_semantics", "p0") {
sql "DROP TABLE IF EXISTS ${nestedTable}"
// ------------------------------------------------------------------
- // Additional coverage: general ternary logic scenarios with articles
+ // Additional coverage: two-valued SEARCH versus three-valued MATCH
// ------------------------------------------------------------------
def ternaryTable = "search_three_value_logic"
diff --git a/regression-test/suites/search/test_search_vs_match_consistency.groovy b/regression-test/suites/search/test_search_vs_match_consistency.groovy
index 235d2b61e12..2b42d181ce2 100644
--- a/regression-test/suites/search/test_search_vs_match_consistency.groovy
+++ b/regression-test/suites/search/test_search_vs_match_consistency.groovy
@@ -16,6 +16,7 @@
// under the License.
suite("test_search_vs_match_consistency", "p0") {
+ // SEARCH uses two-valued matching; SQL comparisons and MATCH retain three-valued logic.
def tableName = "search_match_consistency_test"
// Pin enable_segment_limit_pushdown to prevent CI flakiness from fuzzy testing.
@@ -150,7 +151,7 @@ suite("test_search_vs_match_consistency", "p0") {
WHERE title match "Ronald" or (content match_all "Selma Blair")
"""
- // Test Suite 2: NOT query consistency
+ // Test Suite 2: internal NOT equals SQL NOT SEARCH; NOT MATCH remains three-valued
qt_test_2_1_internal_not """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ count(*) FROM ${tableName}
WHERE search('NOT content:Round', '{"mode":"standard"}')
@@ -226,7 +227,7 @@ suite("test_search_vs_match_consistency", "p0") {
"""
// ------------------------------------------------------------------
- // Additional regression coverage: Mandy Patinkin / Kesha consistency
+ // Additional coverage: Mandy Patinkin / Kesha with SEARCH and MATCH NULL boundaries
// ------------------------------------------------------------------
def mandyTable = "search_match_issue_mandy_kesha"
@@ -278,7 +279,7 @@ suite("test_search_vs_match_consistency", "p0") {
Thread.sleep(10000)
- // Mandy/Kesha consistency checks
+ // Compare two-valued SEARCH with three-valued MATCH; NOT can differ on NULL rows
qt_man_pat_1_search """
SELECT /*+SET_VAR(enable_segment_limit_pushdown=true) */ count(*) FROM ${mandyTable}
WHERE search('content:ALL("Mandy Patinkin") OR NOT (content:ANY("Kesha"))', '{"mode":"standard"}')
diff --git a/regression-test/suites/variant_p0/test_variant_search_historical_rowset_index.groovy b/regression-test/suites/variant_p0/test_variant_search_historical_rowset_index.groovy
new file mode 100644
index 00000000000..a17be09029b
--- /dev/null
+++ b/regression-test/suites/variant_p0/test_variant_search_historical_rowset_index.groovy
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// CREATE INDEX changes table metadata without building an index for historical rowsets.
+// Write objects explicitly: an ordinary STRING-to-VARIANT cast remains a scalar string,
+// even when its displayed text looks like JSON, and cannot establish a subpath match.
+suite("test_variant_search_historical_rowset_index", "p0") {
+ // Cloud requires this flag and a non-tokenized index for metadata-only creation.
+ // A tokenized index uses full schema change and rebuilds the historical rowsets.
+ sql "SET enable_add_index_for_new_data = true"
+ sql "DROP TABLE IF EXISTS test_variant_search_historical_rowset_index"
+ sql """
+ CREATE TABLE test_variant_search_historical_rowset_index (
+ id INT,
+ title TEXT,
+ v VARIANT<PROPERTIES(
+ "variant_enable_doc_mode"="false"
+ )>,
+ INDEX idx_title(title) USING INVERTED
+ PROPERTIES("parser"="unicode", "support_phrase"="true")
+ ) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES(
+ "replication_num"="1",
+ "inverted_index_storage_format"="V2",
+ "disable_auto_compaction"="true"
+ )
+ """
+ sql """INSERT INTO test_variant_search_historical_rowset_index VALUES
+ (1, 'miss', parse_to_variant('{"missing":"hit"}')),
+ (2, 'hit', parse_to_variant('{}')),
+ (3, 'miss', parse_to_variant('{"missing":"miss"}')),
+ (4, 'hit', parse_to_variant('{"missing":"hit"}'))"""
+ sql "SYNC"
+ sql """CREATE INDEX idx_v ON test_variant_search_historical_rowset_index(v)
+ USING INVERTED PROPERTIES("parser"="none")"""
+ sql "SYNC"
+
+ qt_historical_values """
+ SELECT id, variant_type(v), CAST(v['missing'] AS STRING)
+ FROM test_variant_search_historical_rowset_index ORDER BY id
+ """
+ qt_historical_row_matches """
+ SELECT id FROM test_variant_search_historical_rowset_index
+ WHERE CAST(v['missing'] AS STRING) = 'hit' ORDER BY id
+ """
+ qt_historical_title """
+ SELECT id FROM test_variant_search_historical_rowset_index
+ WHERE SEARCH('title:hit') ORDER BY id
+ """
+
+ def checkUnavailable = {
+ ["standard", "lucene"].each { mode ->
+ def options = '{"mode":"' + mode + '","default_operator":"OR"}'
+ [false, true].each { cache ->
+ sql "SET enable_inverted_index_query_cache = ${cache}"
+ // A failed evaluation must not leave a partial result in the cache.
+ ["cold", "repeat"].each { round ->
+ ["v.missing:hit", "title:hit OR v.missing:hit",
+ "NOT v.missing:hit", "v.missing:h*"].each { dsl ->
+ test {
+ sql """SELECT id FROM test_variant_search_historical_rowset_index
+ WHERE SEARCH('${dsl}', '${options}') ORDER BY id"""
+ exception "SEARCH index unavailable for Variant field 'v.missing'"
+ }
+ }
+ ["v.missing:hit", "title:hit OR v.missing:hit"].each { dsl ->
+ test {
+ sql """SELECT id, score() AS s
+ FROM test_variant_search_historical_rowset_index
+ WHERE SEARCH('${dsl}', '${options}')
+ ORDER BY s DESC LIMIT 20"""
+ exception "SEARCH index unavailable for Variant field 'v.missing'"
+ }
+ }
+ }
+ }
+ }
+ }
+ checkUnavailable()
+
+ sql """INSERT INTO test_variant_search_historical_rowset_index VALUES
+ (5, 'miss', parse_to_variant('{"missing":"hit"}')),
+ (6, 'miss', parse_to_variant('{"missing":"miss"}'))"""
+ sql "SYNC"
+ qt_mixed_values """
+ SELECT id, variant_type(v), CAST(v['missing'] AS STRING)
+ FROM test_variant_search_historical_rowset_index ORDER BY id
+ """
+ qt_mixed_row_matches """
+ SELECT id FROM test_variant_search_historical_rowset_index
+ WHERE title = 'hit' OR CAST(v['missing'] AS STRING) = 'hit' ORDER BY id
+ """
+ checkUnavailable()
+
+ ["standard", "lucene"].each { mode ->
+ def options = '{"mode":"' + mode + '","default_operator":"OR"}'
+ [false, true].each { cache ->
+ sql "SET enable_inverted_index_query_cache = ${cache}"
+ ["cold", "repeat"].each { round ->
+ def prefix = "${mode}_${cache}_${round}"
+ // Pruning the historical rowset leaves the newly written, executable index.
+ quickTest("${prefix}_new_rows", """
+ SELECT id FROM test_variant_search_historical_rowset_index
+ WHERE id >= 5 AND SEARCH('v.missing:hit', '${options}') ORDER BY id
+ """)
+ quickTest("${prefix}_new_rows_scored", """
+ SELECT id FROM (
+ SELECT id, score() AS s
+ FROM test_variant_search_historical_rowset_index
+ WHERE id >= 5 AND SEARCH('v.missing:hit', '${options}')
+ ORDER BY s DESC LIMIT 20
+ ) scored ORDER BY id
+ """)
+ // An absent path remains a valid empty match, including in the old rowset.
+ quickTest("${prefix}_absent", """
+ SELECT id FROM test_variant_search_historical_rowset_index
+ WHERE SEARCH('v.absent:hit', '${options}') ORDER BY id
+ """)
+ quickTest("${prefix}_not_absent", """
+ SELECT id FROM test_variant_search_historical_rowset_index
+ WHERE SEARCH('NOT v.absent:hit', '${options}') ORDER BY id
+ """)
+ }
+ }
+ }
+}
diff --git a/regression-test/suites/variant_p0/test_variant_search_missing_doc_path.groovy b/regression-test/suites/variant_p0/test_variant_search_missing_doc_path.groovy
new file mode 100644
index 00000000000..27d7341df5c
--- /dev/null
+++ b/regression-test/suites/variant_p0/test_variant_search_missing_doc_path.groovy
@@ -0,0 +1,106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Different rowsets have different doc paths. Unrelated doc values must not make a
+// proven-missing path unavailable, and NOT must complement the missing leaf before union.
+suite("test_variant_search_missing_doc_path", "p0") {
+ sql "DROP TABLE IF EXISTS test_variant_search_missing_doc_path"
+ sql """
+ CREATE TABLE test_variant_search_missing_doc_path (
+ id INT,
+ v VARIANT<PROPERTIES(
+ "variant_enable_doc_mode"="true",
+ "variant_doc_materialization_min_rows"="0"
+ )>,
+ INDEX idx_v(v) USING INVERTED PROPERTIES("parser"="unicode")
+ ) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES(
+ "replication_num"="1",
+ "inverted_index_storage_format"="V2",
+ "disable_auto_compaction"="true"
+ )
+ """
+ sql """INSERT INTO test_variant_search_missing_doc_path VALUES
+ (1, parse_to_variant('{"a":"hit"}')), (2, NULL), (3, parse_to_variant('{}'))"""
+ sql """INSERT INTO test_variant_search_missing_doc_path VALUES
+ (4, parse_to_variant('{"b":"hit"}'))"""
+
+ ["standard", "lucene"].each { mode ->
+ [false, true].each { cache ->
+ sql "SET enable_inverted_index_query_cache = ${cache}"
+ // Repeating with cache enabled checks both its first fill and its warm read.
+ ["first", "repeat"].each { round ->
+ def prefix = "${mode}_${cache}_${round}"
+ def options = '{"mode":"' + mode + '","default_operator":"OR"}'
+ [
+ "a": "v.a:hit",
+ "b": "v.b:hit",
+ "or": "v.a:hit OR v.b:hit",
+ "and_not": "v.a:hit AND NOT v.b:hit",
+ "not_b": "NOT v.b:hit",
+ "missing": "v.missing:hit",
+ "not_missing": "NOT v.missing:hit",
+ "prefix_b": "v.b:h*"
+ ].each { label, dsl ->
+ quickTest("${prefix}_${label}", """
+ SELECT id FROM test_variant_search_missing_doc_path
+ WHERE SEARCH('${dsl}', '${options}') ORDER BY id
+ """)
+ }
+ quickTest("${prefix}_scored_or", """
+ SELECT id, s FROM (
+ SELECT id, score() AS s FROM test_variant_search_missing_doc_path
+ WHERE SEARCH('v.a:hit OR v.b:hit', '${options}')
+ ORDER BY s DESC LIMIT 100
+ ) scored ORDER BY id
+ """)
+ quickTest("${prefix}_external_not_b", """
+ SELECT id FROM test_variant_search_missing_doc_path
+ WHERE NOT SEARCH('v.b:hit', '${options}') ORDER BY id
+ """)
+ }
+ }
+ }
+
+ // This field exists only in doc storage because its row count is below the materialization
+ // threshold. It has no executable leaf index and must error instead of returning no matches.
+ sql "DROP TABLE IF EXISTS test_variant_search_unindexed_doc_path"
+ sql """
+ CREATE TABLE test_variant_search_unindexed_doc_path (
+ id INT,
+ v VARIANT<PROPERTIES(
+ "variant_enable_doc_mode"="true",
+ "variant_doc_materialization_min_rows"="1000"
+ )>,
+ INDEX idx_v(v) USING INVERTED PROPERTIES("parser"="unicode")
+ ) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES(
+ "replication_num"="1",
+ "inverted_index_storage_format"="V2"
+ )
+ """
+ sql """INSERT INTO test_variant_search_unindexed_doc_path VALUES
+ (1, parse_to_variant('{"present":"hit"}'))"""
+ ["v.present:hit", "NOT v.present:hit"].each { dsl ->
+ test {
+ sql "SELECT id FROM test_variant_search_unindexed_doc_path WHERE SEARCH('${dsl}')"
+ exception "SEARCH index unavailable for Variant field 'v.present'"
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org