You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2026/02/02 08:23:08 UTC
(doris) branch branch-4.0 updated: [feature](inverted index) support multiple tokenize index in one column (#60415)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 9512b4a24b9 [feature](inverted index) support multiple tokenize index in one column (#60415)
9512b4a24b9 is described below
commit 9512b4a24b9a1a9e208a84d001322f205d401427
Author: Jack <ji...@selectdb.com>
AuthorDate: Mon Feb 2 16:23:00 2026 +0800
[feature](inverted index) support multiple tokenize index in one column (#60415)
## Summary
Cherry-pick of #59117 to branch-4.0.
This PR adds support for multiple tokenize indexes on a single column,
allowing users to create multiple inverted indexes with different
analyzers (e.g., chinese, english, standard) on the same text column and
query using specific analyzers.
Key changes:
- Add `USING ANALYZER` syntax for MATCH predicates
- Support multiple inverted indexes with different analyzers on the same
column
- Add analyzer key normalization and matching logic in BE
- Add `filterIndexesByAnalyzer` method in OlapTable for index selection
## Conflicts Resolved
- `MatchPredicate.java` - Updated constructor signatures for analyzer
support
- `OlapTable.java` - Added imports and new filtering methods
- `ExpressionTranslator.java` - Updated to use new MatchPredicate
constructor
## Test plan
- [x] Regression tests included in original PR
- [x] Unit tests for analyzer key matcher and normalizer
## Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test
- Behavior changed:
- [ ] No.
- [x] Yes. Adds new `USING ANALYZER` syntax for MATCH predicates.
- Does this need documentation?
- [x] Yes. (covered by original PR documentation)
## Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.gitignore | 1 +
be/src/olap/inverted_index_parser.cpp | 171 +++++++---
be/src/olap/inverted_index_parser.h | 90 ++++--
.../rowset/segment_v2/analyzer_key_matcher.cpp | 61 ++++
.../olap/rowset/segment_v2/analyzer_key_matcher.h | 75 +++++
.../inverted_index/analyzer/analyzer.cpp | 14 +-
.../rowset/segment_v2/inverted_index_iterator.cpp | 234 +++++++++++---
.../rowset/segment_v2/inverted_index_iterator.h | 46 ++-
.../rowset/segment_v2/inverted_index_reader.cpp | 15 +-
be/src/runtime/index_policy/index_policy_mgr.cpp | 47 ++-
be/src/runtime/index_policy/index_policy_mgr.h | 5 +-
be/src/vec/exprs/vexpr.cpp | 1 +
be/src/vec/exprs/vexpr.h | 6 +
be/src/vec/exprs/vmatch_predicate.cpp | 26 +-
be/src/vec/exprs/vmatch_predicate.h | 1 +
be/src/vec/functions/function.h | 1 +
be/src/vec/functions/function_multi_match.cpp | 1 +
be/src/vec/functions/function_search.cpp | 7 +-
be/src/vec/functions/function_search.h | 1 +
be/src/vec/functions/function_tokenize.cpp | 4 +-
be/src/vec/functions/match.cpp | 47 ++-
be/test/olap/inverted_index_parser_test.cpp | 196 +++++++++++-
.../segment_v2/analyzer_key_matcher_test.cpp | 218 +++++++++++++
.../segment_v2/inverted_index_iterator_test.cpp | 344 +++++++++++++++++++++
be/test/vec/function/function_match_test.cpp | 4 +-
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 3 +-
.../apache/doris/alter/SchemaChangeHandler.java | 26 +-
.../apache/doris/analysis/AnalyzerSelector.java | 198 ++++++++++++
.../java/org/apache/doris/analysis/IndexDef.java | 7 +
.../apache/doris/analysis/InvertedIndexUtil.java | 106 ++++++-
.../org/apache/doris/analysis/MatchPredicate.java | 76 +++--
.../invertedindex/AnalyzerIdentityBuilder.java | 241 +++++++++++++++
.../invertedindex/AnalyzerKeyNormalizer.java | 44 +++
.../invertedindex/InvertedIndexSqlGenerator.java | 44 +++
.../main/java/org/apache/doris/catalog/Index.java | 7 +
.../java/org/apache/doris/catalog/OlapTable.java | 35 ++-
.../apache/doris/indexpolicy/IndexPolicyMgr.java | 85 +++--
.../glue/translator/ExpressionTranslator.java | 10 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 19 +-
.../doris/nereids/trees/expressions/Match.java | 69 ++++-
.../doris/nereids/trees/expressions/MatchAll.java | 16 +-
.../doris/nereids/trees/expressions/MatchAny.java | 16 +-
.../nereids/trees/expressions/MatchPhrase.java | 16 +-
.../nereids/trees/expressions/MatchPhraseEdge.java | 16 +-
.../trees/expressions/MatchPhrasePrefix.java | 16 +-
.../nereids/trees/expressions/MatchRegexp.java | 16 +-
.../trees/plans/commands/info/IndexDefinition.java | 7 +
.../doris/alter/SchemaChangeHandlerTest.java | 2 +-
.../invertedindex/AnalyzerIdentityBuilderTest.java | 104 +++++++
.../invertedindex/AnalyzerKeyNormalizerTest.java | 107 +++++++
.../InvertedIndexSqlGeneratorTest.java | 47 +++
.../test_match_using_analyzer.out | 13 +
.../test_multi_analyzer_column_types.out | 36 +++
.../inverted_index_p0/test_multi_analyzer_dml.out | 24 ++
.../test_multi_analyzer_index_version.out | 15 +
.../test_multi_analyzer_partition.out | 8 +
.../test_multi_analyzer_query.out | 40 +++
.../test_multi_analyzer_table_types.out | 15 +
.../test_multi_tokenize_index_not_built.out | 26 ++
.../test_match_using_analyzer.groovy | 164 ++++++++++
.../test_multi_analyzer_column_types.groovy | 97 ++++++
.../test_multi_analyzer_ddl.groovy | 209 +++++++++++++
.../test_multi_analyzer_dml.groovy | 75 +++++
.../test_multi_analyzer_index_version.groovy | 61 ++++
.../test_multi_analyzer_partition.groovy | 65 ++++
.../test_multi_analyzer_query.groovy | 83 +++++
.../test_multi_analyzer_show.groovy | 123 ++++++++
.../test_multi_analyzer_table_types.groovy | 45 +++
.../test_multi_tokenize_index_not_built.groovy | 151 +++++++++
69 files changed, 3924 insertions(+), 275 deletions(-)
diff --git a/.gitignore b/.gitignore
index c6cd57f5baf..b515321338c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -146,3 +146,4 @@ docker/runtime/be/resource/apache-doris/
compile_commands.json
.github
+.worktrees/
diff --git a/be/src/olap/inverted_index_parser.cpp b/be/src/olap/inverted_index_parser.cpp
index 7d44ee82e2c..779e11f9e65 100644
--- a/be/src/olap/inverted_index_parser.cpp
+++ b/be/src/olap/inverted_index_parser.cpp
@@ -82,37 +82,35 @@ std::string get_parser_string_from_properties(
std::string get_parser_mode_string_from_properties(
const std::map<std::string, std::string>& properties) {
- if (properties.find(INVERTED_INDEX_PARSER_MODE_KEY) != properties.end()) {
- return properties.at(INVERTED_INDEX_PARSER_MODE_KEY);
- } else {
- auto parser_it = properties.find(INVERTED_INDEX_PARSER_KEY);
- if (parser_it == properties.end()) {
- parser_it = properties.find(INVERTED_INDEX_PARSER_KEY_ALIAS);
- }
- if (parser_it != properties.end() && parser_it->second == INVERTED_INDEX_PARSER_IK) {
- return INVERTED_INDEX_PARSER_SMART;
- }
- return INVERTED_INDEX_PARSER_COARSE_GRANULARITY;
+ if (auto it = properties.find(INVERTED_INDEX_PARSER_MODE_KEY); it != properties.end()) {
+ return it->second;
}
+ auto parser_it = properties.find(INVERTED_INDEX_PARSER_KEY);
+ if (parser_it == properties.end()) {
+ parser_it = properties.find(INVERTED_INDEX_PARSER_KEY_ALIAS);
+ }
+ if (parser_it != properties.end() && parser_it->second == INVERTED_INDEX_PARSER_IK) {
+ return INVERTED_INDEX_PARSER_SMART;
+ }
+ return INVERTED_INDEX_PARSER_COARSE_GRANULARITY;
}
std::string get_parser_phrase_support_string_from_properties(
const std::map<std::string, std::string>& properties) {
- if (properties.find(INVERTED_INDEX_PARSER_PHRASE_SUPPORT_KEY) != properties.end()) {
- return properties.at(INVERTED_INDEX_PARSER_PHRASE_SUPPORT_KEY);
- } else {
- return INVERTED_INDEX_PARSER_PHRASE_SUPPORT_NO;
+ if (auto it = properties.find(INVERTED_INDEX_PARSER_PHRASE_SUPPORT_KEY);
+ it != properties.end()) {
+ return it->second;
}
+ return INVERTED_INDEX_PARSER_PHRASE_SUPPORT_NO;
}
CharFilterMap get_parser_char_filter_map_from_properties(
const std::map<std::string, std::string>& properties) {
- CharFilterMap char_filter_map;
-
- if (properties.find(INVERTED_INDEX_PARSER_CHAR_FILTER_TYPE) == properties.end()) {
- return CharFilterMap();
+ if (!properties.contains(INVERTED_INDEX_PARSER_CHAR_FILTER_TYPE)) {
+ return {};
}
+ CharFilterMap char_filter_map;
std::string type = properties.at(INVERTED_INDEX_PARSER_CHAR_FILTER_TYPE);
if (type == INVERTED_INDEX_CHAR_FILTER_CHAR_REPLACE) {
// type
@@ -120,20 +118,20 @@ CharFilterMap get_parser_char_filter_map_from_properties(
INVERTED_INDEX_CHAR_FILTER_CHAR_REPLACE;
// pattern
- if (properties.find(INVERTED_INDEX_PARSER_CHAR_FILTER_PATTERN) == properties.end()) {
- return CharFilterMap();
+ if (!properties.contains(INVERTED_INDEX_PARSER_CHAR_FILTER_PATTERN)) {
+ return {};
}
std::string pattern = properties.at(INVERTED_INDEX_PARSER_CHAR_FILTER_PATTERN);
char_filter_map[INVERTED_INDEX_PARSER_CHAR_FILTER_PATTERN] = pattern;
// placement
std::string replacement = " ";
- if (properties.find(INVERTED_INDEX_PARSER_CHAR_FILTER_REPLACEMENT) != properties.end()) {
+ if (properties.contains(INVERTED_INDEX_PARSER_CHAR_FILTER_REPLACEMENT)) {
replacement = properties.at(INVERTED_INDEX_PARSER_CHAR_FILTER_REPLACEMENT);
}
char_filter_map[INVERTED_INDEX_PARSER_CHAR_FILTER_REPLACEMENT] = replacement;
} else {
- return CharFilterMap();
+ return {};
}
return char_filter_map;
@@ -141,30 +139,28 @@ CharFilterMap get_parser_char_filter_map_from_properties(
std::string get_parser_ignore_above_value_from_properties(
const std::map<std::string, std::string>& properties) {
- if (properties.find(INVERTED_INDEX_PARSER_IGNORE_ABOVE_KEY) != properties.end()) {
- return properties.at(INVERTED_INDEX_PARSER_IGNORE_ABOVE_KEY);
- } else {
- return INVERTED_INDEX_PARSER_IGNORE_ABOVE_VALUE;
+ if (auto it = properties.find(INVERTED_INDEX_PARSER_IGNORE_ABOVE_KEY); it != properties.end()) {
+ return it->second;
}
+ return INVERTED_INDEX_PARSER_IGNORE_ABOVE_VALUE;
}
std::string get_parser_stopwords_from_properties(
const std::map<std::string, std::string>& properties) {
DBUG_EXECUTE_IF("inverted_index_parser.get_parser_stopwords_from_properties", { return ""; })
- if (properties.find(INVERTED_INDEX_PARSER_STOPWORDS_KEY) != properties.end()) {
- return properties.at(INVERTED_INDEX_PARSER_STOPWORDS_KEY);
- } else {
- return "";
+ if (auto it = properties.find(INVERTED_INDEX_PARSER_STOPWORDS_KEY); it != properties.end()) {
+ return it->second;
}
+ return "";
}
std::string get_parser_dict_compression_from_properties(
const std::map<std::string, std::string>& properties) {
- if (properties.find(INVERTED_INDEX_PARSER_DICT_COMPRESSION_KEY) != properties.end()) {
- return properties.at(INVERTED_INDEX_PARSER_DICT_COMPRESSION_KEY);
- } else {
- return "";
+ if (auto it = properties.find(INVERTED_INDEX_PARSER_DICT_COMPRESSION_KEY);
+ it != properties.end()) {
+ return it->second;
}
+ return "";
}
std::string get_analyzer_name_from_properties(
@@ -182,4 +178,109 @@ std::string get_analyzer_name_from_properties(
return "";
}
+std::string normalize_analyzer_key(std::string_view analyzer) {
+ // Simple normalization: lowercase, or empty if input is empty.
+ // Empty string means "user did not specify" - BE will auto-select.
+ // Non-empty string means "user specified this analyzer" - BE will exact match.
+ if (analyzer.empty()) {
+ return "";
+ }
+ return to_lower(std::string(analyzer));
+}
+
+std::string build_analyzer_key_from_properties(
+ const std::map<std::string, std::string>& properties) {
+ // Build analyzer key from index properties for reader registration.
+ // This determines how the index is stored/identified.
+
+ // 1. Check for custom analyzer name
+ auto custom_it = properties.find(INVERTED_INDEX_ANALYZER_NAME_KEY);
+ if (custom_it != properties.end() && !custom_it->second.empty()) {
+ return to_lower(custom_it->second);
+ }
+
+ // 2. Fall back to parser type
+ std::string parser;
+ auto parser_it = properties.find(INVERTED_INDEX_PARSER_KEY);
+ if (parser_it != properties.end()) {
+ parser = parser_it->second;
+ } else {
+ parser_it = properties.find(INVERTED_INDEX_PARSER_KEY_ALIAS);
+ if (parser_it != properties.end()) {
+ parser = parser_it->second;
+ }
+ }
+
+ // 3. Return normalized parser or "" for no explicit configuration
+ if (parser.empty()) {
+ return ""; // No explicit parser - empty key means "no configuration"
+ }
+ return to_lower(parser);
+}
+
+// ============================================================================
+// AnalyzerConfigParser implementation
+// ============================================================================
+
+std::string AnalyzerConfigParser::normalize_to_lower(const std::string& value) {
+ return to_lower(value);
+}
+
+bool AnalyzerConfigParser::is_builtin_analyzer(const std::string& normalized_name) {
+ if (normalized_name.empty()) {
+ return false;
+ }
+ auto parser_type = get_inverted_index_parser_type_from_string(normalized_name);
+ return parser_type != InvertedIndexParserType::PARSER_UNKNOWN;
+}
+
+std::string AnalyzerConfigParser::compute_analyzer_key(const std::string& value) {
+ // Simple: just lowercase, empty stays empty
+ return normalize_analyzer_key(value);
+}
+
+AnalyzerConfig AnalyzerConfigParser::parse(const std::string& analyzer_name,
+ const std::string& parser_type_str) {
+ AnalyzerConfig config;
+
+ // Determine parser type from parser_type_str (from index properties)
+ auto parser_type = get_inverted_index_parser_type_from_string(parser_type_str);
+ const std::string normalized_analyzer = normalize_to_lower(analyzer_name);
+
+ // If parser_type_str didn't yield a valid type, try analyzer_name
+ if (parser_type == InvertedIndexParserType::PARSER_UNKNOWN && !normalized_analyzer.empty()) {
+ parser_type = get_inverted_index_parser_type_from_string(normalized_analyzer);
+ }
+
+ const bool analyzer_is_builtin = is_builtin_analyzer(normalized_analyzer);
+
+ // Case 1: analyzer_name is non-empty and NOT a builtin type => custom analyzer
+ if (!analyzer_name.empty() && !analyzer_is_builtin) {
+ config.custom_analyzer = analyzer_name;
+ config.parser_type = InvertedIndexParserType::PARSER_NONE;
+ config.analyzer_key = normalize_to_lower(analyzer_name);
+ } else {
+ // Case 2: builtin analyzer or user did not specify analyzer
+ config.custom_analyzer.clear();
+
+ // Use parser_type from index properties for slow path tokenization
+ if (parser_type == InvertedIndexParserType::PARSER_UNKNOWN) {
+ config.parser_type = InvertedIndexParserType::PARSER_NONE;
+ } else {
+ config.parser_type = parser_type;
+ }
+
+ // analyzer_key: what user specified (for index selection)
+ // Empty means "user did not specify", BE will auto-select
+ if (normalized_analyzer.empty() && parser_type != InvertedIndexParserType::PARSER_UNKNOWN) {
+ // No analyzer name but valid parser type - use parser type as key
+ config.analyzer_key = inverted_index_parser_type_to_string(parser_type);
+ } else {
+ config.analyzer_key = normalized_analyzer;
+ }
+ }
+
+ return config;
+}
+
} // namespace doris
diff --git a/be/src/olap/inverted_index_parser.h b/be/src/olap/inverted_index_parser.h
index a7eba8f6418..d2d3df47abd 100644
--- a/be/src/olap/inverted_index_parser.h
+++ b/be/src/olap/inverted_index_parser.h
@@ -20,6 +20,7 @@
#include <map>
#include <memory>
#include <string>
+#include <string_view>
#include "util/debug_points.h"
@@ -56,24 +57,6 @@ struct InvertedIndexAnalyzerConfig {
CharFilterMap char_filter_map;
};
-// Runtime context for analyzer
-// Contains only the fields needed at runtime
-struct InvertedIndexAnalyzerCtx {
- // Used by execute_column path to determine if tokenization should be skipped
- std::string analyzer_name;
- InvertedIndexParserType parser_type = InvertedIndexParserType::PARSER_UNKNOWN;
-
- // Used for creating reader and tokenization
- CharFilterMap char_filter_map;
- lucene::analysis::Analyzer* analyzer = nullptr;
-
- // Helper method: returns true if tokenization should be performed
- bool should_tokenize() const {
- return !(parser_type == InvertedIndexParserType::PARSER_NONE && analyzer_name.empty());
- }
-};
-using InvertedIndexAnalyzerCtxSPtr = std::shared_ptr<InvertedIndexAnalyzerCtx>;
-
const std::string INVERTED_INDEX_PARSER_TRUE = "true";
const std::string INVERTED_INDEX_PARSER_FALSE = "false";
@@ -115,6 +98,35 @@ const std::string INVERTED_INDEX_PARSER_DICT_COMPRESSION_KEY = "dict_compression
const std::string INVERTED_INDEX_ANALYZER_NAME_KEY = "analyzer";
const std::string INVERTED_INDEX_NORMALIZER_NAME_KEY = "normalizer";
+const std::string INVERTED_INDEX_PARSER_FIELD_PATTERN_KEY = "field_pattern";
+
+// Normalize an analyzer name to a standardized key format (lowercase).
+// Empty string stays empty (means "user did not specify").
+// Non-empty string is lowercased (means "user specified this analyzer").
+std::string normalize_analyzer_key(std::string_view analyzer);
+
+// Runtime context for analyzer
+// Contains only the fields needed at runtime
+struct InvertedIndexAnalyzerCtx {
+ // analyzer_name: what user specified in USING ANALYZER clause
+ // Empty means user did not specify (BE auto-selects index)
+ // Non-empty means user explicitly specified (BE exact matches)
+ std::string analyzer_name;
+
+ // parser_type: determined from index properties, used for slow path tokenization
+ InvertedIndexParserType parser_type = InvertedIndexParserType::PARSER_UNKNOWN;
+
+ // Used for creating reader and tokenization
+ CharFilterMap char_filter_map;
+ std::shared_ptr<lucene::analysis::Analyzer> analyzer;
+
+ // Returns true if tokenization should be performed.
+ // Decision is based on parser_type (from index properties):
+ // - PARSER_NONE: no tokenization (keyword/exact match)
+ // - Other parsers: tokenize using that parser
+ bool should_tokenize() const { return parser_type != InvertedIndexParserType::PARSER_NONE; }
+};
+using InvertedIndexAnalyzerCtxSPtr = std::shared_ptr<InvertedIndexAnalyzerCtx>;
std::string inverted_index_parser_type_to_string(InvertedIndexParserType parser_type);
@@ -157,4 +169,46 @@ std::string get_parser_dict_compression_from_properties(
std::string get_analyzer_name_from_properties(const std::map<std::string, std::string>& properties);
+// Build a normalized analyzer key from index properties.
+// Checks custom_analyzer first, then falls back to parser type.
+std::string build_analyzer_key_from_properties(
+ const std::map<std::string, std::string>& properties);
+
+// Result structure for analyzer config parsing
+struct AnalyzerConfig {
+ std::string custom_analyzer;
+ InvertedIndexParserType parser_type = InvertedIndexParserType::PARSER_NONE;
+ // analyzer_key: what user specified in USING ANALYZER clause
+ // Empty means "user did not specify" (BE auto-selects)
+ // Non-empty means "user specified this analyzer" (BE exact matches)
+ std::string analyzer_key;
+
+ // Check if this is a custom analyzer (not builtin)
+ bool is_custom() const { return !custom_analyzer.empty(); }
+
+ // Check if user explicitly specified an analyzer
+ bool is_user_specified() const { return !analyzer_key.empty(); }
+};
+
+// Parser for analyzer configuration from Thrift TMatchPredicate.
+// Extracts analyzer_name and parser_type_str, determines if builtin or custom,
+// and produces a normalized AnalyzerConfig.
+class AnalyzerConfigParser {
+public:
+ // Parse from raw analyzer name and parser type string (extracted from Thrift).
+ // @param analyzer_name: User-specified analyzer name (may be custom or builtin, or empty).
+ // @param parser_type_str: Parser type string like "chinese", "standard", etc.
+ [[nodiscard]] static AnalyzerConfig parse(const std::string& analyzer_name,
+ const std::string& parser_type_str);
+
+ // Check if a normalized analyzer name looks like a builtin parser type
+ [[nodiscard]] static bool is_builtin_analyzer(const std::string& normalized_name);
+
+private:
+ static std::string normalize_to_lower(const std::string& value);
+
+ // Compute normalized analyzer_key from raw value.
+ static std::string compute_analyzer_key(const std::string& value);
+};
+
} // namespace doris
diff --git a/be/src/olap/rowset/segment_v2/analyzer_key_matcher.cpp b/be/src/olap/rowset/segment_v2/analyzer_key_matcher.cpp
new file mode 100644
index 00000000000..0423b475679
--- /dev/null
+++ b/be/src/olap/rowset/segment_v2/analyzer_key_matcher.cpp
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/analyzer_key_matcher.h"
+
+#include "olap/rowset/segment_v2/inverted_index_iterator.h"
+
+namespace doris::segment_v2 {
+
+AnalyzerMatchResult AnalyzerKeyMatcher::match(
+ std::string_view analyzer_key, const std::vector<ReaderEntry>& entries,
+ const std::unordered_map<std::string, std::vector<size_t>>& key_index) {
+ AnalyzerMatchResult result;
+
+ if (entries.empty()) {
+ return result;
+ }
+
+ // Step 1: If fallback is allowed (__default__ or empty), return all readers
+ // This allows query-type-based selection (e.g., FULLTEXT for MATCH queries,
+ // STRING_TYPE for EQUAL queries) rather than being constrained to a single index.
+ if (allows_fallback(analyzer_key)) {
+ result.candidates.reserve(entries.size());
+ for (const auto& entry : entries) {
+ result.candidates.push_back(&entry);
+ }
+ result.used_fallback = true;
+ return result;
+ }
+
+ // Step 2: Try exact match for explicit analyzer keys
+ std::string key_str(analyzer_key);
+ auto it = key_index.find(key_str);
+ if (it != key_index.end() && !it->second.empty()) {
+ result.candidates.reserve(it->second.size());
+ for (size_t idx : it->second) {
+ if (idx < entries.size()) {
+ result.candidates.push_back(&entries[idx]);
+ }
+ }
+ }
+
+ // Step 3: If explicit key with no match, return empty (caller will bypass)
+ return result;
+}
+
+} // namespace doris::segment_v2
diff --git a/be/src/olap/rowset/segment_v2/analyzer_key_matcher.h b/be/src/olap/rowset/segment_v2/analyzer_key_matcher.h
new file mode 100644
index 00000000000..6825b189683
--- /dev/null
+++ b/be/src/olap/rowset/segment_v2/analyzer_key_matcher.h
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <string_view>
+#include <unordered_map>
+#include <vector>
+
+#include "olap/inverted_index_parser.h"
+
+namespace doris::segment_v2 {
+
+// Forward declaration
+struct ReaderEntry;
+
+// Result of analyzer key matching operation.
+// Contains candidate readers that match the requested analyzer key.
+struct AnalyzerMatchResult {
+ // Pointers to matching reader entries. Valid only within the scope of
+ // the InvertedIndexIterator that owns the entries.
+ std::vector<const ReaderEntry*> candidates;
+
+ // True if fallback strategy was used (no exact match found)
+ bool used_fallback = false;
+
+ // Convenience methods
+ bool empty() const { return candidates.empty(); }
+ size_t size() const { return candidates.size(); }
+};
+
+// Utility class for matching analyzer keys against reader entries.
+// Stateless - all methods are static.
+//
+// Matching strategy:
+// 1. If key is empty (user did not specify), return all readers for query-type-based selection
+// 2. If key is non-empty (user specified), try exact match
+// 3. If explicit key with no match, return empty (caller decides whether to bypass)
+class AnalyzerKeyMatcher {
+public:
+ // Match analyzer key against reader entries.
+ // @param analyzer_key: Normalized analyzer key (lowercase, or empty for auto-select)
+ // @param entries: Available reader entries
+ // @param key_index: Index mapping analyzer_key -> entry indices
+ // @return MatchResult with candidates and fallback flag
+ static AnalyzerMatchResult match(
+ std::string_view analyzer_key, const std::vector<ReaderEntry>& entries,
+ const std::unordered_map<std::string, std::vector<size_t>>& key_index);
+
+ // Check if analyzer key is explicitly specified by user.
+ // Empty string means "user did not specify" (auto-select mode).
+ // Non-empty means "user wrote USING ANALYZER xxx".
+ static bool is_explicit(std::string_view key) { return !key.empty(); }
+
+ // Check if fallback (returning all readers) is allowed for this key.
+ // Only empty key (user did not specify) allows fallback.
+ static bool allows_fallback(std::string_view key) { return key.empty(); }
+};
+
+} // namespace doris::segment_v2
diff --git a/be/src/olap/rowset/segment_v2/inverted_index/analyzer/analyzer.cpp b/be/src/olap/rowset/segment_v2/inverted_index/analyzer/analyzer.cpp
index 24a114b7bd9..ec9d81f1503 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index/analyzer/analyzer.cpp
+++ b/be/src/olap/rowset/segment_v2/inverted_index/analyzer/analyzer.cpp
@@ -133,11 +133,15 @@ AnalyzerPtr InvertedIndexAnalyzer::create_builtin_analyzer(InvertedIndexParserTy
AnalyzerPtr InvertedIndexAnalyzer::create_analyzer(const InvertedIndexAnalyzerConfig* config) {
DCHECK(config != nullptr);
const std::string& analyzer_name = config->analyzer_name;
+
+ // Handle empty analyzer name - use builtin analyzer based on parser_type.
+ // This is the common case when user does not specify USING ANALYZER.
if (analyzer_name.empty()) {
return create_builtin_analyzer(config->parser_type, config->parser_mode, config->lower_case,
config->stop_words);
}
+ // Check if it's a builtin analyzer name (english, chinese, standard, etc.)
if (is_builtin_analyzer(analyzer_name)) {
InvertedIndexParserType parser_type =
get_inverted_index_parser_type_from_string(analyzer_name);
@@ -145,6 +149,7 @@ AnalyzerPtr InvertedIndexAnalyzer::create_analyzer(const InvertedIndexAnalyzerCo
config->stop_words);
}
+ // Custom analyzer - look up in policy manager
auto* index_policy_mgr = doris::ExecEnv::GetInstance()->index_policy_mgr();
if (!index_policy_mgr) {
throw Exception(ErrorCode::INVERTED_INDEX_ANALYZER_ERROR,
@@ -181,6 +186,13 @@ std::vector<TermInfo> InvertedIndexAnalyzer::get_analyse_result(
std::vector<TermInfo> InvertedIndexAnalyzer::get_analyse_result(
const std::string& search_str, const std::map<std::string, std::string>& properties) {
+ if (!should_analyzer(properties)) {
+ std::vector<TermInfo> result;
+ if (!search_str.empty()) {
+ result.emplace_back(search_str);
+ }
+ return result;
+ }
InvertedIndexAnalyzerConfig config;
config.analyzer_name = get_analyzer_name_from_properties(properties);
config.parser_type = get_inverted_index_parser_type_from_string(
@@ -210,4 +222,4 @@ bool InvertedIndexAnalyzer::should_analyzer(const std::map<std::string, std::str
}
} // namespace doris::segment_v2::inverted_index
-#include "common/compile_check_end.h"
\ No newline at end of file
+#include "common/compile_check_end.h"
diff --git a/be/src/olap/rowset/segment_v2/inverted_index_iterator.cpp b/be/src/olap/rowset/segment_v2/inverted_index_iterator.cpp
index ce3584b0161..4df1560183a 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/inverted_index_iterator.cpp
@@ -20,20 +20,47 @@
#include <memory>
#include "common/cast_set.h"
+#include "common/logging.h"
+#include "olap/inverted_index_parser.h"
#include "olap/rowset/segment_v2/inverted_index_cache.h"
#include "olap/rowset/segment_v2/inverted_index_reader.h"
+#include "olap/utils.h"
namespace doris::segment_v2 {
-InvertedIndexIterator::InvertedIndexIterator() {}
+InvertedIndexIterator::InvertedIndexIterator() = default;
+
+std::string InvertedIndexIterator::ensure_normalized_key(const std::string& analyzer_key) {
+ // Simple normalization: lowercase, empty stays empty.
+ // Empty means "user did not specify" (auto-select mode).
+ // Non-empty means "user specified this analyzer" (exact match mode).
+ return normalize_analyzer_key(analyzer_key);
+}
void InvertedIndexIterator::add_reader(InvertedIndexReaderType type,
const InvertedIndexReaderPtr& reader) {
- _readers[type] = reader;
+ // build_analyzer_key_from_properties already returns a normalized key,
+ // no need for additional normalization.
+ std::string analyzer_key = build_analyzer_key_from_properties(reader->get_index_properties());
+
+ VLOG_DEBUG << "InvertedIndexIterator add_reader: type=" << static_cast<int>(type)
+ << ", analyzer_key=" << analyzer_key;
+
+ const size_t entry_index = _reader_entries.size();
+ _reader_entries.push_back(
+ ReaderEntry {.type = type, .analyzer_key = std::move(analyzer_key), .reader = reader});
+
+ // Update index for O(1) lookup
+ _key_to_entries[_reader_entries.back().analyzer_key].push_back(entry_index);
}
Status InvertedIndexIterator::read_from_index(const IndexParam& param) {
- auto* i_param = std::get<InvertedIndexParam*>(param);
+ const auto* i_param_ptr = std::get_if<InvertedIndexParam*>(¶m);
+ if (i_param_ptr == nullptr) {
+ return Status::Error<ErrorCode::INDEX_INVALID_PARAMETERS>(
+ "param does not hold InvertedIndexParam*");
+ }
+ auto* i_param = *i_param_ptr;
if (i_param == nullptr) {
return Status::Error<ErrorCode::INDEX_INVALID_PARAMETERS>("i_param is null");
}
@@ -41,9 +68,16 @@ Status InvertedIndexIterator::read_from_index(const IndexParam& param) {
return Status::Error<ErrorCode::INVERTED_INDEX_BYPASS>("inverted index bypass");
});
- auto reader = DORIS_TRY(select_best_reader(i_param->column_type, i_param->query_type));
+ // analyzer_name from analyzer_ctx: what user specified in USING ANALYZER clause.
+ // Empty means "user did not specify" (BE auto-selects index).
+ // Non-empty means "user specified this analyzer" (BE exact matches).
+ const std::string& analyzer_name =
+ (i_param->analyzer_ctx != nullptr) ? i_param->analyzer_ctx->analyzer_name : "";
+ auto reader =
+ DORIS_TRY(select_best_reader(i_param->column_type, i_param->query_type, analyzer_name));
if (UNLIKELY(reader == nullptr)) {
- throw CLuceneError(CL_ERR_NullPointer, "bkd index reader is null", false);
+ return Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
+ "inverted index reader is null");
}
auto* runtime_state = _context->runtime_state;
if (!i_param->skip_try && reader->type() == InvertedIndexReaderType::BKD) {
@@ -71,7 +105,7 @@ Status InvertedIndexIterator::read_from_index(const IndexParam& param) {
i_param->query_type, i_param->roaring, i_param->analyzer_ctx);
};
- if (runtime_state->query_options().enable_profile) {
+ if (runtime_state != nullptr && runtime_state->query_options().enable_profile) {
InvertedIndexQueryStatistics query_stats;
{
SCOPED_RAW_TIMER(&query_stats.exec_time);
@@ -88,12 +122,14 @@ Status InvertedIndexIterator::read_from_index(const IndexParam& param) {
}
Status InvertedIndexIterator::read_null_bitmap(InvertedIndexQueryCacheHandle* cache_handle) {
- auto reader = DORIS_TRY(select_best_reader());
+ // For null bitmap, use any available reader (empty = auto-select)
+ auto reader = DORIS_TRY(select_best_reader(""));
return reader->read_null_bitmap(_context, cache_handle, nullptr);
}
Result<bool> InvertedIndexIterator::has_null() {
- auto reader = DORIS_TRY(select_best_reader());
+ // For has_null check, use any available reader (empty = auto-select)
+ auto reader = DORIS_TRY(select_best_reader(""));
return reader->has_null();
}
@@ -113,62 +149,170 @@ Status InvertedIndexIterator::try_read_from_inverted_index(const InvertedIndexRe
return Status::OK();
}
+Result<InvertedIndexReaderPtr> InvertedIndexIterator::select_for_text(
+ const AnalyzerMatchResult& match, InvertedIndexQueryType query_type,
+ const std::string& analyzer_key) {
+ // Bypass: explicit analyzer specified but not found
+ if (match.empty() && AnalyzerKeyMatcher::is_explicit(analyzer_key)) {
+ return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_BYPASS>(
+ "No inverted index reader found for analyzer '{}'. "
+ "The index for this analyzer may not be built yet.",
+ analyzer_key));
+ }
+
+ if (match.empty()) {
+ return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_NO_TERMS>(
+ "No available inverted index readers for text column."));
+ }
+
+ // MATCH queries prefer FULLTEXT
+ if (is_match_query(query_type)) {
+ for (const auto* entry : match.candidates) {
+ if (entry->type == InvertedIndexReaderType::FULLTEXT) {
+ return entry->reader;
+ }
+ }
+ }
+
+ // EQUAL/WILDCARD/REGEXP queries prefer STRING_TYPE
+ if (is_equal_query(query_type)) {
+ for (const auto* entry : match.candidates) {
+ if (entry->type == InvertedIndexReaderType::STRING_TYPE) {
+ return entry->reader;
+ }
+ }
+ }
+
+ // Default: return first candidate
+ return match.candidates.front()->reader;
+}
+
+Result<InvertedIndexReaderPtr> InvertedIndexIterator::select_for_numeric(
+ const AnalyzerMatchResult& match, InvertedIndexQueryType query_type) {
+ if (match.empty()) {
+ return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_NO_TERMS>(
+ "No available inverted index readers for numeric column."));
+ }
+
+ // RANGE queries prefer BKD
+ if (is_range_query(query_type)) {
+ for (const auto* entry : match.candidates) {
+ if (entry->type == InvertedIndexReaderType::BKD) {
+ return entry->reader;
+ }
+ }
+ }
+
+ // Fallback priority: BKD > STRING_TYPE > others
+ for (const auto* entry : match.candidates) {
+ if (entry->type == InvertedIndexReaderType::BKD) {
+ return entry->reader;
+ }
+ }
+ for (const auto* entry : match.candidates) {
+ if (entry->type == InvertedIndexReaderType::STRING_TYPE) {
+ return entry->reader;
+ }
+ }
+
+ // Last resort: first available
+ return match.candidates.front()->reader;
+}
+
Result<InvertedIndexReaderPtr> InvertedIndexIterator::select_best_reader(
- const vectorized::DataTypePtr& column_type, InvertedIndexQueryType query_type) {
- if (_readers.empty()) {
- return ResultError(Status::RuntimeError(
+ const vectorized::DataTypePtr& column_type, InvertedIndexQueryType query_type,
+ const std::string& analyzer_key) {
+ if (_reader_entries.empty()) {
+ return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_NO_TERMS>(
"No available inverted index readers. Check if index is properly initialized."));
}
- // BKD and array types allow only one reader each
- if (_readers.size() == 1) {
- return _readers.begin()->second;
+ // Normalize once at entry point
+ const std::string normalized_key = ensure_normalized_key(analyzer_key);
+
+ // Single reader optimization
+ if (_reader_entries.size() == 1) {
+ const auto& entry = _reader_entries.front();
+ if (AnalyzerKeyMatcher::is_explicit(normalized_key) &&
+ entry.analyzer_key != normalized_key) {
+ return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_BYPASS>(
+ "No inverted index reader found for analyzer '{}'. "
+ "Available analyzer: '{}'.",
+ normalized_key, entry.analyzer_key));
+ }
+ return entry.reader;
}
- // Check for string types
+ // Match analyzer key using AnalyzerKeyMatcher
+ auto match = AnalyzerKeyMatcher::match(normalized_key, _reader_entries, _key_to_entries);
+
+ // Dispatch by column type
const auto field_type = column_type->get_storage_field_type();
- const bool is_string = is_string_type(field_type);
-
- InvertedIndexReaderType preferred_type = InvertedIndexReaderType::UNKNOWN;
- // Handle string type columns
- if (is_string) {
- if (is_match_query(query_type)) {
- preferred_type = InvertedIndexReaderType::FULLTEXT;
- } else if (is_equal_query(query_type)) {
- preferred_type = InvertedIndexReaderType::STRING_TYPE;
- }
+
+ if (is_string_type(field_type)) {
+ return select_for_text(match, query_type, normalized_key);
}
- DBUG_EXECUTE_IF("inverted_index_reader.select_best_reader", {
- auto type = DebugPoints::instance()->get_debug_param_or_default<int32_t>(
- "inverted_index_reader.select_best_reader", "type", -1);
- if ((int32_t)preferred_type != type) {
- return ResultError(Status::RuntimeError(
- "Inverted index reader type mismatch. Expected={}, Actual={}",
- (int32_t)preferred_type, type));
- }
- })
- if (auto reader = get_reader(preferred_type)) {
- return std::static_pointer_cast<InvertedIndexReader>(reader);
+ if (is_numeric_type(field_type)) {
+ return select_for_numeric(match, query_type);
}
- return ResultError(Status::RuntimeError("Index query type not supported"));
+ // Default: return first candidate or error
+ if (match.empty()) {
+ return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_NO_TERMS>(
+ "No available inverted index readers for column type."));
+ }
+ return match.candidates.front()->reader;
}
-Result<InvertedIndexReaderPtr> InvertedIndexIterator::select_best_reader() {
- if (_readers.empty()) {
- return ResultError(Status::RuntimeError(
+Result<InvertedIndexReaderPtr> InvertedIndexIterator::select_best_reader(
+ const std::string& analyzer_key) {
+ if (_reader_entries.empty()) {
+ return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_NO_TERMS>(
"No available inverted index readers. Check if index is properly initialized."));
}
- return _readers.begin()->second;
+
+ const std::string normalized_key = ensure_normalized_key(analyzer_key);
+
+ // Single reader optimization
+ if (_reader_entries.size() == 1) {
+ const auto& entry = _reader_entries.front();
+ if (AnalyzerKeyMatcher::is_explicit(normalized_key) &&
+ entry.analyzer_key != normalized_key) {
+ return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_BYPASS>(
+ "No inverted index reader found for analyzer '{}'. "
+ "Available analyzer: '{}'.",
+ normalized_key, entry.analyzer_key));
+ }
+ return entry.reader;
+ }
+
+ // Match and return first candidate
+ auto match = AnalyzerKeyMatcher::match(normalized_key, _reader_entries, _key_to_entries);
+
+ if (match.empty()) {
+ if (AnalyzerKeyMatcher::is_explicit(normalized_key)) {
+ return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_BYPASS>(
+ "No inverted index reader found for analyzer '{}'.", normalized_key));
+ }
+ return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_NO_TERMS>(
+ "No available inverted index readers."));
+ }
+
+ return match.candidates.front()->reader;
}
IndexReaderPtr InvertedIndexIterator::get_reader(IndexReaderType type) const {
- auto iter = _readers.find(type);
- if (iter == _readers.end()) {
+ const auto* inverted_type = std::get_if<InvertedIndexReaderType>(&type);
+ if (inverted_type == nullptr) {
return nullptr;
}
- return iter->second;
+ for (const auto& entry : _reader_entries) {
+ if (entry.type == *inverted_type) {
+ return entry.reader;
+ }
+ }
+ return nullptr;
}
-} // namespace doris::segment_v2
\ No newline at end of file
+} // namespace doris::segment_v2
diff --git a/be/src/olap/rowset/segment_v2/inverted_index_iterator.h b/be/src/olap/rowset/segment_v2/inverted_index_iterator.h
index f2f72cc58e5..57ac76f1249 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_iterator.h
+++ b/be/src/olap/rowset/segment_v2/inverted_index_iterator.h
@@ -16,8 +16,11 @@
// under the License.
#pragma once
+#include <unordered_map>
+#include <vector>
#include "olap/inverted_index_parser.h"
+#include "olap/rowset/segment_v2/analyzer_key_matcher.h"
#include "olap/rowset/segment_v2/index_iterator.h"
#include "olap/rowset/segment_v2/inverted_index_reader.h"
@@ -31,12 +34,19 @@ struct InvertedIndexParam {
uint32_t num_rows;
std::shared_ptr<roaring::Roaring> roaring;
bool skip_try = false;
-
// Pointer to analyzer context (can be nullptr if not needed)
// Used by FullTextIndexReader for tokenization
const InvertedIndexAnalyzerCtx* analyzer_ctx = nullptr;
};
+// Entry representing an inverted index reader with its type and analyzer key.
+// Used by InvertedIndexIterator and AnalyzerKeyMatcher for reader selection.
+struct ReaderEntry {
+ InvertedIndexReaderType type;
+ std::string analyzer_key;
+ InvertedIndexReaderPtr reader;
+};
+
class InvertedIndexIterator : public IndexIterator {
public:
InvertedIndexIterator();
@@ -53,9 +63,11 @@ public:
IndexReaderPtr get_reader(IndexReaderType reader_type) const override;
- Result<InvertedIndexReaderPtr> select_best_reader(const vectorized::DataTypePtr& column_type,
- InvertedIndexQueryType query_type);
- Result<InvertedIndexReaderPtr> select_best_reader();
+ [[nodiscard]] Result<InvertedIndexReaderPtr> select_best_reader(
+ const vectorized::DataTypePtr& column_type, InvertedIndexQueryType query_type,
+ const std::string& analyzer_key);
+ [[nodiscard]] Result<InvertedIndexReaderPtr> select_best_reader(
+ const std::string& analyzer_key);
private:
ENABLE_FACTORY_CREATOR(InvertedIndexIterator);
@@ -64,7 +76,31 @@ private:
const std::string& column_name, const void* query_value,
InvertedIndexQueryType query_type, size_t* count);
- std::unordered_map<IndexReaderType, InvertedIndexReaderPtr> _readers;
+ // Normalize analyzer_key to lowercase.
+ // Empty input stays empty (means "user did not specify").
+ static std::string ensure_normalized_key(const std::string& analyzer_key);
+
+ // Select best reader for text (string) columns.
+ // Handles FULLTEXT vs STRING_TYPE priority based on query type.
+ // Returns BYPASS error if explicit analyzer not found.
+ [[nodiscard]] Result<InvertedIndexReaderPtr> select_for_text(const AnalyzerMatchResult& match,
+ InvertedIndexQueryType query_type,
+ const std::string& analyzer_key);
+
+ // Select best reader for numeric columns.
+ // Handles BKD priority for range queries.
+ [[nodiscard]] Result<InvertedIndexReaderPtr> select_for_numeric(
+ const AnalyzerMatchResult& match, InvertedIndexQueryType query_type);
+
+ // THREAD SAFETY: _reader_entries and _key_to_entries are populated during initialization
+ // phase (via add_reader) and only read during query phase (via read_from_index/select_best_reader).
+ // These two phases are guaranteed not to overlap, so no synchronization is needed.
+ // Do NOT call add_reader() after any read_from_index() call on the same iterator.
+ std::vector<ReaderEntry> _reader_entries;
+
+ // Index for O(1) lookup by analyzer_key. Maps normalized key to indices in _reader_entries.
+ // Built incrementally in add_reader().
+ std::unordered_map<std::string, std::vector<size_t>> _key_to_entries;
};
} // namespace doris::segment_v2
\ No newline at end of file
diff --git a/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp b/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
index e3a0264905b..f4da76e5c3f 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
@@ -314,13 +314,22 @@ Status FullTextIndexReader::query(const IndexQueryContextPtr& context,
query_info);
} else {
SCOPED_RAW_TIMER(&context->stats->inverted_index_analyzer_timer);
- if (analyzer_ctx != nullptr && analyzer_ctx->analyzer != nullptr) {
+ if (analyzer_ctx != nullptr && !analyzer_ctx->should_tokenize()) {
+ // Don't add empty string as token - empty query should match nothing
+ if (!search_str.empty()) {
+ query_info.term_infos.emplace_back(search_str);
+ }
+ } else if (analyzer_ctx != nullptr && analyzer_ctx->analyzer != nullptr) {
+ // Use analyzer from query context for consistent behavior across all segments.
+ // This ensures that the query uses the same analyzer settings (e.g., lowercase)
+ // regardless of how each segment's index was originally built.
auto reader = inverted_index::InvertedIndexAnalyzer::create_reader(
analyzer_ctx->char_filter_map);
reader->init(search_str.data(), static_cast<int32_t>(search_str.size()), true);
query_info.term_infos = inverted_index::InvertedIndexAnalyzer::get_analyse_result(
- reader, analyzer_ctx->analyzer);
+ reader, analyzer_ctx->analyzer.get());
} else {
+ // No analyzer context available, use index's own analyzer as fallback
query_info.term_infos = inverted_index::InvertedIndexAnalyzer::get_analyse_result(
search_str, _index_meta.properties());
}
@@ -1157,4 +1166,4 @@ template class InvertedIndexVisitor<InvertedIndexQueryType::GREATER_THAN_QUERY>;
template class InvertedIndexVisitor<InvertedIndexQueryType::GREATER_EQUAL_QUERY>;
} // namespace doris::segment_v2
-#include "common/compile_check_end.h"
\ No newline at end of file
+#include "common/compile_check_end.h"
diff --git a/be/src/runtime/index_policy/index_policy_mgr.cpp b/be/src/runtime/index_policy/index_policy_mgr.cpp
index eeb5576330b..ec007c9971a 100644
--- a/be/src/runtime/index_policy/index_policy_mgr.cpp
+++ b/be/src/runtime/index_policy/index_policy_mgr.cpp
@@ -27,6 +27,13 @@ namespace doris {
const std::unordered_set<std::string> IndexPolicyMgr::BUILTIN_NORMALIZERS = {"lowercase"};
+std::string IndexPolicyMgr::normalize_name(const std::string& name) {
+ std::string result = name;
+ boost::algorithm::trim(result);
+ boost::algorithm::to_lower(result);
+ return result;
+}
+
void IndexPolicyMgr::apply_policy_changes(const std::vector<TIndexPolicy>& policys_to_update,
const std::vector<int64_t>& policys_to_delete) {
LOG(INFO) << "Starting policy changes - "
@@ -43,7 +50,8 @@ void IndexPolicyMgr::apply_policy_changes(const std::vector<TIndexPolicy>& polic
<< "ID: " << id << ", "
<< "Name: " << it->second.name;
- _name_to_id.erase(it->second.name);
+ // Use normalized name for deletion
+ _name_to_id.erase(normalize_name(it->second.name));
_policys.erase(it);
success_deletes++;
} else {
@@ -59,15 +67,18 @@ void IndexPolicyMgr::apply_policy_changes(const std::vector<TIndexPolicy>& polic
continue;
}
- if (_name_to_id.contains(policy.name)) {
+ // Use normalized name for case-insensitive lookup
+ std::string normalized_name = normalize_name(policy.name);
+ if (_name_to_id.contains(normalized_name)) {
LOG(ERROR) << "Reject update - Duplicate policy name: " << policy.name
- << " | Existing ID: " << _name_to_id[policy.name]
+ << " | Existing ID: " << _name_to_id[normalized_name]
<< " | New ID: " << policy.id;
continue;
}
_policys.emplace(policy.id, policy);
- _name_to_id.emplace(policy.name, policy.id);
+ // Store with normalized key for case-insensitive lookup
+ _name_to_id.emplace(normalized_name, policy.id);
success_updates++;
LOG(INFO) << "Successfully applied policy - "
@@ -82,18 +93,24 @@ void IndexPolicyMgr::apply_policy_changes(const std::vector<TIndexPolicy>& polic
<< "Total policies: " << _policys.size();
}
-const Policys& IndexPolicyMgr::get_index_policys() {
+Policys IndexPolicyMgr::get_index_policys() {
std::shared_lock<std::shared_mutex> r_lock(_mutex);
- return _policys;
+ return _policys; // Return copy to ensure thread safety after lock release
}
-// TODO: Potential high-concurrency bottleneck
+// NOTE: This function holds a shared_lock while calling build_analyzer_from_policy/
+// build_normalizer_from_policy, which also access _name_to_id and _policys.
+// This is safe because std::shared_mutex allows the same thread to hold multiple
+// shared_locks (read locks are reentrant). The lock is held throughout to ensure
+// consistency when resolving nested policy references (e.g., tokenizer policies).
AnalyzerPtr IndexPolicyMgr::get_policy_by_name(const std::string& name) {
std::shared_lock lock(_mutex);
- auto name_it = _name_to_id.find(name);
+ // Use normalized name for case-insensitive lookup
+ std::string normalized_name = normalize_name(name);
+ auto name_it = _name_to_id.find(normalized_name);
if (name_it == _name_to_id.end()) {
- if (is_builtin_normalizer(name)) {
+ if (is_builtin_normalizer(normalized_name)) {
return build_builtin_normalizer(name);
}
throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with name: " + name);
@@ -125,8 +142,10 @@ AnalyzerPtr IndexPolicyMgr::build_analyzer_from_policy(const TIndexPolicy& index
}
const auto& tokenizer_name = tokenizer_it->second;
- if (_name_to_id.contains(tokenizer_name)) {
- const auto& tokenizer_policy = _policys[_name_to_id[tokenizer_name]];
+ // Use normalized name for case-insensitive lookup
+ std::string normalized_tokenizer_name = normalize_name(tokenizer_name);
+ if (_name_to_id.contains(normalized_tokenizer_name)) {
+ const auto& tokenizer_policy = _policys[_name_to_id[normalized_tokenizer_name]];
auto type_it = tokenizer_policy.properties.find(PROP_TYPE);
if (type_it == tokenizer_policy.properties.end()) {
throw Exception(ErrorCode::INVALID_ARGUMENT,
@@ -201,9 +220,11 @@ void IndexPolicyMgr::process_filter_configs(
continue;
}
- if (_name_to_id.contains(filter_name)) {
+ // Use normalized name for case-insensitive lookup
+ std::string normalized_filter_name = normalize_name(filter_name);
+ if (_name_to_id.contains(normalized_filter_name)) {
// Nested filter policy
- const auto& filter_policy = _policys[_name_to_id[filter_name]];
+ const auto& filter_policy = _policys[_name_to_id[normalized_filter_name]];
auto type_it = filter_policy.properties.find(PROP_TYPE);
if (type_it == filter_policy.properties.end()) {
throw Exception(
diff --git a/be/src/runtime/index_policy/index_policy_mgr.h b/be/src/runtime/index_policy/index_policy_mgr.h
index a30cec7bfdb..96207df343b 100644
--- a/be/src/runtime/index_policy/index_policy_mgr.h
+++ b/be/src/runtime/index_policy/index_policy_mgr.h
@@ -38,7 +38,7 @@ public:
void apply_policy_changes(const std::vector<TIndexPolicy>& policies_to_update,
const std::vector<int64_t>& policies_to_delete);
- const Policys& get_index_policys();
+ Policys get_index_policys();
AnalyzerPtr get_policy_by_name(const std::string& name);
private:
@@ -54,6 +54,9 @@ private:
bool is_builtin_normalizer(const std::string& name);
AnalyzerPtr build_builtin_normalizer(const std::string& name);
+ // Normalize policy name to lowercase for case-insensitive lookup
+ static std::string normalize_name(const std::string& name);
+
constexpr static auto PROP_TOKENIZER = "tokenizer";
constexpr static auto PROP_CHAR_FILTER = "char_filter";
constexpr static auto PROP_TOKEN_FILTER = "token_filter";
diff --git a/be/src/vec/exprs/vexpr.cpp b/be/src/vec/exprs/vexpr.cpp
index 5cd9079bb91..c26f7e0357f 100644
--- a/be/src/vec/exprs/vexpr.cpp
+++ b/be/src/vec/exprs/vexpr.cpp
@@ -914,6 +914,7 @@ Status VExpr::_evaluate_inverted_index(VExprContext* context, const FunctionBase
}
auto result_bitmap = segment_v2::InvertedIndexResultBitmap();
+ // Pass analyzer_key to function (used by match predicates for multi-analyzer index selection)
auto res = function->evaluate_inverted_index(arguments, data_type_with_names, iterators,
segment_num_rows, analyzer_ctx, result_bitmap);
if (!res.ok()) {
diff --git a/be/src/vec/exprs/vexpr.h b/be/src/vec/exprs/vexpr.h
index 452150d5b8a..5e5980de8db 100644
--- a/be/src/vec/exprs/vexpr.h
+++ b/be/src/vec/exprs/vexpr.h
@@ -165,6 +165,12 @@ public:
return Status::OK();
}
+ // Get analyzer key for inverted index queries (overridden by VMatchPredicate)
+ [[nodiscard]] virtual const std::string& get_analyzer_key() const {
+ static const std::string empty;
+ return empty;
+ }
+
Status _evaluate_inverted_index(VExprContext* context, const FunctionBasePtr& function,
uint32_t segment_num_rows);
diff --git a/be/src/vec/exprs/vmatch_predicate.cpp b/be/src/vec/exprs/vmatch_predicate.cpp
index a02704c1b84..a15b791484e 100644
--- a/be/src/vec/exprs/vmatch_predicate.cpp
+++ b/be/src/vec/exprs/vmatch_predicate.cpp
@@ -29,10 +29,10 @@
#include <gen_cpp/Exprs_types.h>
#include <glog/logging.h>
-#include <cstddef>
#include <memory>
-#include <ostream>
+#include <string>
#include <string_view>
+#include <type_traits>
#include <vector>
#include "common/status.h"
@@ -44,6 +44,7 @@
#include "vec/core/column_with_type_and_name.h"
#include "vec/exprs/vexpr_context.h"
#include "vec/exprs/vslot_ref.h"
+#include "vec/functions/match.h"
#include "vec/functions/simple_function_factory.h"
namespace doris {
@@ -73,7 +74,10 @@ VMatchPredicate::VMatchPredicate(const TExprNode& node) : VExpr(node) {
{ config.lower_case = ""; })
config.stop_words = node.match_predicate.parser_stopwords;
- // Step 2: Use config to create analyzer (factory method)
+ // Step 2: Use config to create analyzer (factory method).
+ // Always create analyzer based on parser_type for slow path (tables without index).
+ // For index path, FullTextIndexReader will check analyzer_name to decide whether
+ // to use this analyzer or fallback to index's own analyzer.
_analyzer = inverted_index::InvertedIndexAnalyzer::create_analyzer(&config);
// Step 3: Create runtime context (only extract runtime-needed info)
@@ -81,7 +85,7 @@ VMatchPredicate::VMatchPredicate(const TExprNode& node) : VExpr(node) {
_analyzer_ctx->analyzer_name = config.analyzer_name;
_analyzer_ctx->parser_type = config.parser_type;
_analyzer_ctx->char_filter_map = std::move(config.char_filter_map);
- _analyzer_ctx->analyzer = _analyzer.get();
+ _analyzer_ctx->analyzer = _analyzer;
}
VMatchPredicate::~VMatchPredicate() = default;
@@ -93,7 +97,7 @@ Status VMatchPredicate::prepare(RuntimeState* state, const RowDescriptor& desc,
ColumnsWithTypeAndName argument_template;
argument_template.reserve(_children.size());
std::vector<std::string_view> child_expr_name;
- for (auto child : _children) {
+ for (const auto& child : _children) {
argument_template.emplace_back(nullptr, child->data_type(), child->expr_name());
child_expr_name.emplace_back(child->expr_name());
}
@@ -102,7 +106,7 @@ Status VMatchPredicate::prepare(RuntimeState* state, const RowDescriptor& desc,
argument_template, _data_type, {});
if (_function == nullptr) {
std::string type_str;
- for (auto arg : argument_template) {
+ for (const auto& arg : argument_template) {
type_str = type_str + " " + arg.type->get_name();
}
return Status::NotSupported(
@@ -148,6 +152,10 @@ Status VMatchPredicate::evaluate_inverted_index(VExprContext* context, uint32_t
return _evaluate_inverted_index(context, _function, segment_num_rows);
}
+const std::string& VMatchPredicate::get_analyzer_key() const {
+ return _analyzer_ctx->analyzer_name;
+}
+
Status VMatchPredicate::execute_column(VExprContext* context, const Block* block, size_t count,
ColumnPtr& result_column) const {
DCHECK(_open_finished || block == nullptr);
@@ -167,7 +175,7 @@ Status VMatchPredicate::execute_column(VExprContext* context, const Block* block
auto* column_slot_ref = assert_cast<VSlotRef*>(get_child(0).get());
std::string column_name = column_slot_ref->expr_name();
- auto it = std::find(column_names.begin(), column_names.end(), column_name);
+ auto it = std::ranges::find(column_names, column_name);
if (it == column_names.end()) {
return Status::Error<ErrorCode::INTERNAL_ERROR>(
"column {} should in slow path while VMatchPredicate::execute.", column_name);
@@ -175,12 +183,12 @@ Status VMatchPredicate::execute_column(VExprContext* context, const Block* block
})
ColumnNumbers arguments(_children.size());
Block temp_block;
- for (int i = 0; i < _children.size(); ++i) {
+ for (size_t i = 0; i < _children.size(); ++i) {
ColumnPtr arg_column;
RETURN_IF_ERROR(_children[i]->execute_column(context, block, count, arg_column));
auto arg_type = _children[i]->execute_type(block);
temp_block.insert({arg_column, arg_type, _children[i]->expr_name()});
- arguments[i] = i;
+ arguments[i] = static_cast<uint32_t>(i);
}
uint32_t num_columns_without_result = temp_block.columns();
// prepare a column to save result
diff --git a/be/src/vec/exprs/vmatch_predicate.h b/be/src/vec/exprs/vmatch_predicate.h
index 1a3368261c8..29e07a89697 100644
--- a/be/src/vec/exprs/vmatch_predicate.h
+++ b/be/src/vec/exprs/vmatch_predicate.h
@@ -58,6 +58,7 @@ public:
Status evaluate_inverted_index(VExprContext* context, uint32_t segment_num_rows) override;
const std::string& expr_name() const override;
const std::string& function_name() const;
+ [[nodiscard]] const std::string& get_analyzer_key() const override;
std::string debug_string() const override;
diff --git a/be/src/vec/functions/function.h b/be/src/vec/functions/function.h
index ca872a147a7..5a38216827c 100644
--- a/be/src/vec/functions/function.h
+++ b/be/src/vec/functions/function.h
@@ -31,6 +31,7 @@
#include "common/exception.h"
#include "common/logging.h"
#include "common/status.h"
+#include "olap/inverted_index_parser.h"
#include "olap/rowset/segment_v2/inverted_index_iterator.h" // IWYU pragma: keep
#include "runtime/define_primitive_type.h"
#include "vec/core/block.h"
diff --git a/be/src/vec/functions/function_multi_match.cpp b/be/src/vec/functions/function_multi_match.cpp
index 55368eff4bc..676ab446635 100644
--- a/be/src/vec/functions/function_multi_match.cpp
+++ b/be/src/vec/functions/function_multi_match.cpp
@@ -95,6 +95,7 @@ Status FunctionMultiMatch::evaluate_inverted_index(
}
param.column_name = column_name;
+ param.column_type = data_type_with_names[i].second;
param.roaring = std::make_shared<roaring::Roaring>();
param.analyzer_ctx = analyzer_ctx;
RETURN_IF_ERROR(iter->read_from_index(segment_v2::IndexParam {¶m}));
diff --git a/be/src/vec/functions/function_search.cpp b/be/src/vec/functions/function_search.cpp
index 4a4a397e8d8..6f737146915 100644
--- a/be/src/vec/functions/function_search.cpp
+++ b/be/src/vec/functions/function_search.cpp
@@ -32,6 +32,7 @@
#include "common/status.h"
#include "gen_cpp/Exprs_types.h"
+#include "olap/inverted_index_parser.h"
#include "olap/rowset/segment_v2/index_file_reader.h"
#include "olap/rowset/segment_v2/index_query_context.h"
#include "olap/rowset/segment_v2/inverted_index/analyzer/analyzer.h"
@@ -108,9 +109,9 @@ Status FieldReaderResolver::resolve(const std::string& field_name,
Result<InvertedIndexReaderPtr> reader_result;
const auto& column_type = data_it->second.second;
if (column_type) {
- reader_result = inverted_iterator->select_best_reader(column_type, query_type);
+ reader_result = inverted_iterator->select_best_reader(column_type, query_type, "");
} else {
- reader_result = inverted_iterator->select_best_reader();
+ reader_result = inverted_iterator->select_best_reader("");
}
if (!reader_result.has_value()) {
@@ -167,6 +168,8 @@ Status FieldReaderResolver::resolve(const std::string& field_name,
resolved.lucene_reader = reader_holder;
resolved.index_properties = inverted_reader->get_index_properties();
resolved.binding_key = binding_key;
+ resolved.analyzer_key =
+ normalize_analyzer_key(build_analyzer_key_from_properties(resolved.index_properties));
_binding_readers[binding_key] = reader_holder;
_field_readers[resolved.stored_field_wstr] = reader_holder;
diff --git a/be/src/vec/functions/function_search.h b/be/src/vec/functions/function_search.h
index 4b2f1fa2532..944f07dd1b6 100644
--- a/be/src/vec/functions/function_search.h
+++ b/be/src/vec/functions/function_search.h
@@ -49,6 +49,7 @@ struct FieldReaderBinding {
std::shared_ptr<lucene::index::IndexReader> lucene_reader;
std::map<std::string, std::string> index_properties;
std::string binding_key;
+ std::string analyzer_key;
};
class FieldReaderResolver {
diff --git a/be/src/vec/functions/function_tokenize.cpp b/be/src/vec/functions/function_tokenize.cpp
index 5f0c05e9844..1551e4a6021 100644
--- a/be/src/vec/functions/function_tokenize.cpp
+++ b/be/src/vec/functions/function_tokenize.cpp
@@ -107,7 +107,7 @@ void FunctionTokenize::_do_tokenize(const ColumnString& src_column_string,
auto reader = InvertedIndexAnalyzer::create_reader(analyzer_ctx.char_filter_map);
reader->init(tokenize_str.data, (int)tokenize_str.size, true);
auto analyzer_tokens =
- InvertedIndexAnalyzer::get_analyse_result(reader, analyzer_ctx.analyzer);
+ InvertedIndexAnalyzer::get_analyse_result(reader, analyzer_ctx.analyzer.get());
rapidjson::Document doc;
doc.SetArray();
@@ -194,7 +194,7 @@ Status FunctionTokenize::execute_impl(FunctionContext* /*context*/, Block& block
analyzer_ctx.analyzer_name = config.analyzer_name;
analyzer_ctx.parser_type = config.parser_type;
analyzer_ctx.char_filter_map = config.char_filter_map;
- analyzer_ctx.analyzer = analyzer_holder.get();
+ analyzer_ctx.analyzer = analyzer_holder;
_do_tokenize(*col_left, analyzer_ctx, support_phrase, dest_column_ptr);
block.replace_by_position(result, std::move(dest_column_ptr));
diff --git a/be/src/vec/functions/match.cpp b/be/src/vec/functions/match.cpp
index 2350570341d..c3a2ec9fce9 100644
--- a/be/src/vec/functions/match.cpp
+++ b/be/src/vec/functions/match.cpp
@@ -92,7 +92,13 @@ Status FunctionMatchBase::evaluate_inverted_index(
param.num_rows = num_rows;
param.roaring = std::make_shared<roaring::Roaring>();
param.analyzer_ctx = analyzer_ctx;
- RETURN_IF_ERROR(iter->read_from_index(segment_v2::IndexParam {¶m}));
+ if (is_string_type(param_type)) {
+ RETURN_IF_ERROR(iter->read_from_index(¶m));
+ } else {
+ return Status::Error<ErrorCode::INDEX_INVALID_PARAMETERS>(
+ "invalid params type for FunctionMatchBase::evaluate_inverted_index {}",
+ param_type);
+ }
std::shared_ptr<roaring::Roaring> null_bitmap = std::make_shared<roaring::Roaring>();
if (iter->has_null()) {
segment_v2::InvertedIndexQueryCacheHandle null_bitmap_cache_handle;
@@ -121,7 +127,6 @@ Status FunctionMatchBase::execute_impl(FunctionContext* context, Block& block,
VLOG_DEBUG << "begin to execute match directly, column_name=" << column_name
<< ", match_query_str=" << match_query_str;
auto* analyzer_ctx = get_match_analyzer_ctx(context);
-
const ColumnPtr source_col =
block.get_by_position(arguments[0]).column->convert_to_full_column_if_const();
const auto* values = check_and_get_column<ColumnString>(source_col.get());
@@ -192,17 +197,37 @@ std::vector<TermInfo> FunctionMatchBase::analyse_query_str_token(
if (analyzer_ctx == nullptr) {
return query_tokens;
}
+
VLOG_DEBUG << "begin to run " << get_name() << ", parser_type: "
<< inverted_index_parser_type_to_string(analyzer_ctx->parser_type);
+
+ // Decision is based on parser_type (from index properties):
+ // - PARSER_NONE: no tokenization (keyword/exact match)
+ // - Other parsers: tokenize using the analyzer
if (!analyzer_ctx->should_tokenize()) {
- query_tokens.emplace_back(match_query_str);
+ // Keyword index or no tokenization needed
+ // Don't add empty string as token - empty query should match nothing
+ if (!match_query_str.empty()) {
+ query_tokens.emplace_back(match_query_str);
+ }
return query_tokens;
}
+
+ // Safety check: if analyzer is nullptr but tokenization is expected, fall back to no tokenization
+ if (analyzer_ctx->analyzer == nullptr) {
+ VLOG_DEBUG << "Analyzer is nullptr, falling back to no tokenization";
+ if (!match_query_str.empty()) {
+ query_tokens.emplace_back(match_query_str);
+ }
+ return query_tokens;
+ }
+
+ // Tokenize using the analyzer
auto reader = doris::segment_v2::inverted_index::InvertedIndexAnalyzer::create_reader(
analyzer_ctx->char_filter_map);
reader->init(match_query_str.data(), (int)match_query_str.size(), true);
query_tokens = doris::segment_v2::inverted_index::InvertedIndexAnalyzer::get_analyse_result(
- reader, analyzer_ctx->analyzer);
+ reader, analyzer_ctx->analyzer.get());
return query_tokens;
}
@@ -214,25 +239,29 @@ inline std::vector<TermInfo> FunctionMatchBase::analyse_data_token(
if (analyzer_ctx == nullptr) {
return data_tokens;
}
+
+ // Determine tokenization strategy based on parser_type
+ const bool should_tokenize =
+ analyzer_ctx->should_tokenize() && analyzer_ctx->analyzer != nullptr;
+
if (array_offsets) {
for (auto next_src_array_offset = (*array_offsets)[current_block_row_idx];
current_src_array_offset < next_src_array_offset; ++current_src_array_offset) {
const auto& str_ref = string_col->get_data_at(current_src_array_offset);
- if (!analyzer_ctx->should_tokenize()) {
+ if (!should_tokenize) {
data_tokens.emplace_back(str_ref.to_string());
continue;
}
auto reader = doris::segment_v2::inverted_index::InvertedIndexAnalyzer::create_reader(
analyzer_ctx->char_filter_map);
reader->init(str_ref.data, (int)str_ref.size, true);
-
data_tokens =
doris::segment_v2::inverted_index::InvertedIndexAnalyzer::get_analyse_result(
- reader, analyzer_ctx->analyzer);
+ reader, analyzer_ctx->analyzer.get());
}
} else {
const auto& str_ref = string_col->get_data_at(current_block_row_idx);
- if (!analyzer_ctx->should_tokenize()) {
+ if (!should_tokenize) {
data_tokens.emplace_back(str_ref.to_string());
} else {
auto reader = doris::segment_v2::inverted_index::InvertedIndexAnalyzer::create_reader(
@@ -240,7 +269,7 @@ inline std::vector<TermInfo> FunctionMatchBase::analyse_data_token(
reader->init(str_ref.data, (int)str_ref.size, true);
data_tokens =
doris::segment_v2::inverted_index::InvertedIndexAnalyzer::get_analyse_result(
- reader, analyzer_ctx->analyzer);
+ reader, analyzer_ctx->analyzer.get());
}
}
return data_tokens;
diff --git a/be/test/olap/inverted_index_parser_test.cpp b/be/test/olap/inverted_index_parser_test.cpp
index eeb9b262dab..7224cdebfe2 100644
--- a/be/test/olap/inverted_index_parser_test.cpp
+++ b/be/test/olap/inverted_index_parser_test.cpp
@@ -265,16 +265,27 @@ TEST_F(InvertedIndexParserTest, TestGetAnalyzerNameFromProperties) {
TEST_F(InvertedIndexParserTest, TestInvertedIndexAnalyzerCtxShouldTokenize) {
InvertedIndexAnalyzerCtx ctx;
+
+ // New design: should_tokenize() only depends on parser_type
+ // PARSER_NONE means no tokenization (keyword index)
ctx.parser_type = InvertedIndexParserType::PARSER_NONE;
ctx.analyzer_name.clear();
EXPECT_FALSE(ctx.should_tokenize());
+ // Any parser other than NONE means tokenization
ctx.parser_type = InvertedIndexParserType::PARSER_ENGLISH;
EXPECT_TRUE(ctx.should_tokenize());
+ ctx.parser_type = InvertedIndexParserType::PARSER_CHINESE;
+ EXPECT_TRUE(ctx.should_tokenize());
+
+ ctx.parser_type = InvertedIndexParserType::PARSER_STANDARD;
+ EXPECT_TRUE(ctx.should_tokenize());
+
+ // Even with custom_analyzer name, PARSER_NONE means no tokenization
ctx.parser_type = InvertedIndexParserType::PARSER_NONE;
ctx.analyzer_name = "custom_analyzer";
- EXPECT_TRUE(ctx.should_tokenize());
+ EXPECT_FALSE(ctx.should_tokenize());
}
// Test constants
@@ -318,6 +329,189 @@ TEST_F(InvertedIndexParserTest, TestConstants) {
EXPECT_EQ(INVERTED_INDEX_PARSER_CHAR_FILTER_TYPE, "char_filter_type");
EXPECT_EQ(INVERTED_INDEX_PARSER_CHAR_FILTER_PATTERN, "char_filter_pattern");
EXPECT_EQ(INVERTED_INDEX_PARSER_CHAR_FILTER_REPLACEMENT, "char_filter_replacement");
+
+ // Note: INVERTED_INDEX_DEFAULT_ANALYZER_KEY has been removed in the new design.
+ // Empty string now means "user did not specify" (BE auto-selects index).
+}
+
+// ============================================================================
+// normalize_analyzer_key Tests
+// New design: empty string stays empty, non-empty gets lowercased
+// ============================================================================
+
+TEST_F(InvertedIndexParserTest, NormalizeAnalyzerKey_EmptyInput) {
+ // New design: empty string stays empty (means "user did not specify")
+ EXPECT_EQ(normalize_analyzer_key(""), "");
+}
+
+TEST_F(InvertedIndexParserTest, NormalizeAnalyzerKey_UppercaseToLowercase) {
+ EXPECT_EQ(normalize_analyzer_key("CHINESE"), "chinese");
+ EXPECT_EQ(normalize_analyzer_key("STANDARD"), "standard");
+ EXPECT_EQ(normalize_analyzer_key("ENGLISH"), "english");
+}
+
+TEST_F(InvertedIndexParserTest, NormalizeAnalyzerKey_MixedCase) {
+ EXPECT_EQ(normalize_analyzer_key("ChInEsE"), "chinese");
+ EXPECT_EQ(normalize_analyzer_key("StAnDaRd"), "standard");
+ EXPECT_EQ(normalize_analyzer_key("My_Custom_Analyzer"), "my_custom_analyzer");
+}
+
+TEST_F(InvertedIndexParserTest, NormalizeAnalyzerKey_NoneParser) {
+ // "none" is a distinct key - means keyword index (no tokenization)
+ EXPECT_EQ(normalize_analyzer_key("none"), "none");
+ EXPECT_EQ(normalize_analyzer_key("NONE"), "none");
+}
+
+TEST_F(InvertedIndexParserTest, NormalizeAnalyzerKey_AlreadyLowercase) {
+ EXPECT_EQ(normalize_analyzer_key("chinese"), "chinese");
+ EXPECT_EQ(normalize_analyzer_key("my_custom"), "my_custom");
+}
+
+// ============================================================================
+// build_analyzer_key_from_properties Tests
+// New design: returns actual parser/analyzer name, empty means no properties
+// ============================================================================
+
+TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_EmptyProperties) {
+ std::map<std::string, std::string> properties;
+ // Empty properties = empty key (no explicit configuration)
+ EXPECT_EQ(build_analyzer_key_from_properties(properties), "");
+}
+
+TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_CustomAnalyzer) {
+ std::map<std::string, std::string> properties;
+ properties[INVERTED_INDEX_ANALYZER_NAME_KEY] = "my_custom_analyzer";
+ EXPECT_EQ(build_analyzer_key_from_properties(properties), "my_custom_analyzer");
+}
+
+TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_CustomAnalyzerUppercase) {
+ std::map<std::string, std::string> properties;
+ properties[INVERTED_INDEX_ANALYZER_NAME_KEY] = "MY_CUSTOM";
+ EXPECT_EQ(build_analyzer_key_from_properties(properties), "my_custom");
+}
+
+TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_ParserKey) {
+ std::map<std::string, std::string> properties;
+ properties[INVERTED_INDEX_PARSER_KEY] = "chinese";
+ EXPECT_EQ(build_analyzer_key_from_properties(properties), "chinese");
+}
+
+TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_ParserKeyAlias) {
+ std::map<std::string, std::string> properties;
+ properties[INVERTED_INDEX_PARSER_KEY_ALIAS] = "standard";
+ EXPECT_EQ(build_analyzer_key_from_properties(properties), "standard");
+}
+
+TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_ParserNone) {
+ std::map<std::string, std::string> properties;
+ properties[INVERTED_INDEX_PARSER_KEY] = "none";
+ // "none" is a distinct analyzer key - it means no tokenization (keyword analyzer)
+ // This is different from __default__ which means use default behavior
+ EXPECT_EQ(build_analyzer_key_from_properties(properties), "none");
+}
+
+TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_CustomOverridesParser) {
+ std::map<std::string, std::string> properties;
+ properties[INVERTED_INDEX_ANALYZER_NAME_KEY] = "my_custom";
+ properties[INVERTED_INDEX_PARSER_KEY] = "chinese";
+ // Custom analyzer takes precedence
+ EXPECT_EQ(build_analyzer_key_from_properties(properties), "my_custom");
+}
+
+// ============================================================================
+// AnalyzerConfigParser Tests
+// ============================================================================
+
+TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_EmptyInput) {
+ auto config = AnalyzerConfigParser::parse("", "");
+ // New design: empty input gives empty analyzer_key (means "user did not specify")
+ EXPECT_EQ(config.analyzer_key, "");
+ EXPECT_EQ(config.parser_type, InvertedIndexParserType::PARSER_NONE);
+ EXPECT_TRUE(config.custom_analyzer.empty());
+ EXPECT_FALSE(config.is_custom());
+}
+
+TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_OnlyAnalyzerCustom) {
+ auto config = AnalyzerConfigParser::parse("my_custom_analyzer", "");
+ EXPECT_EQ(config.custom_analyzer, "my_custom_analyzer");
+ EXPECT_EQ(config.analyzer_key, "my_custom_analyzer");
+ EXPECT_EQ(config.parser_type, InvertedIndexParserType::PARSER_NONE);
+ EXPECT_TRUE(config.is_custom());
+}
+
+TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_OnlyAnalyzerBuiltin) {
+ auto config = AnalyzerConfigParser::parse("chinese", "");
+ EXPECT_TRUE(config.custom_analyzer.empty());
+ EXPECT_EQ(config.analyzer_key, "chinese");
+ EXPECT_EQ(config.parser_type, InvertedIndexParserType::PARSER_CHINESE);
+ EXPECT_FALSE(config.is_custom());
+}
+
+TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_OnlyParserTypeStr) {
+ auto config = AnalyzerConfigParser::parse("", "standard");
+ EXPECT_TRUE(config.custom_analyzer.empty());
+ EXPECT_EQ(config.analyzer_key, "standard");
+ EXPECT_EQ(config.parser_type, InvertedIndexParserType::PARSER_STANDARD);
+ EXPECT_FALSE(config.is_custom());
+}
+
+TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_BothAnalyzerAndParser) {
+ // parser_type_str takes precedence for determining parser_type
+ auto config = AnalyzerConfigParser::parse("ik", "chinese");
+ EXPECT_TRUE(config.custom_analyzer.empty());
+ EXPECT_EQ(config.parser_type, InvertedIndexParserType::PARSER_CHINESE);
+ EXPECT_EQ(config.analyzer_key, "ik"); // analyzer_name used for key
+}
+
+TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_CaseInsensitive) {
+ auto config = AnalyzerConfigParser::parse("CHINESE", "");
+ EXPECT_TRUE(config.custom_analyzer.empty());
+ EXPECT_EQ(config.analyzer_key, "chinese");
+ EXPECT_EQ(config.parser_type, InvertedIndexParserType::PARSER_CHINESE);
+}
+
+TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_UnknownAnalyzerAsCustom) {
+ auto config = AnalyzerConfigParser::parse("unknown_xyz", "");
+ EXPECT_EQ(config.custom_analyzer, "unknown_xyz");
+ EXPECT_EQ(config.analyzer_key, "unknown_xyz");
+ EXPECT_EQ(config.parser_type, InvertedIndexParserType::PARSER_NONE);
+ EXPECT_TRUE(config.is_custom());
+}
+
+TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_AllBuiltinTypes) {
+ // Test all builtin parser types
+ std::vector<std::pair<std::string, InvertedIndexParserType>> builtin_types = {
+ {"none", InvertedIndexParserType::PARSER_NONE},
+ {"standard", InvertedIndexParserType::PARSER_STANDARD},
+ {"unicode", InvertedIndexParserType::PARSER_UNICODE},
+ {"english", InvertedIndexParserType::PARSER_ENGLISH},
+ {"chinese", InvertedIndexParserType::PARSER_CHINESE},
+ {"icu", InvertedIndexParserType::PARSER_ICU},
+ {"basic", InvertedIndexParserType::PARSER_BASIC},
+ {"ik", InvertedIndexParserType::PARSER_IK},
+ };
+
+ for (const auto& [name, expected_type] : builtin_types) {
+ auto config = AnalyzerConfigParser::parse(name, "");
+ EXPECT_EQ(config.parser_type, expected_type) << "Failed for: " << name;
+ EXPECT_TRUE(config.custom_analyzer.empty()) << "Failed for: " << name;
+ EXPECT_FALSE(config.is_custom()) << "Failed for: " << name;
+ }
+}
+
+TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_IsBuiltinAnalyzer) {
+ EXPECT_TRUE(AnalyzerConfigParser::is_builtin_analyzer("chinese"));
+ EXPECT_TRUE(AnalyzerConfigParser::is_builtin_analyzer("standard"));
+ EXPECT_TRUE(AnalyzerConfigParser::is_builtin_analyzer("english"));
+ EXPECT_TRUE(AnalyzerConfigParser::is_builtin_analyzer("unicode"));
+ EXPECT_TRUE(AnalyzerConfigParser::is_builtin_analyzer("icu"));
+ EXPECT_TRUE(AnalyzerConfigParser::is_builtin_analyzer("basic"));
+ EXPECT_TRUE(AnalyzerConfigParser::is_builtin_analyzer("ik"));
+ EXPECT_TRUE(AnalyzerConfigParser::is_builtin_analyzer("none"));
+
+ EXPECT_FALSE(AnalyzerConfigParser::is_builtin_analyzer("my_custom"));
+ EXPECT_FALSE(AnalyzerConfigParser::is_builtin_analyzer("unknown"));
+ EXPECT_FALSE(AnalyzerConfigParser::is_builtin_analyzer(""));
}
} // namespace doris
diff --git a/be/test/olap/rowset/segment_v2/analyzer_key_matcher_test.cpp b/be/test/olap/rowset/segment_v2/analyzer_key_matcher_test.cpp
new file mode 100644
index 00000000000..379c7a6cdbb
--- /dev/null
+++ b/be/test/olap/rowset/segment_v2/analyzer_key_matcher_test.cpp
@@ -0,0 +1,218 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/analyzer_key_matcher.h"
+
+#include <gtest/gtest.h>
+
+#include "olap/rowset/segment_v2/inverted_index_iterator.h"
+
+namespace doris::segment_v2 {
+
+// New design:
+// - Empty string "" means "user did not specify" (BE auto-selects index)
+// - Non-empty string means "user specified this analyzer" (exact match)
+// - "__default__" is no longer special - it's just another explicit key
+
+class AnalyzerKeyMatcherTest : public testing::Test {
+protected:
+ void SetUp() override {
+ // Create test entries
+ entries_.push_back({InvertedIndexReaderType::FULLTEXT, "chinese", nullptr});
+ entries_.push_back({InvertedIndexReaderType::FULLTEXT, "english", nullptr});
+ entries_.push_back({InvertedIndexReaderType::STRING_TYPE, "", nullptr}); // empty key reader
+
+ // Build index
+ for (size_t i = 0; i < entries_.size(); ++i) {
+ key_index_[entries_[i].analyzer_key].push_back(i);
+ }
+ }
+
+ std::vector<ReaderEntry> entries_;
+ std::unordered_map<std::string, std::vector<size_t>> key_index_;
+};
+
+// Test exact match
+TEST_F(AnalyzerKeyMatcherTest, ExactMatch) {
+ auto result = AnalyzerKeyMatcher::match("chinese", entries_, key_index_);
+
+ EXPECT_FALSE(result.empty());
+ EXPECT_EQ(result.size(), 1);
+ EXPECT_EQ(result.candidates[0]->analyzer_key, "chinese");
+ EXPECT_FALSE(result.used_fallback);
+}
+
+TEST_F(AnalyzerKeyMatcherTest, ExactMatchEnglish) {
+ auto result = AnalyzerKeyMatcher::match("english", entries_, key_index_);
+
+ EXPECT_FALSE(result.empty());
+ EXPECT_EQ(result.size(), 1);
+ EXPECT_EQ(result.candidates[0]->analyzer_key, "english");
+ EXPECT_FALSE(result.used_fallback);
+}
+
+// Test empty string returns all readers for query-type-based selection
+TEST_F(AnalyzerKeyMatcherTest, EmptyKeyReturnsAll) {
+ auto result = AnalyzerKeyMatcher::match("", entries_, key_index_);
+
+ // Empty string means "no specific analyzer requested", so return all readers
+ // to allow query-type-based selection (e.g., FULLTEXT for MATCH, STRING_TYPE for EQUAL)
+ EXPECT_FALSE(result.empty());
+ EXPECT_EQ(result.size(), 3);
+ EXPECT_TRUE(result.used_fallback);
+}
+
+// Test explicit key with no match returns empty
+TEST_F(AnalyzerKeyMatcherTest, ExplicitKeyNoMatchReturnsEmpty) {
+ auto result = AnalyzerKeyMatcher::match("japanese", entries_, key_index_);
+
+ EXPECT_TRUE(result.empty());
+ EXPECT_FALSE(result.used_fallback);
+}
+
+// Test is_explicit
+TEST_F(AnalyzerKeyMatcherTest, IsExplicit) {
+ EXPECT_TRUE(AnalyzerKeyMatcher::is_explicit("chinese"));
+ EXPECT_TRUE(AnalyzerKeyMatcher::is_explicit("english"));
+ EXPECT_TRUE(AnalyzerKeyMatcher::is_explicit("none")); // "none" is explicit
+ EXPECT_TRUE(AnalyzerKeyMatcher::is_explicit(
+ "__default__")); // "__default__" is explicit (just another key)
+ EXPECT_FALSE(AnalyzerKeyMatcher::is_explicit("")); // Only empty is not explicit
+}
+
+// Test allows_fallback
+TEST_F(AnalyzerKeyMatcherTest, AllowsFallback) {
+ EXPECT_TRUE(AnalyzerKeyMatcher::allows_fallback("")); // Only empty allows fallback
+ EXPECT_FALSE(AnalyzerKeyMatcher::allows_fallback("chinese"));
+ EXPECT_FALSE(AnalyzerKeyMatcher::allows_fallback("english"));
+ EXPECT_FALSE(AnalyzerKeyMatcher::allows_fallback("none"));
+ EXPECT_FALSE(AnalyzerKeyMatcher::allows_fallback("__default__")); // Not special anymore
+}
+
+// Test empty entries
+TEST_F(AnalyzerKeyMatcherTest, EmptyEntries) {
+ std::vector<ReaderEntry> empty_entries;
+ std::unordered_map<std::string, std::vector<size_t>> empty_index;
+
+ auto result = AnalyzerKeyMatcher::match("chinese", empty_entries, empty_index);
+ EXPECT_TRUE(result.empty());
+}
+
+// Test multiple entries with same analyzer key
+class AnalyzerKeyMatcherMultiEntryTest : public testing::Test {
+protected:
+ void SetUp() override {
+ // Multiple readers with same analyzer key (different types)
+ entries_.push_back({InvertedIndexReaderType::FULLTEXT, "chinese", nullptr});
+ entries_.push_back({InvertedIndexReaderType::STRING_TYPE, "chinese", nullptr});
+
+ for (size_t i = 0; i < entries_.size(); ++i) {
+ key_index_[entries_[i].analyzer_key].push_back(i);
+ }
+ }
+
+ std::vector<ReaderEntry> entries_;
+ std::unordered_map<std::string, std::vector<size_t>> key_index_;
+};
+
+TEST_F(AnalyzerKeyMatcherMultiEntryTest, MatchReturnsAllWithSameKey) {
+ auto result = AnalyzerKeyMatcher::match("chinese", entries_, key_index_);
+
+ EXPECT_FALSE(result.empty());
+ EXPECT_EQ(result.size(), 2);
+ EXPECT_FALSE(result.used_fallback);
+
+ // Both should be "chinese"
+ for (const auto* entry : result.candidates) {
+ EXPECT_EQ(entry->analyzer_key, "chinese");
+ }
+}
+
+// Test empty key fallback when no exact match
+class AnalyzerKeyMatcherFallbackTest : public testing::Test {
+protected:
+ void SetUp() override {
+ // Only specific entries (no empty key)
+ entries_.push_back({InvertedIndexReaderType::FULLTEXT, "chinese", nullptr});
+ entries_.push_back({InvertedIndexReaderType::FULLTEXT, "english", nullptr});
+
+ for (size_t i = 0; i < entries_.size(); ++i) {
+ key_index_[entries_[i].analyzer_key].push_back(i);
+ }
+ }
+
+ std::vector<ReaderEntry> entries_;
+ std::unordered_map<std::string, std::vector<size_t>> key_index_;
+};
+
+TEST_F(AnalyzerKeyMatcherFallbackTest, EmptyKeyFallbacksToAll) {
+ auto result = AnalyzerKeyMatcher::match("", entries_, key_index_);
+
+ // When empty string is requested, fallback to all readers
+ EXPECT_FALSE(result.empty());
+ EXPECT_EQ(result.size(), 2);
+ EXPECT_TRUE(result.used_fallback);
+}
+
+TEST_F(AnalyzerKeyMatcherFallbackTest, ExplicitKeyNoFallback) {
+ auto result = AnalyzerKeyMatcher::match("japanese", entries_, key_index_);
+
+ // Explicit key (not empty) should not fallback
+ EXPECT_TRUE(result.empty());
+ EXPECT_FALSE(result.used_fallback);
+}
+
+// Test "none" analyzer is an explicit key (no tokenization index)
+class AnalyzerKeyMatcherNoneAnalyzerTest : public testing::Test {
+protected:
+ void SetUp() override {
+ // Entry with empty key (default/no properties) and entry with "none" key
+ entries_.push_back({InvertedIndexReaderType::STRING_TYPE, "", nullptr});
+ entries_.push_back(
+ {InvertedIndexReaderType::STRING_TYPE, INVERTED_INDEX_PARSER_NONE, nullptr});
+
+ for (size_t i = 0; i < entries_.size(); ++i) {
+ key_index_[entries_[i].analyzer_key].push_back(i);
+ }
+ }
+
+ std::vector<ReaderEntry> entries_;
+ std::unordered_map<std::string, std::vector<size_t>> key_index_;
+};
+
+TEST_F(AnalyzerKeyMatcherNoneAnalyzerTest, NoneIsDistinctFromEmpty) {
+ // "none" is an explicit analyzer key, should do exact match
+ auto result_none = AnalyzerKeyMatcher::match(INVERTED_INDEX_PARSER_NONE, entries_, key_index_);
+ EXPECT_FALSE(result_none.empty());
+ EXPECT_EQ(result_none.size(), 1);
+ EXPECT_EQ(result_none.candidates[0]->analyzer_key, INVERTED_INDEX_PARSER_NONE);
+ EXPECT_FALSE(result_none.used_fallback);
+
+ // Empty string means "no specific analyzer", should return all readers
+ auto result_empty = AnalyzerKeyMatcher::match("", entries_, key_index_);
+ EXPECT_FALSE(result_empty.empty());
+ EXPECT_EQ(result_empty.size(), 2); // Should return both entries
+ EXPECT_TRUE(result_empty.used_fallback);
+}
+
+TEST_F(AnalyzerKeyMatcherNoneAnalyzerTest, NoneIsExplicit) {
+ // "none" is treated as an explicit analyzer key
+ EXPECT_TRUE(AnalyzerKeyMatcher::is_explicit(INVERTED_INDEX_PARSER_NONE));
+ EXPECT_FALSE(AnalyzerKeyMatcher::allows_fallback(INVERTED_INDEX_PARSER_NONE));
+}
+
+} // namespace doris::segment_v2
diff --git a/be/test/olap/rowset/segment_v2/inverted_index_iterator_test.cpp b/be/test/olap/rowset/segment_v2/inverted_index_iterator_test.cpp
new file mode 100644
index 00000000000..71e29d89407
--- /dev/null
+++ b/be/test/olap/rowset/segment_v2/inverted_index_iterator_test.cpp
@@ -0,0 +1,344 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/inverted_index_iterator.h"
+
+#include <gtest/gtest.h>
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "olap/inverted_index_parser.h"
+#include "olap/rowset/segment_v2/inverted_index_reader.h"
+#include "olap/tablet_schema.h"
+#include "vec/data_types/data_type_string.h"
+
+namespace doris::segment_v2 {
+
+// Mock InvertedIndexReader for testing
+class MockInvertedIndexReader : public InvertedIndexReader {
+public:
+ // Factory method to create instances
+ static std::shared_ptr<MockInvertedIndexReader> create(
+ const std::map<std::string, std::string>& properties) {
+ auto index = std::make_shared<TabletIndex>();
+ // Initialize TabletIndex with protobuf to ensure all fields are properly set
+ TabletIndexPB pb;
+ pb.set_index_id(1);
+ pb.set_index_name("test_index");
+ pb.set_index_type(IndexType::INVERTED);
+ index->init_from_pb(pb);
+ return std::shared_ptr<MockInvertedIndexReader>(
+ new MockInvertedIndexReader(index, properties));
+ }
+
+ InvertedIndexReaderType type() override { return _type; }
+ void set_type(InvertedIndexReaderType type) { _type = type; }
+ const std::map<std::string, std::string>& get_index_properties() const override {
+ return _properties;
+ }
+
+ Status query(const IndexQueryContextPtr& context, const std::string& column_name,
+ const void* query_value, InvertedIndexQueryType query_type,
+ std::shared_ptr<roaring::Roaring>& roaring,
+ const InvertedIndexAnalyzerCtx* analyzer_ctx = nullptr) override {
+ return Status::OK();
+ }
+
+ Status try_query(const IndexQueryContextPtr& context, const std::string& column_name,
+ const void* query_value, InvertedIndexQueryType query_type,
+ size_t* count) override {
+ *count = 0;
+ return Status::OK();
+ }
+
+ Status new_iterator(std::unique_ptr<IndexIterator>* iterator) override { return Status::OK(); }
+
+private:
+ MockInvertedIndexReader(std::shared_ptr<TabletIndex> index,
+ const std::map<std::string, std::string>& properties)
+ : InvertedIndexReader(index.get(), nullptr),
+ _mock_index(index), // Copy shared_ptr to keep index alive
+ _properties(properties) {}
+
+ std::shared_ptr<TabletIndex> _mock_index; // Keep index alive
+ std::map<std::string, std::string> _properties;
+ InvertedIndexReaderType _type = InvertedIndexReaderType::FULLTEXT;
+};
+
+class InvertedIndexIteratorTest : public testing::Test {
+protected:
+ std::shared_ptr<MockInvertedIndexReader> create_mock_reader(
+ const std::string& analyzer_key,
+ InvertedIndexReaderType type = InvertedIndexReaderType::FULLTEXT) {
+ std::map<std::string, std::string> properties;
+ // New design: empty string means "user did not specify", non-empty means explicit.
+ // We only set properties when analyzer_key is non-empty.
+ if (!analyzer_key.empty()) {
+ if (AnalyzerConfigParser::is_builtin_analyzer(analyzer_key)) {
+ properties[INVERTED_INDEX_PARSER_KEY] = analyzer_key;
+ } else {
+ properties[INVERTED_INDEX_ANALYZER_NAME_KEY] = analyzer_key;
+ }
+ }
+ auto reader = MockInvertedIndexReader::create(properties);
+ reader->set_type(type);
+ return reader;
+ }
+};
+
+// ensure_normalized_key tests
+TEST_F(InvertedIndexIteratorTest, EnsureNormalizedKey_EmptyInput) {
+ // New design: empty string stays empty (means "user did not specify")
+ EXPECT_EQ(InvertedIndexIterator::ensure_normalized_key(""), "");
+}
+
+TEST_F(InvertedIndexIteratorTest, EnsureNormalizedKey_Uppercase) {
+ EXPECT_EQ(InvertedIndexIterator::ensure_normalized_key("CHINESE"), "chinese");
+}
+
+TEST_F(InvertedIndexIteratorTest, EnsureNormalizedKey_MixedCase) {
+ EXPECT_EQ(InvertedIndexIterator::ensure_normalized_key("ChInEsE"), "chinese");
+}
+
+TEST_F(InvertedIndexIteratorTest, EnsureNormalizedKey_NonEmptyString) {
+ // Non-empty strings are normalized to lowercase
+ EXPECT_EQ(InvertedIndexIterator::ensure_normalized_key("__default__"), "__default__");
+ EXPECT_EQ(InvertedIndexIterator::ensure_normalized_key("NONE"), "none");
+}
+
+// add_reader tests
+TEST_F(InvertedIndexIteratorTest, AddReader_SingleReader) {
+ InvertedIndexIterator iterator;
+ try {
+ auto reader = create_mock_reader("chinese");
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, reader);
+
+ auto result = iterator.select_best_reader("chinese");
+ EXPECT_TRUE(result.has_value());
+ EXPECT_EQ(result.value(), reader);
+ } catch (const std::exception& e) {
+ EXPECT_TRUE(false) << "Exception thrown: " << e.what();
+ }
+}
+
+TEST_F(InvertedIndexIteratorTest, AddReader_MultipleReadersWithDifferentKeys) {
+ InvertedIndexIterator iterator;
+ auto reader1 = create_mock_reader("chinese");
+ auto reader2 = create_mock_reader("english");
+ auto reader3 = create_mock_reader(""); // empty key stays empty (no properties set)
+
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, reader1);
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, reader2);
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, reader3);
+
+ // Explicit analyzer keys (chinese, english) do exact match
+ auto result1 = iterator.select_best_reader("chinese");
+ EXPECT_TRUE(result1.has_value());
+ EXPECT_EQ(result1.value(), reader1);
+
+ auto result2 = iterator.select_best_reader("english");
+ EXPECT_TRUE(result2.has_value());
+ EXPECT_EQ(result2.value(), reader2);
+
+ // Empty string = fallback mode (user did not specify).
+ // The simple select_best_reader(key) overload returns the first candidate.
+ // This is by design: empty means "no preference, let system choose".
+ auto result3 = iterator.select_best_reader("");
+ EXPECT_TRUE(result3.has_value());
+ // Don't assert specific reader - fallback mode returns first available
+}
+
+// Test that "none" is treated as a distinct analyzer key (not empty string)
+TEST_F(InvertedIndexIteratorTest, AddReader_NoneAnalyzerIsDistinct) {
+ InvertedIndexIterator iterator;
+ auto empty_reader = create_mock_reader(""); // empty key (no properties)
+ auto none_reader = create_mock_reader("none"); // explicit "none" key
+
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, empty_reader);
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, none_reader);
+
+ // Query with empty string = fallback mode, returns first available
+ auto result_empty = iterator.select_best_reader("");
+ EXPECT_TRUE(result_empty.has_value());
+ // Don't assert which reader - fallback mode returns first available
+
+ // Query for "none" should return none_reader (exact match)
+ auto result_none = iterator.select_best_reader(INVERTED_INDEX_PARSER_NONE);
+ EXPECT_TRUE(result_none.has_value());
+ EXPECT_EQ(result_none.value(), none_reader);
+}
+
+// find_reader_candidates tests (via select_best_reader)
+TEST_F(InvertedIndexIteratorTest, FindReaderCandidates_ExactMatch) {
+ InvertedIndexIterator iterator;
+ auto reader = create_mock_reader("chinese");
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, reader);
+
+ auto result = iterator.select_best_reader("chinese");
+ EXPECT_TRUE(result.has_value());
+ EXPECT_EQ(result.value(), reader);
+}
+
+TEST_F(InvertedIndexIteratorTest, FindReaderCandidates_FallbackWithEmptyKey) {
+ InvertedIndexIterator iterator;
+ auto empty_reader = create_mock_reader("");
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, empty_reader);
+
+ // Empty string = fallback mode, returns available reader
+ auto result = iterator.select_best_reader("");
+ EXPECT_TRUE(result.has_value());
+ EXPECT_EQ(result.value(), empty_reader);
+}
+
+TEST_F(InvertedIndexIteratorTest, FindReaderCandidates_FallbackToAny) {
+ InvertedIndexIterator iterator;
+ auto chinese_reader = create_mock_reader("chinese");
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, chinese_reader);
+
+ // Empty string = fallback mode, returns any available reader
+ auto result = iterator.select_best_reader("");
+ EXPECT_TRUE(result.has_value());
+ EXPECT_EQ(result.value(), chinese_reader);
+}
+
+TEST_F(InvertedIndexIteratorTest, FindReaderCandidates_EmptyReaders) {
+ InvertedIndexIterator iterator;
+ auto result = iterator.select_best_reader("chinese");
+ EXPECT_FALSE(result.has_value());
+}
+
+// Test: explicit analyzer that doesn't exist returns error
+TEST_F(InvertedIndexIteratorTest, ExplicitAnalyzer_NotFound_ReturnsBypass) {
+ InvertedIndexIterator iterator;
+ auto chinese_reader = create_mock_reader("chinese");
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, chinese_reader);
+
+ // Query for "english" (explicit) but only "chinese" exists
+ auto result = iterator.select_best_reader("english");
+ EXPECT_FALSE(result.has_value());
+ // Error should be INVERTED_INDEX_BYPASS
+}
+
+// Test: empty string (user did not specify) returns any available reader
+TEST_F(InvertedIndexIteratorTest, EmptyString_NoSpecifiedAnalyzer_ReturnsAny) {
+ InvertedIndexIterator iterator;
+ auto chinese_reader = create_mock_reader("chinese");
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, chinese_reader);
+
+ // Empty = fallback mode, should return the chinese_reader
+ auto result = iterator.select_best_reader("");
+ EXPECT_TRUE(result.has_value());
+ EXPECT_EQ(result.value(), chinese_reader);
+}
+
+// select_best_reader with column_type tests
+TEST_F(InvertedIndexIteratorTest, SelectBestReader_MatchQuerySelectsFulltext) {
+ InvertedIndexIterator iterator;
+ auto fulltext_reader = create_mock_reader("chinese", InvertedIndexReaderType::FULLTEXT);
+ auto string_reader = create_mock_reader("chinese", InvertedIndexReaderType::STRING_TYPE);
+
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, fulltext_reader);
+ iterator.add_reader(InvertedIndexReaderType::STRING_TYPE, string_reader);
+
+ auto string_type = std::make_shared<vectorized::DataTypeString>();
+ auto result = iterator.select_best_reader(string_type, InvertedIndexQueryType::MATCH_ANY_QUERY,
+ "chinese");
+ EXPECT_TRUE(result.has_value());
+ EXPECT_EQ(result.value(), fulltext_reader);
+}
+
+TEST_F(InvertedIndexIteratorTest, SelectBestReader_EqualQuerySelectsStringType) {
+ InvertedIndexIterator iterator;
+ auto fulltext_reader = create_mock_reader("chinese", InvertedIndexReaderType::FULLTEXT);
+ auto string_reader = create_mock_reader("chinese", InvertedIndexReaderType::STRING_TYPE);
+
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, fulltext_reader);
+ iterator.add_reader(InvertedIndexReaderType::STRING_TYPE, string_reader);
+
+ auto string_type = std::make_shared<vectorized::DataTypeString>();
+ auto result = iterator.select_best_reader(string_type, InvertedIndexQueryType::EQUAL_QUERY,
+ "chinese");
+ EXPECT_TRUE(result.has_value());
+ EXPECT_EQ(result.value(), string_reader);
+}
+
+// Index lookup performance test
+TEST_F(InvertedIndexIteratorTest, IndexLookup_ManyReadersStillFast) {
+ InvertedIndexIterator iterator;
+
+ std::vector<std::shared_ptr<MockInvertedIndexReader>> readers;
+ for (int i = 0; i < 100; i++) {
+ auto reader = create_mock_reader("analyzer_" + std::to_string(i));
+ readers.push_back(reader);
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, reader);
+ }
+
+ auto result = iterator.select_best_reader("analyzer_50");
+ EXPECT_TRUE(result.has_value());
+ EXPECT_EQ(result.value(), readers[50]);
+
+ auto result_first = iterator.select_best_reader("analyzer_0");
+ EXPECT_TRUE(result_first.has_value());
+ EXPECT_EQ(result_first.value(), readers[0]);
+
+ auto result_last = iterator.select_best_reader("analyzer_99");
+ EXPECT_TRUE(result_last.has_value());
+ EXPECT_EQ(result_last.value(), readers[99]);
+}
+
+// Edge cases
+TEST_F(InvertedIndexIteratorTest, EdgeCase_EmptyAnalyzerKeyQuery) {
+ InvertedIndexIterator iterator;
+ auto reader = create_mock_reader("");
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, reader);
+
+ auto result = iterator.select_best_reader("");
+ EXPECT_TRUE(result.has_value());
+ EXPECT_EQ(result.value(), reader);
+}
+
+TEST_F(InvertedIndexIteratorTest, EdgeCase_CaseInsensitiveQuery) {
+ InvertedIndexIterator iterator;
+ auto reader = create_mock_reader("chinese");
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, reader);
+
+ auto result = iterator.select_best_reader("CHINESE");
+ EXPECT_TRUE(result.has_value());
+ EXPECT_EQ(result.value(), reader);
+}
+
+TEST_F(InvertedIndexIteratorTest, EdgeCase_GetReaderByType) {
+ InvertedIndexIterator iterator;
+ auto fulltext = create_mock_reader("chinese", InvertedIndexReaderType::FULLTEXT);
+ auto string_type = create_mock_reader("english", InvertedIndexReaderType::STRING_TYPE);
+
+ iterator.add_reader(InvertedIndexReaderType::FULLTEXT, fulltext);
+ iterator.add_reader(InvertedIndexReaderType::STRING_TYPE, string_type);
+
+ auto reader = iterator.get_reader(InvertedIndexReaderType::FULLTEXT);
+ EXPECT_NE(reader, nullptr);
+
+ auto reader2 = iterator.get_reader(InvertedIndexReaderType::STRING_TYPE);
+ EXPECT_NE(reader2, nullptr);
+
+ auto reader3 = iterator.get_reader(InvertedIndexReaderType::BKD);
+ EXPECT_EQ(reader3, nullptr);
+}
+
+} // namespace doris::segment_v2
diff --git a/be/test/vec/function/function_match_test.cpp b/be/test/vec/function/function_match_test.cpp
index 6cf275c4ce8..16db2f52cf8 100644
--- a/be/test/vec/function/function_match_test.cpp
+++ b/be/test/vec/function/function_match_test.cpp
@@ -48,7 +48,7 @@ TestInvertedIndexCtx create_inverted_index_ctx(InvertedIndexParserType parser_ty
config.parser_type = parser_type;
test_ctx.analyzer_holder =
doris::segment_v2::inverted_index::InvertedIndexAnalyzer::create_analyzer(&config);
- test_ctx.ctx->analyzer = test_ctx.analyzer_holder.get();
+ test_ctx.ctx->analyzer = test_ctx.analyzer_holder;
}
return test_ctx;
}
@@ -78,7 +78,7 @@ TEST(FunctionMatchTest, analyse_query_str) {
config.parser_type = InvertedIndexParserType::PARSER_ENGLISH;
auto analyzer =
doris::segment_v2::inverted_index::InvertedIndexAnalyzer::create_analyzer(&config);
- inverted_index_ctx->analyzer = analyzer.get();
+ inverted_index_ctx->analyzer = analyzer;
auto query_tokens = func_match_phrase.analyse_query_str_token(inverted_index_ctx.get(),
"a b c", "name");
ASSERT_EQ(query_tokens.size(), 3);
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index 890d9e78bb7..7b2d0af6fc5 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -1600,7 +1600,8 @@ predicate
: NOT? kind=BETWEEN lower=valueExpression AND upper=valueExpression
| NOT? kind=(REGEXP | RLIKE) pattern=valueExpression
| NOT? kind=LIKE pattern=valueExpression (ESCAPE escape=valueExpression)?
- | NOT? kind=(MATCH | MATCH_ANY | MATCH_ALL | MATCH_PHRASE | MATCH_PHRASE_PREFIX | MATCH_REGEXP | MATCH_PHRASE_EDGE) pattern=valueExpression
+ | NOT? kind=(MATCH | MATCH_ANY | MATCH_ALL | MATCH_PHRASE | MATCH_PHRASE_PREFIX | MATCH_REGEXP | MATCH_PHRASE_EDGE)
+ pattern=valueExpression (USING ANALYZER analyzer=identifierOrText)?
| NOT? kind=IN LEFT_PAREN query RIGHT_PAREN
| NOT? kind=IN LEFT_PAREN expression (COMMA expression)* RIGHT_PAREN
| IS NOT? kind=NULL
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
index 96b2c609138..94609e94ca0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
@@ -2827,12 +2827,26 @@ public class SchemaChangeHandler extends AlterHandler {
String columnName = indexDef.getColumns().get(0);
Column column = olapTable.getColumn(columnName);
if (column != null && (column.getType().isStringType() || column.getType().isVariantType())) {
- boolean isExistingIndexAnalyzer = index.isAnalyzedInvertedIndex();
- boolean isNewIndexAnalyzer = indexDef.isAnalyzedInvertedIndex();
- if (isExistingIndexAnalyzer == isNewIndexAnalyzer) {
- throw new DdlException(
- indexDef.getIndexType() + " index for column (" + columnName + ") with "
- + (isNewIndexAnalyzer ? "analyzed" : "non-analyzed") + " type already exists.");
+ if (index.getIndexType() == IndexType.INVERTED) {
+ String existingIdentity = index.getAnalyzerIdentity();
+ String newIdentity = indexDef.getAnalyzerIdentity();
+ if (Objects.equals(existingIdentity, newIdentity)) {
+ String analyzerDesc = "__default__".equals(newIdentity)
+ ? "default analyzer"
+ : "analyzer identity '" + newIdentity + "'";
+ throw new DdlException(indexDef.getIndexType()
+ + " index for column (" + columnName + ") with analyzer "
+ + analyzerDesc + " already exists.");
+ }
+ } else {
+ boolean isExistingIndexAnalyzer = index.isAnalyzedInvertedIndex();
+ boolean isNewIndexAnalyzer = indexDef.isAnalyzedInvertedIndex();
+ if (isExistingIndexAnalyzer == isNewIndexAnalyzer) {
+ throw new DdlException(
+ indexDef.getIndexType() + " index for column (" + columnName + ") with "
+ + (isNewIndexAnalyzer ? "analyzed" : "non-analyzed")
+ + " type already exists.");
+ }
}
} else {
throw new DdlException(
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzerSelector.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzerSelector.java
new file mode 100644
index 00000000000..8933cd79d00
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzerSelector.java
@@ -0,0 +1,198 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Index;
+
+import com.google.common.base.Strings;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Helper for selecting the analyzer that should be attached to a MATCH predicate.
+ *
+ * <p>Design principles:
+ * <ul>
+ * <li>FE transmits user intent clearly to BE</li>
+ * <li>Empty string = user did not specify analyzer (BE chooses)</li>
+ * <li>Non-empty string = user explicitly specified analyzer (BE exact matches)</li>
+ * </ul>
+ *
+ * <p>This simple protocol eliminates ambiguity between "no preference" and "none" analyzer.
+ */
+public final class AnalyzerSelector {
+
+ private AnalyzerSelector() {
+ }
+
+ /**
+ * Select analyzer configuration based on index and user request.
+ *
+ * @param index The inverted index (may be null if table has no index)
+ * @param requestedAnalyzer User-specified analyzer from USING ANALYZER clause (may be null)
+ * @return Selection containing analyzer and parser information
+ */
+ public static Selection select(Index index, String requestedAnalyzer) {
+ Map<String, String> properties = index == null ? Collections.emptyMap() : index.getProperties();
+ return select(properties, requestedAnalyzer, index == null);
+ }
+
+ /**
+ * Select analyzer configuration from properties map.
+ * Used when Index object is not available.
+ */
+ public static Selection select(Map<String, String> properties, String requestedAnalyzer) {
+ return select(properties, requestedAnalyzer, properties == null);
+ }
+
+ private static Selection select(Map<String, String> properties, String requestedAnalyzer, boolean noIndex) {
+ if (properties == null) {
+ properties = Collections.emptyMap();
+ }
+
+ // 1. Normalize user request
+ String userAnalyzer = normalize(requestedAnalyzer);
+
+ // 2. Get index configuration
+ String indexParser = InvertedIndexUtil.getInvertedIndexParser(properties);
+ String indexCustomAnalyzer = InvertedIndexUtil.getPreferredAnalyzer(properties);
+
+ // 3. Determine parser for slow path tokenization
+ // - No index: use english (more aggressive tokenization)
+ // - Has index with explicit parser: use index's configured parser
+ // - Has index with custom analyzer (no explicit parser): use english
+ // (custom analyzer implies fulltext tokenization)
+ // - Has index but no analyzer/parser: use none (keyword matching)
+ String parser;
+ if (noIndex) {
+ parser = InvertedIndexUtil.INVERTED_INDEX_PARSER_ENGLISH;
+ } else if (!Strings.isNullOrEmpty(indexParser)
+ && !InvertedIndexUtil.INVERTED_INDEX_PARSER_NONE.equalsIgnoreCase(indexParser)) {
+ parser = indexParser;
+ } else if (!Strings.isNullOrEmpty(indexCustomAnalyzer)) {
+ // Custom analyzer exists but no explicit parser
+ // Use english for slow-path tokenization since custom analyzer implies fulltext
+ parser = InvertedIndexUtil.INVERTED_INDEX_PARSER_ENGLISH;
+ } else {
+ parser = InvertedIndexUtil.INVERTED_INDEX_PARSER_NONE;
+ }
+
+ // 4. Create selection
+ // - userAnalyzer: what user explicitly specified (empty if not specified)
+ // - indexAnalyzer: what the index is configured with (for slow path fallback)
+ String indexAnalyzer = determineIndexAnalyzer(indexCustomAnalyzer, indexParser);
+
+ return new Selection(userAnalyzer, indexAnalyzer, parser, !Strings.isNullOrEmpty(userAnalyzer));
+ }
+
+ /**
+ * Determine the analyzer configured on the index.
+ * Priority: custom analyzer > builtin parser (if not "none")
+ */
+ private static String determineIndexAnalyzer(String customAnalyzer, String parser) {
+ if (!Strings.isNullOrEmpty(customAnalyzer)) {
+ return customAnalyzer.trim().toLowerCase();
+ }
+ if (!Strings.isNullOrEmpty(parser)
+ && !InvertedIndexUtil.INVERTED_INDEX_PARSER_NONE.equalsIgnoreCase(parser)) {
+ return parser.trim().toLowerCase();
+ }
+ // Keyword index (parser=none) or no analyzer configured
+ return "";
+ }
+
+ private static String normalize(String analyzer) {
+ return analyzer == null ? "" : analyzer.trim().toLowerCase();
+ }
+
+ /**
+ * Result of analyzer selection.
+ *
+ * <p>Key semantics:
+ * <ul>
+ * <li>{@link #effectiveAnalyzerName()}: What to send to BE for index selection</li>
+ * <li>{@link #parser()}: Parser type for slow path tokenization</li>
+ * </ul>
+ */
+ public static final class Selection {
+ // User explicitly specified analyzer (empty if not specified)
+ private final String userAnalyzer;
+ // Analyzer configured on index (empty for keyword index)
+ private final String indexAnalyzer;
+ // Parser type for slow path
+ private final String parser;
+ // Whether user explicitly specified an analyzer
+ private final boolean explicit;
+
+ private Selection(String userAnalyzer, String indexAnalyzer, String parser, boolean explicit) {
+ this.userAnalyzer = userAnalyzer;
+ this.indexAnalyzer = indexAnalyzer;
+ this.parser = Strings.isNullOrEmpty(parser)
+ ? InvertedIndexUtil.INVERTED_INDEX_PARSER_NONE
+ : parser.trim().toLowerCase();
+ this.explicit = explicit;
+ }
+
+ /**
+ * Get the analyzer to use. For backwards compatibility, returns either
+ * the user-specified analyzer or the index-configured analyzer.
+ */
+ public String analyzer() {
+ return !Strings.isNullOrEmpty(userAnalyzer) ? userAnalyzer : indexAnalyzer;
+ }
+
+ /**
+ * Get the parser type for slow path tokenization.
+ */
+ public String parser() {
+ return parser;
+ }
+
+ /**
+ * Whether user explicitly specified an analyzer via USING ANALYZER clause.
+ */
+ public boolean explicit() {
+ return explicit;
+ }
+
+ /**
+ * Compute the analyzer name to send to BE.
+ *
+ * <p>Returns the analyzer that should be used for query tokenization:
+ * <ul>
+ * <li>User specified analyzer → use that analyzer</li>
+ * <li>User did not specify → use index's configured analyzer</li>
+ * </ul>
+ *
+ * <p>This ensures the query uses the same analyzer as the index for consistent
+ * tokenization. For custom analyzers, this is critical because the query needs
+ * to use the same custom analyzer that was used for indexing.
+ *
+ * @param hasIndex Whether the table has an inverted index
+ * @param fallbackParser Unused, kept for API compatibility
+ * @return Analyzer name for BE
+ */
+ public String effectiveAnalyzerName(boolean hasIndex, String fallbackParser) {
+ // Return userAnalyzer if specified, otherwise indexAnalyzer
+ // This ensures custom analyzers work correctly - the query must use
+ // the same analyzer as the index for proper tokenization matching.
+ return analyzer();
+ }
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java
index 55c26ce3a93..e02dc70e1ee 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java
@@ -355,4 +355,11 @@ public class IndexDef {
|| properties.containsKey(InvertedIndexUtil.INVERTED_INDEX_ANALYZER_NAME_KEY)
|| properties.containsKey(InvertedIndexUtil.INVERTED_INDEX_NORMALIZER_NAME_KEY));
}
+
+ public String getAnalyzerIdentity() {
+ if (indexType != IndexType.INVERTED) {
+ return "";
+ }
+ return InvertedIndexUtil.buildAnalyzerIdentity(properties);
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java
index 65ac4cdbd8f..b7c31c33b7d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java
@@ -17,6 +17,9 @@
package org.apache.doris.analysis;
+import org.apache.doris.analysis.invertedindex.AnalyzerIdentityBuilder;
+import org.apache.doris.analysis.invertedindex.AnalyzerKeyNormalizer;
+import org.apache.doris.analysis.invertedindex.InvertedIndexSqlGenerator;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.common.AnalysisException;
@@ -25,6 +28,10 @@ import org.apache.doris.nereids.trees.plans.commands.info.IndexDefinition;
import org.apache.doris.nereids.types.DataType;
import org.apache.doris.thrift.TInvertedIndexFileStorageFormat;
+import com.google.common.base.Strings;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
@@ -33,6 +40,7 @@ import java.util.Map;
import java.util.Set;
public class InvertedIndexUtil {
+ private static final Logger LOG = LogManager.getLogger(InvertedIndexUtil.class);
public static String INVERTED_INDEX_PARSER_KEY = "parser";
public static String INVERTED_INDEX_PARSER_KEY_ALIAS = "built_in_analyzer";
@@ -73,6 +81,9 @@ public class InvertedIndexUtil {
public static String INVERTED_INDEX_PARSER_FIELD_PATTERN_KEY = "field_pattern";
+ // Default analyzer key constant - matches BE's INVERTED_INDEX_DEFAULT_ANALYZER_KEY
+ public static final String INVERTED_INDEX_DEFAULT_ANALYZER_KEY = "__default__";
+
public static String getInvertedIndexParser(Map<String, String> properties) {
if (properties == null) {
return INVERTED_INDEX_PARSER_NONE;
@@ -85,7 +96,10 @@ public class InvertedIndexUtil {
}
public static String getInvertedIndexParserMode(Map<String, String> properties) {
- String mode = properties == null ? null : properties.get(INVERTED_INDEX_PARSER_MODE_KEY);
+ if (properties == null) {
+ return INVERTED_INDEX_PARSER_COARSE_GRANULARITY;
+ }
+ String mode = properties.get(INVERTED_INDEX_PARSER_MODE_KEY);
String parser = properties.get(INVERTED_INDEX_PARSER_KEY);
if (parser == null) {
parser = properties.get(INVERTED_INDEX_PARSER_KEY_ALIAS);
@@ -106,6 +120,19 @@ public class InvertedIndexUtil {
return supportPhrase != null ? Boolean.parseBoolean(supportPhrase) : true;
}
+ public static String getPreferredAnalyzer(Map<String, String> properties) {
+ if (properties == null || properties.isEmpty()) {
+ return "";
+ }
+ // Check analyzer first, then normalizer
+ String analyzer = properties.get(INVERTED_INDEX_ANALYZER_NAME_KEY);
+ if (analyzer != null && !analyzer.isEmpty()) {
+ return analyzer;
+ }
+ String normalizer = properties.get(INVERTED_INDEX_NORMALIZER_NAME_KEY);
+ return normalizer != null ? normalizer : "";
+ }
+
public static Map<String, String> getInvertedIndexCharFilter(Map<String, String> properties) {
if (properties == null) {
return new HashMap<>();
@@ -365,6 +392,22 @@ public class InvertedIndexUtil {
"dict_compression can only be set when storage format is V3");
}
}
+
+ // Normalize analyzer and normalizer names to lowercase for case-insensitive matching
+ normalizeInvertedIndexProperties(properties);
+ }
+
+ /**
+ * Normalize analyzer and normalizer names in index properties to lowercase.
+ * This ensures case-insensitive matching between table creation and query time.
+ */
+ public static void normalizeInvertedIndexProperties(Map<String, String> properties) {
+ AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(
+ properties,
+ INVERTED_INDEX_ANALYZER_NAME_KEY,
+ INVERTED_INDEX_NORMALIZER_NAME_KEY,
+ INVERTED_INDEX_PARSER_KEY,
+ INVERTED_INDEX_PARSER_KEY_ALIAS);
}
private static void checkAnalyzerName(String analyzerName, PrimitiveType colType) throws AnalysisException {
@@ -398,27 +441,62 @@ public class InvertedIndexUtil {
}
public static boolean canHaveMultipleInvertedIndexes(DataType colType, List<IndexDefinition> indexDefs) {
- if (indexDefs.size() == 0 || indexDefs.size() == 1) {
+ if (indexDefs.size() <= 1) {
return true;
}
if (!colType.isStringLikeType() && !colType.isVariantType()) {
return false;
}
- if (indexDefs.size() > 2) {
- return false;
- }
- boolean findParsedInvertedIndex = false;
- boolean findNonParsedInvertedIndex = false;
+
+ Set<String> analyzerKeys = new HashSet<>();
for (IndexDefinition indexDef : indexDefs) {
- if (indexDef.isAnalyzedInvertedIndex()) {
- findParsedInvertedIndex = true;
- } else {
- findNonParsedInvertedIndex = true;
+ String key = buildAnalyzerIdentity(indexDef.getProperties());
+ // HashSet.add() returns false if element already exists
+ if (!analyzerKeys.add(key)) {
+ return false;
}
}
- if (findParsedInvertedIndex && findNonParsedInvertedIndex) {
- return true;
+ return true;
+ }
+
+ public static String buildAnalyzerIdentity(Map<String, String> properties) {
+ String preferredAnalyzer = getPreferredAnalyzer(properties);
+ String parser = getInvertedIndexParser(properties);
+ return AnalyzerIdentityBuilder.buildAnalyzerIdentity(
+ properties,
+ preferredAnalyzer,
+ parser,
+ INVERTED_INDEX_DEFAULT_ANALYZER_KEY,
+ INVERTED_INDEX_PARSER_NONE,
+ LOG);
+ }
+
+ public static boolean isAnalyzerMatched(Map<String, String> properties, String analyzer) {
+ String normalizedAnalyzer = Strings.isNullOrEmpty(analyzer) ? "" : analyzer.trim();
+
+ if (Strings.isNullOrEmpty(normalizedAnalyzer)) {
+ return INVERTED_INDEX_DEFAULT_ANALYZER_KEY.equals(buildAnalyzerIdentity(properties));
+ }
+
+ String preferredAnalyzer = getPreferredAnalyzer(properties);
+ if (!Strings.isNullOrEmpty(preferredAnalyzer)) {
+ return normalizedAnalyzer.equalsIgnoreCase(preferredAnalyzer);
}
- return false;
+
+ String parser = getInvertedIndexParser(properties);
+ if (Strings.isNullOrEmpty(parser)) {
+ return normalizedAnalyzer.equalsIgnoreCase("default")
+ || normalizedAnalyzer.equalsIgnoreCase(INVERTED_INDEX_PARSER_NONE);
+ }
+ return normalizedAnalyzer.equalsIgnoreCase(parser);
+ }
+
+ /**
+ * Builds the SQL fragment for USING ANALYZER clause.
+ * Returns empty string if analyzer is null or empty.
+ * Otherwise returns " USING ANALYZER <analyzer>" with proper quoting.
+ */
+ public static String buildAnalyzerSqlFragment(String analyzer) {
+ return InvertedIndexSqlGenerator.buildAnalyzerSqlFragment(analyzer);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/MatchPredicate.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/MatchPredicate.java
index 9392981c3e6..eeb343eb745 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/MatchPredicate.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/MatchPredicate.java
@@ -28,10 +28,11 @@ import org.apache.doris.thrift.TExprNodeType;
import org.apache.doris.thrift.TExprOpcode;
import org.apache.doris.thrift.TMatchPredicate;
-import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
import com.google.common.collect.Lists;
import com.google.gson.annotations.SerializedName;
+import java.util.Collections;
import java.util.Map;
import java.util.Objects;
@@ -76,12 +77,15 @@ public class MatchPredicate extends Predicate {
@SerializedName("op")
private Operator op;
+ // Fields for thrift serialization (restored from old version)
private String invertedIndexParser;
private String invertedIndexParserMode;
private Map<String, String> invertedIndexCharFilter;
private boolean invertedIndexParserLowercase = true;
private String invertedIndexParserStopwords = "";
private String invertedIndexAnalyzerName = "";
+ // Fields for SQL generation
+ private String explicitAnalyzer = "";
private MatchPredicate() {
// use for serde only
@@ -89,19 +93,6 @@ public class MatchPredicate extends Predicate {
invertedIndexParserMode = InvertedIndexUtil.INVERTED_INDEX_PARSER_FINE_GRANULARITY;
}
- public MatchPredicate(Operator op, Expr e1, Expr e2) {
- super();
- this.op = op;
- Preconditions.checkNotNull(e1);
- children.add(e1);
- Preconditions.checkNotNull(e2);
- children.add(e2);
- // TODO: Calculate selectivity
- selectivity = Expr.DEFAULT_SELECTIVITY;
- invertedIndexParser = InvertedIndexUtil.INVERTED_INDEX_PARSER_UNKNOWN;
- invertedIndexParserMode = InvertedIndexUtil.INVERTED_INDEX_PARSER_FINE_GRANULARITY;
- }
-
protected MatchPredicate(MatchPredicate other) {
super(other);
op = other.op;
@@ -111,24 +102,44 @@ public class MatchPredicate extends Predicate {
invertedIndexParserLowercase = other.invertedIndexParserLowercase;
invertedIndexParserStopwords = other.invertedIndexParserStopwords;
invertedIndexAnalyzerName = other.invertedIndexAnalyzerName;
+ explicitAnalyzer = other.explicitAnalyzer;
}
/**
* use for Nereids ONLY
*/
public MatchPredicate(Operator op, Expr e1, Expr e2, Type retType,
- NullableMode nullableMode, Index invertedIndex) {
- this(op, e1, e2);
- if (invertedIndex != null) {
- this.invertedIndexParser = invertedIndex.getInvertedIndexParser();
- this.invertedIndexParserMode = invertedIndex.getInvertedIndexParserMode();
- this.invertedIndexCharFilter = invertedIndex.getInvertedIndexCharFilter();
- this.invertedIndexParserLowercase = invertedIndex.getInvertedIndexParserLowercase();
- this.invertedIndexParserStopwords = invertedIndex.getInvertedIndexParserStopwords();
- this.invertedIndexAnalyzerName = invertedIndex.getInvertedIndexAnalyzerName();
+ NullableMode nullableMode, Index invertedIndex, boolean nullable) {
+ this(op, e1, e2, retType, nullableMode, invertedIndex, nullable, null);
+ }
+
+ public MatchPredicate(Operator op, Expr e1, Expr e2, Type retType,
+ NullableMode nullableMode, Index invertedIndex, boolean nullable, String analyzer) {
+ super();
+ this.op = op;
+ children.add(e1);
+ children.add(e2);
+ Map<String, String> properties = invertedIndex == null ? Collections.emptyMap() : invertedIndex.getProperties();
+ AnalyzerSelector.Selection selection = AnalyzerSelector.select(invertedIndex, analyzer);
+
+ // Set parser and analyzer fields
+ this.invertedIndexParser = selection.parser();
+ this.invertedIndexAnalyzerName = selection.effectiveAnalyzerName(
+ invertedIndex != null, this.invertedIndexParser);
+
+ // Extract additional index properties for thrift serialization
+ this.invertedIndexParserMode = InvertedIndexUtil.getInvertedIndexParserMode(properties);
+ this.invertedIndexCharFilter = InvertedIndexUtil.getInvertedIndexCharFilter(properties);
+ this.invertedIndexParserLowercase = InvertedIndexUtil.getInvertedIndexParserLowercase(properties);
+ this.invertedIndexParserStopwords = InvertedIndexUtil.getInvertedIndexParserStopwords(properties);
+
+ if (!Strings.isNullOrEmpty(analyzer)) {
+ // Normalize to lowercase for case-insensitive matching
+ this.explicitAnalyzer = analyzer.trim().toLowerCase();
}
fn = new Function(new FunctionName(op.name), Lists.newArrayList(e1.getType(), e2.getType()), retType,
false, true, nullableMode);
+ setNullableFromNereids(nullable);
}
@Override
@@ -145,25 +156,32 @@ public class MatchPredicate extends Predicate {
if (!super.equals(obj)) {
return false;
}
- return ((MatchPredicate) obj).op == op;
+ MatchPredicate other = (MatchPredicate) obj;
+ return other.op == op
+ && Objects.equals(explicitAnalyzer, other.explicitAnalyzer)
+ && Objects.equals(invertedIndexAnalyzerName, other.invertedIndexAnalyzerName)
+ && Objects.equals(invertedIndexParser, other.invertedIndexParser);
}
@Override
public String toSqlImpl() {
- return getChild(0).toSql() + " " + op.toString() + " " + getChild(1).toSql();
+ return getChild(0).toSql() + " " + op.toString() + " " + getChild(1).toSql()
+ + analyzerSqlFragment();
}
@Override
public String toSqlImpl(boolean disableTableName, boolean needExternalSql, TableType tableType,
TableIf table) {
return getChild(0).toSql(disableTableName, needExternalSql, tableType, table) + " " + op.toString() + " "
- + getChild(1).toSql(disableTableName, needExternalSql, tableType, table);
+ + getChild(1).toSql(disableTableName, needExternalSql, tableType, table)
+ + analyzerSqlFragment();
}
@Override
protected void toThrift(TExprNode msg) {
msg.node_type = TExprNodeType.MATCH_PRED;
msg.setOpcode(op.getOpcode());
+ // Use new TMatchPredicate constructor with required fields
msg.match_predicate = new TMatchPredicate(invertedIndexParser, invertedIndexParserMode);
msg.match_predicate.setCharFilterMap(invertedIndexCharFilter);
msg.match_predicate.setParserLowercase(invertedIndexParserLowercase);
@@ -173,6 +191,10 @@ public class MatchPredicate extends Predicate {
@Override
public int hashCode() {
- return 31 * super.hashCode() + Objects.hashCode(op);
+ return Objects.hash(super.hashCode(), op, explicitAnalyzer, invertedIndexAnalyzerName, invertedIndexParser);
+ }
+
+ private String analyzerSqlFragment() {
+ return InvertedIndexUtil.buildAnalyzerSqlFragment(explicitAnalyzer);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java
new file mode 100644
index 00000000000..1e640fd7d22
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java
@@ -0,0 +1,241 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis.invertedindex;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.indexpolicy.IndexPolicy;
+import org.apache.doris.indexpolicy.IndexPolicyTypeEnum;
+
+import com.google.common.base.Strings;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Map;
+import java.util.TreeMap;
+
+public final class AnalyzerIdentityBuilder {
+ private AnalyzerIdentityBuilder() {
+ }
+
+ public static String buildAnalyzerIdentity(
+ Map<String, String> properties,
+ String preferredAnalyzer,
+ String parser,
+ String defaultAnalyzerKey,
+ String parserNone,
+ Logger log) {
+ if (properties == null || properties.isEmpty()) {
+ return defaultAnalyzerKey;
+ }
+
+ if (!Strings.isNullOrEmpty(preferredAnalyzer)) {
+ // For custom analyzer/normalizer, resolve to underlying config to build identity
+ return resolveAnalyzerIdentity(preferredAnalyzer, defaultAnalyzerKey, log);
+ }
+
+ if (Strings.isNullOrEmpty(parser) || parserNone.equalsIgnoreCase(parser)) {
+ return defaultAnalyzerKey;
+ }
+ return parser;
+ }
+
+ /**
+ * Resolve analyzer/normalizer name to its underlying configuration identity.
+ * Two analyzers with same underlying config (tokenizer + token_filter + char_filter)
+ * will have the same identity, even if they have different names.
+ */
+ private static String resolveAnalyzerIdentity(String analyzerName, String defaultAnalyzerKey, Logger log) {
+ if (Strings.isNullOrEmpty(analyzerName)) {
+ return defaultAnalyzerKey;
+ }
+
+ // Check if it's a built-in analyzer
+ if (IndexPolicy.BUILTIN_ANALYZERS.contains(analyzerName)) {
+ return analyzerName;
+ }
+
+ // Check if it's a built-in normalizer
+ if (IndexPolicy.BUILTIN_NORMALIZERS.contains(analyzerName)) {
+ return "normalizer:" + analyzerName;
+ }
+
+ // For custom analyzer/normalizer, get underlying config from IndexPolicyMgr
+ try {
+ Env env = Env.getCurrentEnv();
+ if (env == null || env.getIndexPolicyMgr() == null) {
+ // Env not initialized - this can happen during early startup or tests
+ if (log != null) {
+ log.debug("Env or IndexPolicyMgr not available, using name '{}' as identity", analyzerName);
+ }
+ return analyzerName;
+ }
+
+ IndexPolicy policy = env.getIndexPolicyMgr().getPolicyByName(analyzerName);
+ if (policy == null) {
+ // Policy not found - this is expected for custom analyzers not yet registered
+ if (log != null) {
+ log.debug("Analyzer/normalizer policy not found for '{}', using name as identity", analyzerName);
+ }
+ return analyzerName;
+ }
+
+ Map<String, String> policyProps = policy.getProperties();
+ if (policyProps == null || policyProps.isEmpty()) {
+ if (log != null) {
+ log.debug("Policy '{}' has no properties, using name as identity", analyzerName);
+ }
+ return analyzerName;
+ }
+
+ // Build identity from underlying config using sorted keys for consistent ordering
+ return buildIdentityFromPolicyProperties(policy.getType(), policyProps);
+ } catch (RuntimeException e) {
+ // Catch RuntimeException specifically rather than generic Exception
+ if (log != null) {
+ log.warn("Failed to resolve analyzer identity for '{}', using name as identity. "
+ + "This may cause incorrect duplicate detection. Error: {}",
+ analyzerName, e.getMessage());
+ }
+ return analyzerName;
+ }
+ }
+
+ /**
+ * Build identity string from policy properties.
+ * Uses TreeMap to ensure consistent key ordering.
+ */
+ private static String buildIdentityFromPolicyProperties(IndexPolicyTypeEnum type,
+ Map<String, String> properties) {
+ // Use TreeMap to sort keys for consistent identity
+ TreeMap<String, String> sortedProps = new TreeMap<>(properties);
+
+ StringBuilder sb = new StringBuilder();
+ sb.append(type.name()).append(":");
+
+ for (Map.Entry<String, String> entry : sortedProps.entrySet()) {
+ String key = entry.getKey();
+ String value = entry.getValue();
+
+ // For tokenizer, token_filter, char_filter - resolve recursively if needed
+ if (IndexPolicy.PROP_TOKENIZER.equals(key)) {
+ sb.append("tokenizer=").append(resolveComponentIdentity(value, IndexPolicyTypeEnum.TOKENIZER));
+ } else if (IndexPolicy.PROP_TOKEN_FILTER.equals(key)) {
+ sb.append("token_filter=").append(resolveTokenFilterIdentity(value));
+ } else if (IndexPolicy.PROP_CHAR_FILTER.equals(key)) {
+ sb.append("char_filter=").append(resolveCharFilterIdentity(value));
+ }
+ sb.append(";");
+ }
+
+ return sb.toString();
+ }
+
+ /**
+ * Resolve a component (tokenizer) to its identity.
+ */
+ private static String resolveComponentIdentity(String name, IndexPolicyTypeEnum expectedType) {
+ if (Strings.isNullOrEmpty(name)) {
+ return "";
+ }
+
+ // Check if it's a built-in component
+ if (expectedType == IndexPolicyTypeEnum.TOKENIZER
+ && IndexPolicy.BUILTIN_TOKENIZERS.contains(name)) {
+ return name;
+ }
+
+ // For custom component, get its properties
+ try {
+ Env env = Env.getCurrentEnv();
+ if (env == null || env.getIndexPolicyMgr() == null) {
+ return name;
+ }
+
+ IndexPolicy policy = env.getIndexPolicyMgr().getPolicyByName(name);
+ if (policy == null || policy.getType() != expectedType) {
+ return name;
+ }
+
+ Map<String, String> props = policy.getProperties();
+ if (props == null || props.isEmpty()) {
+ return name;
+ }
+
+ // Build identity from sorted properties
+ TreeMap<String, String> sortedProps = new TreeMap<>(props);
+ return sortedProps.toString();
+ } catch (RuntimeException e) {
+ return name;
+ }
+ }
+
+ /**
+ * Resolve token filter list to identity string.
+ * IMPORTANT: Order is preserved because filter order is semantically significant.
+ */
+ private static String resolveTokenFilterIdentity(String filterList) {
+ if (Strings.isNullOrEmpty(filterList)) {
+ return "";
+ }
+
+ StringBuilder sb = new StringBuilder();
+ String[] filters = filterList.split(",\\s*");
+ // DO NOT sort - filter order is semantically significant
+
+ for (int i = 0; i < filters.length; i++) {
+ String filter = filters[i].trim();
+ if (i > 0) {
+ sb.append(",");
+ }
+
+ if (IndexPolicy.BUILTIN_TOKEN_FILTERS.contains(filter)) {
+ sb.append(filter);
+ } else {
+ sb.append(resolveComponentIdentity(filter, IndexPolicyTypeEnum.TOKEN_FILTER));
+ }
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Resolve char filter list to identity string.
+ * IMPORTANT: Order is preserved because filter order is semantically significant.
+ */
+ private static String resolveCharFilterIdentity(String filterList) {
+ if (Strings.isNullOrEmpty(filterList)) {
+ return "";
+ }
+
+ StringBuilder sb = new StringBuilder();
+ String[] filters = filterList.split(",\\s*");
+ // DO NOT sort - filter order is semantically significant
+
+ for (int i = 0; i < filters.length; i++) {
+ String filter = filters[i].trim();
+ if (i > 0) {
+ sb.append(",");
+ }
+
+ if (IndexPolicy.BUILTIN_CHAR_FILTERS.contains(filter)) {
+ sb.append(filter);
+ } else {
+ sb.append(resolveComponentIdentity(filter, IndexPolicyTypeEnum.CHAR_FILTER));
+ }
+ }
+ return sb.toString();
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerKeyNormalizer.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerKeyNormalizer.java
new file mode 100644
index 00000000000..057ad463d05
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerKeyNormalizer.java
@@ -0,0 +1,44 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis.invertedindex;
+
+import java.util.Map;
+
+public final class AnalyzerKeyNormalizer {
+ private AnalyzerKeyNormalizer() {
+ }
+
+ public static void normalizeInvertedIndexProperties(Map<String, String> properties, String... keys) {
+ if (properties == null || keys == null) {
+ return;
+ }
+ for (String key : keys) {
+ normalizePropertyIfPresent(properties, key);
+ }
+ }
+
+ private static void normalizePropertyIfPresent(Map<String, String> properties, String key) {
+ if (key == null) {
+ return;
+ }
+ String value = properties.get(key);
+ if (value != null && !value.isEmpty()) {
+ properties.put(key, value.trim().toLowerCase());
+ }
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/InvertedIndexSqlGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/InvertedIndexSqlGenerator.java
new file mode 100644
index 00000000000..be09d884047
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/InvertedIndexSqlGenerator.java
@@ -0,0 +1,44 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis.invertedindex;
+
+import com.google.common.base.Strings;
+
+public final class InvertedIndexSqlGenerator {
+ private InvertedIndexSqlGenerator() {
+ }
+
+ /**
+ * Builds the SQL fragment for USING ANALYZER clause.
+ * Returns empty string if analyzer is null or empty.
+ * Otherwise returns " USING ANALYZER <analyzer>" with proper quoting.
+ */
+ public static String buildAnalyzerSqlFragment(String analyzer) {
+ if (Strings.isNullOrEmpty(analyzer)) {
+ return "";
+ }
+ String trimmed = analyzer.trim();
+ if (trimmed.isEmpty()) {
+ return "";
+ }
+ if (trimmed.matches("[A-Za-z_][A-Za-z0-9_]*")) {
+ return " USING ANALYZER " + trimmed;
+ }
+ return " USING ANALYZER '" + trimmed.replace("'", "''") + "'";
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java
index 215c4ce4c8c..352f89c7959 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java
@@ -393,4 +393,11 @@ public class Index implements Writable {
|| properties.containsKey(InvertedIndexUtil.INVERTED_INDEX_ANALYZER_NAME_KEY)
|| properties.containsKey(InvertedIndexUtil.INVERTED_INDEX_NORMALIZER_NAME_KEY));
}
+
+ public String getAnalyzerIdentity() {
+ if (indexType != IndexDef.IndexType.INVERTED) {
+ return "";
+ }
+ return InvertedIndexUtil.buildAnalyzerIdentity(properties);
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index 1e69e9004fe..6b6168cee4c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -21,6 +21,7 @@ import org.apache.doris.alter.MaterializedViewHandler;
import org.apache.doris.analysis.ColumnDef;
import org.apache.doris.analysis.DataSortInfo;
import org.apache.doris.analysis.IndexDef;
+import org.apache.doris.analysis.InvertedIndexUtil;
import org.apache.doris.backup.Status;
import org.apache.doris.backup.Status.ErrCode;
import org.apache.doris.catalog.DistributionInfo.DistributionInfoType;
@@ -3693,6 +3694,10 @@ public class OlapTable extends Table implements MTMVRelatedTableIf, GsonPostProc
}
public Index getInvertedIndex(Column column, List<String> subPath) {
+ return getInvertedIndex(column, subPath, null);
+ }
+
+ public Index getInvertedIndex(Column column, List<String> subPath, String analyzer) {
List<Index> invertedIndexes = new ArrayList<>();
for (Index index : indexes.getIndexes()) {
if (index.getIndexType() == IndexDef.IndexType.INVERTED) {
@@ -3703,9 +3708,11 @@ public class OlapTable extends Table implements MTMVRelatedTableIf, GsonPostProc
}
}
+ List<Index> filteredInvertedIndexes = filterIndexesByAnalyzer(invertedIndexes, analyzer);
+
if (subPath == null || subPath.isEmpty()) {
- return invertedIndexes.size() == 1 ? invertedIndexes.get(0)
- : invertedIndexes.stream().filter(Index::isAnalyzedInvertedIndex).findFirst().orElse(null);
+ return filteredInvertedIndexes.size() == 1 ? filteredInvertedIndexes.get(0)
+ : filteredInvertedIndexes.stream().filter(Index::isAnalyzedInvertedIndex).findFirst().orElse(null);
}
// subPath is not empty, means it is a variant column, find the field pattern from children
@@ -3742,12 +3749,13 @@ public class OlapTable extends Table implements MTMVRelatedTableIf, GsonPostProc
}
}
}
- if (invertedIndexesWithFieldPattern.isEmpty()) {
- return invertedIndexes.size() == 1 ? invertedIndexes.get(0)
- : invertedIndexes.stream().filter(Index::isAnalyzedInvertedIndex).findFirst().orElse(null);
+ List<Index> filteredFieldPatternIndexes = filterIndexesByAnalyzer(invertedIndexesWithFieldPattern, analyzer);
+ if (filteredFieldPatternIndexes.isEmpty()) {
+ return filteredInvertedIndexes.size() == 1 ? filteredInvertedIndexes.get(0)
+ : filteredInvertedIndexes.stream().filter(Index::isAnalyzedInvertedIndex).findFirst().orElse(null);
} else {
- return invertedIndexesWithFieldPattern.size() == 1 ? invertedIndexesWithFieldPattern.get(0)
- : invertedIndexesWithFieldPattern.stream()
+ return filteredFieldPatternIndexes.size() == 1 ? filteredFieldPatternIndexes.get(0)
+ : filteredFieldPatternIndexes.stream()
.filter(Index::isAnalyzedInvertedIndex).findFirst().orElse(null);
}
}
@@ -3795,4 +3803,17 @@ public class OlapTable extends Table implements MTMVRelatedTableIf, GsonPostProc
public ImmutableMap<Long, Backend> getAllBackendsByAllCluster() throws AnalysisException {
return Env.getCurrentSystemInfo().getAllBackendsByAllCluster();
}
+
+ private List<Index> filterIndexesByAnalyzer(List<Index> original, String analyzer) {
+ if (analyzer == null || analyzer.trim().isEmpty()) {
+ return original;
+ }
+ List<Index> matched = new ArrayList<>();
+ for (Index index : original) {
+ if (InvertedIndexUtil.isAnalyzerMatched(index.getProperties(), analyzer)) {
+ matched.add(index);
+ }
+ }
+ return matched;
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java b/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java
index d6c9f127047..e84cbe3ffab 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java
@@ -53,8 +53,17 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
@SerializedName(value = "idToIndexPolicy")
private final Map<Long, IndexPolicy> idToIndexPolicy = Maps.newHashMap();
+ // Keys are normalized to lowercase for case-insensitive lookup
private final Map<String, IndexPolicy> nameToIndexPolicy = Maps.newHashMap();
+ /**
+ * Normalize policy name to lowercase for case-insensitive lookup.
+ * Policy names are case-insensitive in Doris.
+ */
+ private static String normalizeKey(String name) {
+ return name == null ? null : name.trim().toLowerCase();
+ }
+
private void writeLock() {
lock.writeLock().lock();
}
@@ -83,13 +92,15 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
}
public void validateAnalyzerExists(String analyzerName) throws DdlException {
- if (IndexPolicy.BUILTIN_ANALYZERS.contains(analyzerName)) {
+ String normalizedName = normalizeKey(analyzerName);
+ // Built-in analyzers are stored in lowercase, so use normalized name for comparison
+ if (IndexPolicy.BUILTIN_ANALYZERS.contains(normalizedName)) {
return;
}
readLock();
try {
- IndexPolicy policy = nameToIndexPolicy.get(analyzerName);
+ IndexPolicy policy = nameToIndexPolicy.get(normalizedName);
if (policy == null) {
throw new DdlException("Analyzer '" + analyzerName + "' does not exist");
}
@@ -105,13 +116,15 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
}
public void validateNormalizerExists(String normalizerName) throws DdlException {
- if (IndexPolicy.BUILTIN_NORMALIZERS.contains(normalizerName)) {
+ String normalizedName = normalizeKey(normalizerName);
+ // Built-in normalizers are stored in lowercase, so use normalized name for comparison
+ if (IndexPolicy.BUILTIN_NORMALIZERS.contains(normalizedName)) {
return;
}
readLock();
try {
- IndexPolicy policy = nameToIndexPolicy.get(normalizerName);
+ IndexPolicy policy = nameToIndexPolicy.get(normalizedName);
if (policy == null) {
throw new DdlException("Normalizer '" + normalizerName + "' does not exist");
}
@@ -131,16 +144,18 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
if (policyName == null || policyName.trim().isEmpty()) {
throw new DdlException("Policy name cannot be empty or null");
}
- if (IndexPolicy.BUILTIN_TOKENIZERS.contains(policyName)) {
+ // Normalize policy name for case-insensitive comparison with built-in names
+ String normalizedName = normalizeKey(policyName);
+ if (IndexPolicy.BUILTIN_TOKENIZERS.contains(normalizedName)) {
throw new DdlException("Policy name '" + policyName + "' conflicts with built-in tokenizer name");
}
- if (IndexPolicy.BUILTIN_TOKEN_FILTERS.contains(policyName)) {
+ if (IndexPolicy.BUILTIN_TOKEN_FILTERS.contains(normalizedName)) {
throw new DdlException("Policy name '" + policyName + "' conflicts with built-in token filter name");
}
- if (IndexPolicy.BUILTIN_CHAR_FILTERS.contains(policyName)) {
+ if (IndexPolicy.BUILTIN_CHAR_FILTERS.contains(normalizedName)) {
throw new DdlException("Policy name '" + policyName + "' conflicts with built-in char filter name");
}
- if (IndexPolicy.BUILTIN_ANALYZERS.contains(policyName)) {
+ if (IndexPolicy.BUILTIN_ANALYZERS.contains(normalizedName)) {
throw new DdlException("Policy name '" + policyName + "' conflicts with built-in analyzer name");
}
@@ -150,7 +165,7 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
try {
validatePolicyProperties(type, properties);
- if (nameToIndexPolicy.containsKey(policyName)) {
+ if (nameToIndexPolicy.containsKey(normalizedName)) {
if (ifNotExists) {
return;
}
@@ -161,7 +176,8 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
throw new DdlException("Index policy number cannot exceed 100");
}
- nameToIndexPolicy.put(policyName, indexPolicy);
+ // Store with normalized key for case-insensitive lookup
+ nameToIndexPolicy.put(normalizedName, indexPolicy);
idToIndexPolicy.put(indexPolicy.getId(), indexPolicy);
Env.getCurrentEnv().getEditLog().logCreateIndexPolicy(indexPolicy);
} finally {
@@ -173,7 +189,7 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
public IndexPolicy getPolicyByName(String name) {
readLock();
try {
- return nameToIndexPolicy.get(name);
+ return nameToIndexPolicy.get(normalizeKey(name));
} finally {
readUnlock();
}
@@ -399,9 +415,10 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
public void dropIndexPolicy(boolean isIfExists, String indexPolicyName,
IndexPolicyTypeEnum type) throws DdlException, AnalysisException {
+ String normalizedName = normalizeKey(indexPolicyName);
writeLock();
try {
- IndexPolicy policyToDrop = nameToIndexPolicy.get(indexPolicyName);
+ IndexPolicy policyToDrop = nameToIndexPolicy.get(normalizedName);
if (policyToDrop == null) {
if (isIfExists) {
return;
@@ -424,7 +441,7 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
}
long id = policyToDrop.getId();
idToIndexPolicy.remove(id);
- nameToIndexPolicy.remove(indexPolicyName);
+ nameToIndexPolicy.remove(normalizedName);
Env.getCurrentEnv().getEditLog().logDropIndexPolicy(new DropIndexPolicyLog(id));
} finally {
writeUnlock();
@@ -432,7 +449,18 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
LOG.info("Drop index policy success: {}", indexPolicyName);
}
+ /**
+ * Check if an analyzer is used by any inverted index.
+ *
+ * <p><b>PERFORMANCE WARNING:</b> This method performs a full scan of all databases,
+ * tables, and indexes. In large-scale clusters with many tables, this can be slow.
+ * Consider maintaining a reverse index (analyzer -> tables) if this becomes a bottleneck.
+ *
+ * @param analyzerName the analyzer name to check
+ * @throws DdlException if the analyzer is in use by any index
+ */
private void checkAnalyzerNotUsedByIndex(String analyzerName) throws DdlException {
+ String normalizedName = normalizeKey(analyzerName);
List<Database> databases = Env.getCurrentEnv().getInternalCatalog().getDbs();
for (Database db : databases) {
List<Table> tables = db.getTables();
@@ -441,8 +469,10 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
OlapTable olapTable = (OlapTable) table;
for (Index index : olapTable.getIndexes()) {
Map<String, String> properties = index.getProperties();
- if (properties != null
- && analyzerName.equals(properties.get(IndexPolicy.PROP_ANALYZER))) {
+ String indexAnalyzer = properties == null ? null
+ : properties.get(IndexPolicy.PROP_ANALYZER);
+ if (indexAnalyzer != null
+ && normalizedName.equals(normalizeKey(indexAnalyzer))) {
throw new DdlException("the analyzer " + analyzerName + " is used by index: "
+ index.getIndexName() + " in table: "
+ db.getFullName() + "." + table.getName());
@@ -453,7 +483,18 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
}
}
+ /**
+ * Check if a normalizer is used by any inverted index.
+ *
+ * <p><b>PERFORMANCE WARNING:</b> This method performs a full scan of all databases,
+ * tables, and indexes. In large-scale clusters with many tables, this can be slow.
+ * Consider maintaining a reverse index (normalizer -> tables) if this becomes a bottleneck.
+ *
+ * @param normalizerName the normalizer name to check
+ * @throws DdlException if the normalizer is in use by any index
+ */
private void checkNormalizerNotUsedByIndex(String normalizerName) throws DdlException {
+ String normalizedName = normalizeKey(normalizerName);
List<Database> databases = Env.getCurrentEnv().getInternalCatalog().getDbs();
for (Database db : databases) {
List<Table> tables = db.getTables();
@@ -462,8 +503,10 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
OlapTable olapTable = (OlapTable) table;
for (Index index : olapTable.getIndexes()) {
Map<String, String> properties = index.getProperties();
- if (properties != null
- && normalizerName.equals(properties.get(IndexPolicy.PROP_NORMALIZER))) {
+ String indexNormalizer = properties == null ? null
+ : properties.get(IndexPolicy.PROP_NORMALIZER);
+ if (indexNormalizer != null
+ && normalizedName.equals(normalizeKey(indexNormalizer))) {
throw new DdlException("the normalizer " + normalizerName + " is used by index: "
+ index.getIndexName() + " in table: "
+ db.getFullName() + "." + table.getName());
@@ -542,7 +585,8 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
writeLock();
try {
idToIndexPolicy.put(indexPolicy.getId(), indexPolicy);
- nameToIndexPolicy.put(indexPolicy.getName(), indexPolicy);
+ // Store with normalized key for case-insensitive lookup
+ nameToIndexPolicy.put(normalizeKey(indexPolicy.getName()), indexPolicy);
LOG.debug("Replayed index policy: id={}, name={}",
indexPolicy.getId(), indexPolicy.getName());
} finally {
@@ -559,7 +603,7 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
}
IndexPolicy indexPolicy = idToIndexPolicy.get(id);
idToIndexPolicy.remove(id);
- nameToIndexPolicy.remove(indexPolicy.getName());
+ nameToIndexPolicy.remove(normalizeKey(indexPolicy.getName()));
LOG.debug("Replayed drop index policy: {}", indexPolicy.getName());
} finally {
writeUnlock();
@@ -580,7 +624,8 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {
@Override
public void gsonPostProcess() throws IOException {
+ // Store with normalized key for case-insensitive lookup
idToIndexPolicy.forEach(
- (id, indexPolicy) -> nameToIndexPolicy.put(indexPolicy.getName(), indexPolicy));
+ (id, indexPolicy) -> nameToIndexPolicy.put(normalizeKey(indexPolicy.getName()), indexPolicy));
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java
index 2b2d1235fa5..80accc33c33 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java
@@ -232,13 +232,17 @@ public class ExpressionTranslator extends DefaultExpressionVisitor<Expr, PlanTra
throw new AnalysisException("SlotReference in Match failed to get OlapTable, SQL is " + match.toSql());
}
- Index invertedIndex = olapTbl.getInvertedIndex(column, slot.getSubPath());
+ String analyzer = match.getAnalyzer().orElse(null);
+ Index invertedIndex = olapTbl.getInvertedIndex(column, slot.getSubPath(), analyzer);
+ if (analyzer != null && invertedIndex == null) {
+ throw new AnalysisException("No inverted index found for analyzer '" + analyzer
+ + "' on column " + column.getName());
+ }
MatchPredicate.Operator op = match.op();
MatchPredicate matchPredicate = new MatchPredicate(op, match.left().accept(this, context),
match.right().accept(this, context), match.getDataType().toCatalogDataType(),
- NullableMode.DEPEND_ON_ARGUMENT, invertedIndex);
- matchPredicate.setNullableFromNereids(match.nullable());
+ NullableMode.DEPEND_ON_ARGUMENT, invertedIndex, match.nullable(), analyzer);
return matchPredicate;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 41517b51f82..ab53622370d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -4635,6 +4635,7 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor<Object> {
private Expression withPredicate(Expression valueExpression, PredicateContext ctx) {
return ParserUtils.withOrigin(ctx, () -> {
Expression outExpression;
+ String analyzer = ctx.analyzer != null ? visitIdentifierOrText(ctx.analyzer) : null;
switch (ctx.kind.getType()) {
case DorisParser.BETWEEN:
// don't compare lower and upper before bind expression,
@@ -4695,37 +4696,43 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor<Object> {
case DorisParser.MATCH_ANY:
outExpression = new MatchAny(
valueExpression,
- getExpression(ctx.pattern)
+ getExpression(ctx.pattern),
+ analyzer
);
break;
case DorisParser.MATCH_ALL:
outExpression = new MatchAll(
valueExpression,
- getExpression(ctx.pattern)
+ getExpression(ctx.pattern),
+ analyzer
);
break;
case DorisParser.MATCH_PHRASE:
outExpression = new MatchPhrase(
valueExpression,
- getExpression(ctx.pattern)
+ getExpression(ctx.pattern),
+ analyzer
);
break;
case DorisParser.MATCH_PHRASE_PREFIX:
outExpression = new MatchPhrasePrefix(
valueExpression,
- getExpression(ctx.pattern)
+ getExpression(ctx.pattern),
+ analyzer
);
break;
case DorisParser.MATCH_REGEXP:
outExpression = new MatchRegexp(
valueExpression,
- getExpression(ctx.pattern)
+ getExpression(ctx.pattern),
+ analyzer
);
break;
case DorisParser.MATCH_PHRASE_EDGE:
outExpression = new MatchPhraseEdge(
valueExpression,
- getExpression(ctx.pattern)
+ getExpression(ctx.pattern),
+ analyzer
);
break;
default:
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Match.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Match.java
index 405e3cb8fe4..386d5f63666 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Match.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Match.java
@@ -17,6 +17,7 @@
package org.apache.doris.nereids.trees.expressions;
+import org.apache.doris.analysis.InvertedIndexUtil;
import org.apache.doris.analysis.MatchPredicate.Operator;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.exceptions.UnboundException;
@@ -26,15 +27,48 @@ import org.apache.doris.nereids.types.BooleanType;
import org.apache.doris.nereids.types.DataType;
import org.apache.doris.nereids.types.coercion.AnyDataType;
+import com.google.common.base.Preconditions;
+
import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
/**
* like expression: a MATCH 'hello'.
*/
public abstract class Match extends BinaryOperator implements PropagateNullable {
+ private final Optional<String> analyzer;
+
public Match(List<Expression> children, String symbol) {
+ this(children, symbol, null);
+ }
+
+ /**
+ * Constructor with analyzer parameter.
+ * @param children child expressions
+ * @param symbol the match operator symbol
+ * @param analyzer the analyzer name (will be normalized to lowercase)
+ */
+ public Match(List<Expression> children, String symbol, String analyzer) {
super(children, symbol);
+ // Normalize analyzer name to lowercase for case-insensitive matching
+ this.analyzer = Optional.ofNullable(analyzer)
+ .map(String::trim)
+ .filter(s -> !s.isEmpty())
+ .map(String::toLowerCase);
+ }
+
+ /**
+ * Template method for creating a new instance with the given children and analyzer.
+ * Each subclass must implement this to return a new instance of itself.
+ */
+ protected abstract Match createInstance(List<Expression> children, String analyzer);
+
+ @Override
+ public final Match withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 2);
+ return createInstance(children, analyzer.orElse(null));
}
/**
@@ -77,15 +111,46 @@ public abstract class Match extends BinaryOperator implements PropagateNullable
@Override
public String computeToSql() {
- return "(" + left().toSql() + " " + symbol + " " + right().toSql() + ")";
+ return "(" + left().toSql() + " " + symbol + " " + right().toSql()
+ + analyzerSqlFragment() + ")";
}
@Override
public String toString() {
- return "(" + left().toString() + " " + symbol + " " + right().toString() + ")";
+ return "(" + left().toString() + " " + symbol + " " + right().toString()
+ + analyzerSqlFragment() + ")";
}
public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
return visitor.visitMatch(this, context);
}
+
+ public Optional<String> getAnalyzer() {
+ return analyzer;
+ }
+
+ protected Optional<String> analyzer() {
+ return analyzer;
+ }
+
+ protected String analyzerSqlFragment() {
+ return InvertedIndexUtil.buildAnalyzerSqlFragment(analyzer.orElse(null));
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (!(obj instanceof Match)) {
+ return false;
+ }
+ if (!super.equals(obj)) {
+ return false;
+ }
+ Match other = (Match) obj;
+ return Objects.equals(analyzer, other.analyzer);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(super.hashCode(), analyzer);
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchAll.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchAll.java
index f3d334d9ed8..44f78b8c7da 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchAll.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchAll.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.trees.expressions;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
-import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import java.util.List;
@@ -29,17 +28,20 @@ import java.util.List;
*/
public class MatchAll extends Match {
public MatchAll(Expression left, Expression right) {
- super(ImmutableList.of(left, right), "MATCH_ALL");
+ this(left, right, null);
}
- private MatchAll(List<Expression> children) {
- super(children, "MATCH_ALL");
+ public MatchAll(Expression left, Expression right, String analyzer) {
+ super(ImmutableList.of(left, right), "MATCH_ALL", analyzer);
+ }
+
+ private MatchAll(List<Expression> children, String analyzer) {
+ super(children, "MATCH_ALL", analyzer);
}
@Override
- public MatchAll withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 2);
- return new MatchAll(children);
+ protected MatchAll createInstance(List<Expression> children, String analyzer) {
+ return new MatchAll(children, analyzer);
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchAny.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchAny.java
index 9f4b45370a6..24ecebbe77c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchAny.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchAny.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.trees.expressions;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
-import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import java.util.List;
@@ -29,17 +28,20 @@ import java.util.List;
*/
public class MatchAny extends Match {
public MatchAny(Expression left, Expression right) {
- super(ImmutableList.of(left, right), "MATCH_ANY");
+ this(left, right, null);
}
- private MatchAny(List<Expression> children) {
- super(children, "MATCH_ANY");
+ public MatchAny(Expression left, Expression right, String analyzer) {
+ super(ImmutableList.of(left, right), "MATCH_ANY", analyzer);
+ }
+
+ private MatchAny(List<Expression> children, String analyzer) {
+ super(children, "MATCH_ANY", analyzer);
}
@Override
- public MatchAny withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 2);
- return new MatchAny(children);
+ protected MatchAny createInstance(List<Expression> children, String analyzer) {
+ return new MatchAny(children, analyzer);
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhrase.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhrase.java
index 529e693400e..7299fcbc97a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhrase.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhrase.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.trees.expressions;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
-import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import java.util.List;
@@ -29,17 +28,20 @@ import java.util.List;
*/
public class MatchPhrase extends Match {
public MatchPhrase(Expression left, Expression right) {
- super(ImmutableList.of(left, right), "MATCH_PHRASE");
+ this(left, right, null);
}
- private MatchPhrase(List<Expression> children) {
- super(children, "MATCH_PHRASE");
+ public MatchPhrase(Expression left, Expression right, String analyzer) {
+ super(ImmutableList.of(left, right), "MATCH_PHRASE", analyzer);
+ }
+
+ private MatchPhrase(List<Expression> children, String analyzer) {
+ super(children, "MATCH_PHRASE", analyzer);
}
@Override
- public MatchPhrase withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 2);
- return new MatchPhrase(children);
+ protected MatchPhrase createInstance(List<Expression> children, String analyzer) {
+ return new MatchPhrase(children, analyzer);
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhraseEdge.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhraseEdge.java
index 188f3317a0f..7d70317e352 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhraseEdge.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhraseEdge.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.trees.expressions;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
-import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import java.util.List;
@@ -29,17 +28,20 @@ import java.util.List;
*/
public class MatchPhraseEdge extends Match {
public MatchPhraseEdge(Expression left, Expression right) {
- super(ImmutableList.of(left, right), "MATCH_PHRASE_EDGE");
+ this(left, right, null);
}
- private MatchPhraseEdge(List<Expression> children) {
- super(children, "MATCH_PHRASE_EDGE");
+ public MatchPhraseEdge(Expression left, Expression right, String analyzer) {
+ super(ImmutableList.of(left, right), "MATCH_PHRASE_EDGE", analyzer);
+ }
+
+ private MatchPhraseEdge(List<Expression> children, String analyzer) {
+ super(children, "MATCH_PHRASE_EDGE", analyzer);
}
@Override
- public MatchPhraseEdge withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 2);
- return new MatchPhraseEdge(children);
+ protected MatchPhraseEdge createInstance(List<Expression> children, String analyzer) {
+ return new MatchPhraseEdge(children, analyzer);
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhrasePrefix.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhrasePrefix.java
index 748da21ce30..288749fc4f1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhrasePrefix.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchPhrasePrefix.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.trees.expressions;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
-import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import java.util.List;
@@ -29,17 +28,20 @@ import java.util.List;
*/
public class MatchPhrasePrefix extends Match {
public MatchPhrasePrefix(Expression left, Expression right) {
- super(ImmutableList.of(left, right), "MATCH_PHRASE_PREFIX");
+ this(left, right, null);
}
- private MatchPhrasePrefix(List<Expression> children) {
- super(children, "MATCH_PHRASE_PREFIX");
+ public MatchPhrasePrefix(Expression left, Expression right, String analyzer) {
+ super(ImmutableList.of(left, right), "MATCH_PHRASE_PREFIX", analyzer);
+ }
+
+ private MatchPhrasePrefix(List<Expression> children, String analyzer) {
+ super(children, "MATCH_PHRASE_PREFIX", analyzer);
}
@Override
- public MatchPhrasePrefix withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 2);
- return new MatchPhrasePrefix(children);
+ protected MatchPhrasePrefix createInstance(List<Expression> children, String analyzer) {
+ return new MatchPhrasePrefix(children, analyzer);
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchRegexp.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchRegexp.java
index 6bb55aeb897..dda29d6a116 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchRegexp.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/MatchRegexp.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.trees.expressions;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
-import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import java.util.List;
@@ -29,17 +28,20 @@ import java.util.List;
*/
public class MatchRegexp extends Match {
public MatchRegexp(Expression left, Expression right) {
- super(ImmutableList.of(left, right), "MATCH_REGEXP");
+ this(left, right, null);
}
- private MatchRegexp(List<Expression> children) {
- super(children, "MATCH_REGEXP");
+ public MatchRegexp(Expression left, Expression right, String analyzer) {
+ super(ImmutableList.of(left, right), "MATCH_REGEXP", analyzer);
+ }
+
+ private MatchRegexp(List<Expression> children, String analyzer) {
+ super(children, "MATCH_REGEXP", analyzer);
}
@Override
- public MatchRegexp withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 2);
- return new MatchRegexp(children);
+ protected MatchRegexp createInstance(List<Expression> children, String analyzer) {
+ return new MatchRegexp(children, analyzer);
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java
index 8509a4e5fd8..e8676623c6a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java
@@ -367,4 +367,11 @@ public class IndexDefinition {
|| properties.containsKey(InvertedIndexUtil.INVERTED_INDEX_ANALYZER_NAME_KEY)
|| properties.containsKey(InvertedIndexUtil.INVERTED_INDEX_NORMALIZER_NAME_KEY));
}
+
+ public String getAnalyzerIdentity() {
+ if (indexType != IndexType.INVERTED) {
+ return "";
+ }
+ return InvertedIndexUtil.buildAnalyzerIdentity(properties);
+ }
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/alter/SchemaChangeHandlerTest.java b/fe/fe-core/src/test/java/org/apache/doris/alter/SchemaChangeHandlerTest.java
index 05bea450c0b..a7a311ae140 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/alter/SchemaChangeHandlerTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/alter/SchemaChangeHandlerTest.java
@@ -833,7 +833,7 @@ public class SchemaChangeHandlerTest extends TestWithFeService {
} catch (Exception e) {
// Verify the error message contains relevant info
Assertions.assertTrue(e.getMessage().contains("INVERTED index for column (error_msg) "
- + "with non-analyzed type already exists"));
+ + "with analyzer default analyzer already exists"));
}
addInvertedIndexStmtStr = "alter table test.sc_dup add index idx_error_msg(error_msg), "
+ "add index idx_error_msg(error_msg)";
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilderTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilderTest.java
new file mode 100644
index 00000000000..e264b9831bb
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilderTest.java
@@ -0,0 +1,104 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis.invertedindex;
+
+import org.apache.doris.indexpolicy.IndexPolicy;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+public class AnalyzerIdentityBuilderTest {
+
+ private Map<String, String> nonEmptyProperties() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("k", "v");
+ return properties;
+ }
+
+ @Test
+ public void testEmptyPropertiesReturnsDefault() {
+ String identity = AnalyzerIdentityBuilder.buildAnalyzerIdentity(
+ new HashMap<>(),
+ "",
+ "",
+ "__default__",
+ "none",
+ null);
+ Assertions.assertEquals("__default__", identity);
+ }
+
+ @Test
+ public void testBuiltInAnalyzerPreferred() {
+ Assertions.assertFalse(IndexPolicy.BUILTIN_ANALYZERS.isEmpty());
+ Iterator<String> iterator = IndexPolicy.BUILTIN_ANALYZERS.iterator();
+ String analyzer = iterator.next();
+
+ String identity = AnalyzerIdentityBuilder.buildAnalyzerIdentity(
+ nonEmptyProperties(),
+ analyzer,
+ "",
+ "__default__",
+ "none",
+ null);
+ Assertions.assertEquals(analyzer, identity);
+ }
+
+ @Test
+ public void testBuiltInNormalizerPreferred() {
+ Assertions.assertFalse(IndexPolicy.BUILTIN_NORMALIZERS.isEmpty());
+ Iterator<String> iterator = IndexPolicy.BUILTIN_NORMALIZERS.iterator();
+ String normalizer = iterator.next();
+
+ String identity = AnalyzerIdentityBuilder.buildAnalyzerIdentity(
+ nonEmptyProperties(),
+ normalizer,
+ "",
+ "__default__",
+ "none",
+ null);
+ Assertions.assertEquals("normalizer:" + normalizer, identity);
+ }
+
+ @Test
+ public void testParserNoneReturnsDefault() {
+ String identity = AnalyzerIdentityBuilder.buildAnalyzerIdentity(
+ nonEmptyProperties(),
+ "",
+ "none",
+ "__default__",
+ "none",
+ null);
+ Assertions.assertEquals("__default__", identity);
+ }
+
+ @Test
+ public void testParserReturnsParserName() {
+ String identity = AnalyzerIdentityBuilder.buildAnalyzerIdentity(
+ nonEmptyProperties(),
+ "",
+ "standard",
+ "__default__",
+ "none",
+ null);
+ Assertions.assertEquals("standard", identity);
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerKeyNormalizerTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerKeyNormalizerTest.java
new file mode 100644
index 00000000000..ed7ec7f1caa
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerKeyNormalizerTest.java
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis.invertedindex;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class AnalyzerKeyNormalizerTest {
+
+ @Test
+ public void testNullPropertiesNoOp() {
+ // Should not throw when properties is null
+ AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(null, "key");
+ }
+
+ @Test
+ public void testNullKeysNoOp() {
+ Map<String, String> props = new HashMap<>();
+ props.put("analyzer", "CHINESE");
+ // Should not throw when keys is null
+ AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(props, (String[]) null);
+ // Value should remain unchanged
+ Assertions.assertEquals("CHINESE", props.get("analyzer"));
+ }
+
+ @Test
+ public void testNormalizesToLowercase() {
+ Map<String, String> props = new HashMap<>();
+ props.put("analyzer", "CHINESE");
+ props.put("parser", "STANDARD");
+ AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(props, "analyzer", "parser");
+ Assertions.assertEquals("chinese", props.get("analyzer"));
+ Assertions.assertEquals("standard", props.get("parser"));
+ }
+
+ @Test
+ public void testTrimsWhitespace() {
+ Map<String, String> props = new HashMap<>();
+ props.put("parser", " standard ");
+ AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(props, "parser");
+ Assertions.assertEquals("standard", props.get("parser"));
+ }
+
+ @Test
+ public void testMixedCaseAndWhitespace() {
+ Map<String, String> props = new HashMap<>();
+ props.put("analyzer", " ChInEsE ");
+ AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(props, "analyzer");
+ Assertions.assertEquals("chinese", props.get("analyzer"));
+ }
+
+ @Test
+ public void testEmptyValueUnchanged() {
+ Map<String, String> props = new HashMap<>();
+ props.put("analyzer", "");
+ AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(props, "analyzer");
+ Assertions.assertEquals("", props.get("analyzer"));
+ }
+
+ @Test
+ public void testMissingKeyNoOp() {
+ Map<String, String> props = new HashMap<>();
+ props.put("other_key", "VALUE");
+ // Normalizing a key that doesn't exist should not throw
+ AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(props, "analyzer");
+ // Other keys should remain unchanged
+ Assertions.assertEquals("VALUE", props.get("other_key"));
+ Assertions.assertNull(props.get("analyzer"));
+ }
+
+ @Test
+ public void testMultipleKeysPartialMatch() {
+ Map<String, String> props = new HashMap<>();
+ props.put("analyzer", "CHINESE");
+ // parser key doesn't exist
+ AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(props, "analyzer", "parser");
+ Assertions.assertEquals("chinese", props.get("analyzer"));
+ Assertions.assertNull(props.get("parser"));
+ }
+
+ @Test
+ public void testNullKeyInArray() {
+ Map<String, String> props = new HashMap<>();
+ props.put("analyzer", "CHINESE");
+ // Should handle null key in the array gracefully
+ AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(props, "analyzer", null, "parser");
+ Assertions.assertEquals("chinese", props.get("analyzer"));
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/InvertedIndexSqlGeneratorTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/InvertedIndexSqlGeneratorTest.java
new file mode 100644
index 00000000000..e42a4ad7f5f
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/InvertedIndexSqlGeneratorTest.java
@@ -0,0 +1,47 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis.invertedindex;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class InvertedIndexSqlGeneratorTest {
+
+ @Test
+ public void testNullOrBlankAnalyzer() {
+ Assertions.assertEquals("", InvertedIndexSqlGenerator.buildAnalyzerSqlFragment(null));
+ Assertions.assertEquals("", InvertedIndexSqlGenerator.buildAnalyzerSqlFragment(""));
+ Assertions.assertEquals("", InvertedIndexSqlGenerator.buildAnalyzerSqlFragment(" "));
+ }
+
+ @Test
+ public void testIdentifierAnalyzer() {
+ Assertions.assertEquals(" USING ANALYZER standard",
+ InvertedIndexSqlGenerator.buildAnalyzerSqlFragment("standard"));
+ Assertions.assertEquals(" USING ANALYZER foo_bar",
+ InvertedIndexSqlGenerator.buildAnalyzerSqlFragment("foo_bar"));
+ }
+
+ @Test
+ public void testQuotedAnalyzer() {
+ Assertions.assertEquals(" USING ANALYZER 'foo bar'",
+ InvertedIndexSqlGenerator.buildAnalyzerSqlFragment("foo bar"));
+ Assertions.assertEquals(" USING ANALYZER 'O''Reilly'",
+ InvertedIndexSqlGenerator.buildAnalyzerSqlFragment("O'Reilly"));
+ }
+}
diff --git a/regression-test/data/inverted_index_p0/test_match_using_analyzer.out b/regression-test/data/inverted_index_p0/test_match_using_analyzer.out
new file mode 100644
index 00000000000..64d73c32909
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_match_using_analyzer.out
@@ -0,0 +1,13 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !match_default --
+1
+2
+
+-- !match_keyword --
+1
+
+-- !match_edge --
+1
+2
+4
+
diff --git a/regression-test/data/inverted_index_p0/test_multi_analyzer_column_types.out b/regression-test/data/inverted_index_p0/test_multi_analyzer_column_types.out
new file mode 100644
index 00000000000..561ff623ab4
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_multi_analyzer_column_types.out
@@ -0,0 +1,36 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !char_std --
+1
+2
+
+-- !char_kw --
+1
+
+-- !varchar_std --
+1
+2
+
+-- !varchar_kw --
+1
+
+-- !text_std --
+1
+2
+
+-- !text_kw --
+1
+
+-- !string_std --
+1
+2
+
+-- !string_kw --
+1
+
+-- !variant_std --
+1
+2
+
+-- !variant_kw --
+1
+
diff --git a/regression-test/data/inverted_index_p0/test_multi_analyzer_dml.out b/regression-test/data/inverted_index_p0/test_multi_analyzer_dml.out
new file mode 100644
index 00000000000..cbd3bbdd4ca
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_multi_analyzer_dml.out
@@ -0,0 +1,24 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !insert_std --
+1
+2
+
+-- !insert_kw --
+1
+
+-- !before_update --
+1
+
+-- !after_update_old --
+
+-- !after_update_new --
+1
+
+-- !before_delete --
+1
+
+-- !after_delete --
+
+-- !batch_count --
+0
+
diff --git a/regression-test/data/inverted_index_p0/test_multi_analyzer_index_version.out b/regression-test/data/inverted_index_p0/test_multi_analyzer_index_version.out
new file mode 100644
index 00000000000..cb2e7294897
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_multi_analyzer_index_version.out
@@ -0,0 +1,15 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !v2_std --
+1
+2
+
+-- !v2_kw --
+1
+
+-- !v3_std --
+1
+2
+
+-- !v3_kw --
+1
+
diff --git a/regression-test/data/inverted_index_p0/test_multi_analyzer_partition.out b/regression-test/data/inverted_index_p0/test_multi_analyzer_partition.out
new file mode 100644
index 00000000000..75283ef7908
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_multi_analyzer_partition.out
@@ -0,0 +1,8 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !manual_std --
+1
+2
+
+-- !manual_kw --
+1
+
diff --git a/regression-test/data/inverted_index_p0/test_multi_analyzer_query.out b/regression-test/data/inverted_index_p0/test_multi_analyzer_query.out
new file mode 100644
index 00000000000..666344742e9
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_multi_analyzer_query.out
@@ -0,0 +1,40 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !match_default --
+1
+2
+
+-- !match_keyword --
+1
+
+-- !match_standard --
+1
+2
+
+-- !match_any --
+1
+2
+3
+
+-- !match_all --
+2
+
+-- !match_phrase --
+2
+
+-- !match_phrase_prefix --
+1
+2
+4
+
+-- !match_regexp --
+1
+2
+
+-- !match_case_insensitive --
+1
+
+-- !match_null --
+1
+2
+5
+
diff --git a/regression-test/data/inverted_index_p0/test_multi_analyzer_table_types.out b/regression-test/data/inverted_index_p0/test_multi_analyzer_table_types.out
new file mode 100644
index 00000000000..6026f4580b9
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_multi_analyzer_table_types.out
@@ -0,0 +1,15 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !dup_std --
+1
+3
+
+-- !dup_kw --
+1
+
+-- !mow_std --
+1
+3
+
+-- !mow_kw --
+1
+
diff --git a/regression-test/data/inverted_index_p0/test_multi_tokenize_index_not_built.out b/regression-test/data/inverted_index_p0/test_multi_tokenize_index_not_built.out
new file mode 100644
index 00000000000..ed35a2de06f
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_multi_tokenize_index_not_built.out
@@ -0,0 +1,26 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !standard_before --
+1
+2
+4
+
+-- !keyword_not_built --
+2
+
+-- !standard_after_add --
+1
+2
+4
+
+-- !keyword_after_build --
+2
+
+-- !full_string_keyword --
+1
+
+-- !full_string_standard --
+1
+2
+3
+4
+
diff --git a/regression-test/suites/inverted_index_p0/test_match_using_analyzer.groovy b/regression-test/suites/inverted_index_p0/test_match_using_analyzer.groovy
new file mode 100644
index 00000000000..55897ed293e
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_match_using_analyzer.groovy
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_match_using_analyzer", "p0") {
+ def tableName = "tbl_match_using_analyzer"
+ def tokenizerName = "edge_ngram_name_tokenizer_match_test"
+ def analyzerStandard = "first_name_standard_match_test"
+ def analyzerEdge = "edge_ngram_name_match_test"
+ def analyzerKeyword = "first_name_keyword_match_test"
+
+ def waitAnalyzerReady = { analyzerName ->
+ sleep(10000)
+ int maxRetry = 30
+ boolean ready = false
+ Exception lastException = null
+ for (int i = 0; i < maxRetry && !ready; i++) {
+ try {
+ sql """ select tokenize("probe", '"analyzer"="${analyzerName}"'); """
+ ready = true
+ } catch (Exception e) {
+ lastException = e
+ logger.info("Analyzer ${analyzerName} not ready yet: ${e.message}")
+ sql """ select sleep(1) """
+ }
+ }
+ assertTrue(ready, "Analyzer ${analyzerName} not ready after waiting: ${lastException?.message}")
+ }
+
+ try {
+ sql "DROP TABLE IF EXISTS ${tableName}"
+
+ sql """
+ CREATE INVERTED INDEX TOKENIZER IF NOT EXISTS ${tokenizerName}
+ PROPERTIES
+ (
+ "type" = "edge_ngram",
+ "min_gram" = "1",
+ "max_gram" = "20",
+ "token_chars" = "letter"
+ );
+ """
+
+ sql """
+ CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerStandard}
+ PROPERTIES
+ (
+ "tokenizer" = "standard",
+ "token_filter" = "lowercase"
+ );
+ """
+
+ sql """
+ CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerEdge}
+ PROPERTIES
+ (
+ "tokenizer" = "${tokenizerName}",
+ "token_filter" = "lowercase"
+ );
+ """
+
+ sql """
+ CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerKeyword}
+ PROPERTIES
+ (
+ "tokenizer" = "keyword",
+ "token_filter" = "lowercase"
+ );
+ """
+
+ //waitAnalyzerReady(analyzerStandard)
+ //waitAnalyzerReady(analyzerEdge)
+ //waitAnalyzerReady(analyzerKeyword)
+
+ sql """
+ CREATE TABLE ${tableName} (
+ id INT,
+ first_name STRING,
+ INDEX idx_fn_std (first_name) USING INVERTED PROPERTIES("analyzer"="${analyzerStandard}", "support_phrase"="true"),
+ INDEX idx_fn_ngr (first_name) USING INVERTED PROPERTIES("analyzer"="${analyzerEdge}"),
+ INDEX idx_fn_exact (first_name) USING INVERTED PROPERTIES("analyzer"="${analyzerKeyword}")
+ ) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sleep(10000)
+ sql """
+ INSERT INTO ${tableName} VALUES
+ (1, 'alice'),
+ (2, 'alice cooper'),
+ (3, 'bob'),
+ (4, 'allyson');
+ """
+
+ sql "sync"
+
+ // default analyzer should match both tokenized rows that contain "alice"
+ qt_match_default """
+ SELECT id FROM ${tableName}
+ WHERE first_name MATCH 'alice'
+ ORDER BY id
+ """
+
+ // keyword analyzer only matches the full literal "alice"
+ qt_match_keyword """
+ SELECT id FROM ${tableName}
+ WHERE first_name MATCH 'alice' USING ANALYZER ${analyzerKeyword}
+ ORDER BY id
+ """
+
+ // edge ngram analyzer should cover names beginning with "al"
+ qt_match_edge """
+ SELECT id FROM ${tableName}
+ WHERE first_name MATCH_PHRASE_PREFIX 'al' USING ANALYZER ${analyzerEdge}
+ ORDER BY id
+ """
+
+ // show create table should contain analyzer names
+ def showCreate = sql """ SHOW CREATE TABLE ${tableName} """
+ def createStmt = showCreate.collect { row -> row[1].toString().toLowerCase() }.join("\n")
+ assertTrue(createStmt.contains(analyzerStandard.toLowerCase()))
+ assertTrue(createStmt.contains(analyzerEdge.toLowerCase()))
+ assertTrue(createStmt.contains(analyzerKeyword.toLowerCase()))
+ } finally {
+ /*sql "DROP TABLE IF EXISTS ${tableName}"
+
+ try {
+ sql "DROP INVERTED INDEX ANALYZER ${analyzerStandard}"
+ } catch (Exception ignored) {
+ }
+
+ try {
+ sql "DROP INVERTED INDEX ANALYZER ${analyzerEdge}"
+ } catch (Exception ignored) {
+ }
+
+ try {
+ sql "DROP INVERTED INDEX ANALYZER ${analyzerKeyword}"
+ } catch (Exception ignored) {
+ }
+
+ try {
+ sql "DROP INVERTED INDEX TOKENIZER ${tokenizerName}"
+ } catch (Exception ignored) {
+ }*/
+ }
+}
+
diff --git a/regression-test/suites/inverted_index_p0/test_multi_analyzer_column_types.groovy b/regression-test/suites/inverted_index_p0/test_multi_analyzer_column_types.groovy
new file mode 100644
index 00000000000..359d2a5a4d7
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_multi_analyzer_column_types.groovy
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_multi_analyzer_column_types", "p0") {
+ // Test multi-analyzer indexes on different column types: char, varchar, text, variant
+
+ def analyzerStd = "multi_col_type_std_analyzer"
+ def analyzerKw = "multi_col_type_kw_analyzer"
+
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerStd} PROPERTIES ("tokenizer" = "standard", "token_filter" = "lowercase");"""
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerKw} PROPERTIES ("tokenizer" = "keyword", "token_filter" = "lowercase");"""
+ sleep(10000)
+
+ // TC-COL-001: CHAR column
+ def tblChar = "test_multi_analyzer_char"
+ sql "DROP TABLE IF EXISTS ${tblChar}"
+ sql """CREATE TABLE ${tblChar} (id INT, name CHAR(50), INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """INSERT INTO ${tblChar} VALUES (1, 'alice'), (2, 'alice cooper'), (3, 'bob');"""
+ sql "sync"
+ qt_char_std """SELECT id FROM ${tblChar} WHERE name MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_char_kw """SELECT id FROM ${tblChar} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // TC-COL-002: VARCHAR column
+ def tblVarchar = "test_multi_analyzer_varchar"
+ sql "DROP TABLE IF EXISTS ${tblVarchar}"
+ sql """CREATE TABLE ${tblVarchar} (id INT, name VARCHAR(100), INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """INSERT INTO ${tblVarchar} VALUES (1, 'alice'), (2, 'alice cooper'), (3, 'bob');"""
+ sql "sync"
+ qt_varchar_std """SELECT id FROM ${tblVarchar} WHERE name MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_varchar_kw """SELECT id FROM ${tblVarchar} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // TC-COL-003: TEXT column
+ def tblText = "test_multi_analyzer_text"
+ sql "DROP TABLE IF EXISTS ${tblText}"
+ sql """CREATE TABLE ${tblText} (id INT, content TEXT, INDEX idx_std (content) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (content) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """INSERT INTO ${tblText} VALUES (1, 'alice'), (2, 'alice cooper'), (3, 'bob');"""
+ sql "sync"
+ qt_text_std """SELECT id FROM ${tblText} WHERE content MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_text_kw """SELECT id FROM ${tblText} WHERE content MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // TC-COL-004: STRING column
+ def tblString = "test_multi_analyzer_string"
+ sql "DROP TABLE IF EXISTS ${tblString}"
+ sql """CREATE TABLE ${tblString} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """INSERT INTO ${tblString} VALUES (1, 'alice'), (2, 'alice cooper'), (3, 'bob');"""
+ sql "sync"
+ qt_string_std """SELECT id FROM ${tblString} WHERE name MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_string_kw """SELECT id FROM ${tblString} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // TC-COL-005: VARIANT column with field_pattern
+ // VARIANT column needs typed path definition to use field_pattern
+ def tblVariant = "test_multi_analyzer_variant"
+ sql "DROP TABLE IF EXISTS ${tblVariant}"
+ sql """CREATE TABLE ${tblVariant} (
+ id INT,
+ data VARIANT<'name_*' : string>,
+ INDEX idx_std (data) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}", "field_pattern"="name_*"),
+ INDEX idx_kw (data) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}", "field_pattern"="name_*")
+ ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """INSERT INTO ${tblVariant} VALUES (1, '{"name_1":"alice"}'), (2, '{"name_1":"alice cooper"}'), (3, '{"name_1":"bob"}');"""
+ sql "sync"
+ qt_variant_std """SELECT id FROM ${tblVariant} WHERE cast(data['name_1'] as varchar) MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_variant_kw """SELECT id FROM ${tblVariant} WHERE cast(data['name_1'] as varchar) MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // TC-COL-006: Non-string column should only allow one index
+ def tblInt = "test_multi_analyzer_int"
+ sql "DROP TABLE IF EXISTS ${tblInt}"
+ try {
+ sql """CREATE TABLE ${tblInt} (id INT, v INT, INDEX i1 (v) USING INVERTED, INDEX i2 (v) USING INVERTED) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ assertTrue(false, "Should fail for INT type")
+ } catch (Exception e) {
+ logger.info("Expected error for INT: ${e.message}")
+ }
+
+ // Cleanup
+ //sql "DROP TABLE IF EXISTS ${tblChar}"
+ //sql "DROP TABLE IF EXISTS ${tblVarchar}"
+ //sql "DROP TABLE IF EXISTS ${tblText}"
+ //sql "DROP TABLE IF EXISTS ${tblString}"
+ //sql "DROP TABLE IF EXISTS ${tblVariant}"
+ //sql "DROP TABLE IF EXISTS ${tblInt}"
+}
diff --git a/regression-test/suites/inverted_index_p0/test_multi_analyzer_ddl.groovy b/regression-test/suites/inverted_index_p0/test_multi_analyzer_ddl.groovy
new file mode 100644
index 00000000000..a44a114159d
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_multi_analyzer_ddl.groovy
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_multi_analyzer_ddl", "p0") {
+ def analyzerStd = "multi_ddl_test_std_analyzer"
+ def analyzerKw = "multi_ddl_test_kw_analyzer"
+ def timeout = 60000
+ def delta_time = 1000
+
+ def wait_for_latest_op_on_table_finish = { table_name, OpTimeout ->
+ def useTime = 0
+ for (int t = delta_time; t <= OpTimeout; t += delta_time) {
+ def alter_res = sql """SHOW ALTER TABLE COLUMN WHERE TableName = "${table_name}" ORDER BY CreateTime DESC LIMIT 1;"""
+ if (alter_res.toString().contains("FINISHED")) {
+ sleep(3000)
+ break
+ }
+ useTime = t
+ sleep(delta_time)
+ }
+ assertTrue(useTime <= OpTimeout)
+ }
+
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerStd} PROPERTIES ("tokenizer" = "standard", "token_filter" = "lowercase");"""
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerKw} PROPERTIES ("tokenizer" = "keyword", "token_filter" = "lowercase");"""
+ sleep(10000)
+
+ // TC-DDL-001: Create table with multiple analyzers
+ def tbl1 = "test_multi_analyzer_ddl_001"
+ sql "DROP TABLE IF EXISTS ${tbl1}"
+ sql """CREATE TABLE ${tbl1} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ def res1 = sql """ SHOW CREATE TABLE ${tbl1} """
+ assertTrue(res1.collect { it[1].toString().toLowerCase() }.join("\n").contains(analyzerStd.toLowerCase()))
+ assertTrue(res1.collect { it[1].toString().toLowerCase() }.join("\n").contains(analyzerKw.toLowerCase()))
+
+ // TC-DDL-002: Duplicate analyzer identity should fail
+ def tbl2 = "test_multi_analyzer_ddl_002"
+ sql "DROP TABLE IF EXISTS ${tbl2}"
+ try {
+ sql """CREATE TABLE ${tbl2} (id INT, v STRING, INDEX i1 (v) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX i2 (v) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ assertTrue(false, "Should fail with duplicate analyzer")
+ } catch (Exception e) {
+ // Error message: "column: xxx cannot have multiple inverted indexes of the same type."
+ assertTrue(e.message.contains("multiple inverted indexes") || e.message.contains("same type") || e.message.contains("analyzer identity") || e.message.contains("already exists"))
+ }
+
+ // TC-DDL-003: Non-string type multiple indexes should fail
+ def tbl3 = "test_multi_analyzer_ddl_003"
+ sql "DROP TABLE IF EXISTS ${tbl3}"
+ try {
+ sql """CREATE TABLE ${tbl3} (id INT, v INT, INDEX i1 (v) USING INVERTED, INDEX i2 (v) USING INVERTED) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ assertTrue(false, "Should fail for INT type")
+ } catch (Exception e) {
+ logger.info("Expected error: ${e.message}")
+ }
+
+ // TC-DDL-004: ALTER TABLE ADD INDEX
+ def tbl4 = "test_multi_analyzer_ddl_004"
+ sql "DROP TABLE IF EXISTS ${tbl4}"
+ sql """CREATE TABLE ${tbl4} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """ ALTER TABLE ${tbl4} ADD INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}"); """
+ wait_for_latest_op_on_table_finish(tbl4, timeout)
+ def res4 = sql """ SHOW CREATE TABLE ${tbl4} """
+ assertTrue(res4.collect { it[1].toString().toLowerCase() }.join("\n").contains(analyzerKw.toLowerCase()))
+
+ // TC-DDL-005: DROP INDEX
+ def tbl5 = "test_multi_analyzer_ddl_005"
+ sql "DROP TABLE IF EXISTS ${tbl5}"
+ sql """CREATE TABLE ${tbl5} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """ DROP INDEX idx_std ON ${tbl5}; """
+ wait_for_latest_op_on_table_finish(tbl5, timeout)
+ def res5 = sql """ SHOW CREATE TABLE ${tbl5} """
+ assertFalse(res5.collect { it[1].toString().toLowerCase() }.join("\n").contains("idx_std"))
+ assertTrue(res5.collect { it[1].toString().toLowerCase() }.join("\n").contains("idx_kw"))
+
+ // TC-DDL-006: Using built-in parser
+ def tbl6 = "test_multi_analyzer_ddl_006"
+ sql "DROP TABLE IF EXISTS ${tbl6}"
+ sql """CREATE TABLE ${tbl6} (id INT, content STRING, INDEX idx_cn (content) USING INVERTED PROPERTIES("parser"="chinese"), INDEX idx_en (content) USING INVERTED PROPERTIES("parser"="english"), INDEX idx_none (content) USING INVERTED) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ def res6 = sql """ SHOW CREATE TABLE ${tbl6} """
+ assertTrue(res6.collect { it[1].toString().toLowerCase() }.join("\n").contains("chinese"))
+ assertTrue(res6.collect { it[1].toString().toLowerCase() }.join("\n").contains("english"))
+
+ // TC-DDL-007: Different analyzer names but same identity (properties order different but semantically same)
+ // analyzer_order_a: {"tokenizer":"standard","token_filter":"lowercase"}
+ // analyzer_order_b: {"token_filter":"lowercase","tokenizer":"standard"}
+ // They have the same semantic identity, should fail when creating table with both
+ def analyzerOrderA = "analyzer_order_a_ddl_test"
+ def analyzerOrderB = "analyzer_order_b_ddl_test"
+
+ // Create two analyzers with different property order but same semantics
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerOrderA} PROPERTIES ("tokenizer" = "standard", "token_filter" = "lowercase");"""
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerOrderB} PROPERTIES ("token_filter" = "lowercase", "tokenizer" = "standard");"""
+ sleep(10000)
+
+ def tbl7 = "test_multi_analyzer_same_identity_diff_order"
+ sql "DROP TABLE IF EXISTS ${tbl7}"
+ def tbl7CreateSuccess = false
+ try {
+ // Both analyzers have the same identity (standard + lowercase), should fail
+ sql """CREATE TABLE ${tbl7} (
+ k1 INT,
+ v1 TEXT,
+ INDEX idx_a (v1) USING INVERTED PROPERTIES("analyzer" = "${analyzerOrderA}", "support_phrase" = "true"),
+ INDEX idx_b (v1) USING INVERTED PROPERTIES("analyzer" = "${analyzerOrderB}", "support_phrase" = "true")
+ ) DUPLICATE KEY(k1) DISTRIBUTED BY HASH(k1) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ // If we reach here, it means the table was created successfully - this is unexpected
+ tbl7CreateSuccess = true
+ def res7 = sql """ SHOW CREATE TABLE ${tbl7} """
+ logger.info("SHOW CREATE TABLE result: ${res7}")
+ } catch (Exception e) {
+ // Expected: should fail because both analyzers have the same identity
+ logger.info("Expected error for same identity analyzers: ${e.message}")
+ assertTrue(e.message.contains("multiple inverted indexes") || e.message.contains("same type") || e.message.contains("analyzer identity") || e.message.contains("already exists"))
+ }
+ // Verify: table creation should fail because both analyzers have same identity
+ assertFalse(tbl7CreateSuccess, "TC-DDL-007 FAILED: Table should NOT be created when two analyzers have same identity (same tokenizer+filter but different property order)")
+
+ // TC-DDL-008: Different analyzer names pointing to same tokenizer + filter combination
+ // When two different analyzer names resolve to same tokenizer + filter, they should be considered same identity
+ def tbl8 = "test_multi_analyzer_diff_name_same_combo"
+ sql "DROP TABLE IF EXISTS ${tbl8}"
+ def tbl8CreateSuccess = false
+ // Both analyzerStd and analyzerOrderA have same tokenizer(standard) + token_filter(lowercase)
+ try {
+ sql """CREATE TABLE ${tbl8} (
+ k1 INT,
+ v1 TEXT,
+ INDEX idx_a (v1) USING INVERTED PROPERTIES("analyzer" = "${analyzerStd}", "support_phrase" = "true"),
+ INDEX idx_b (v1) USING INVERTED PROPERTIES("analyzer" = "${analyzerOrderA}", "support_phrase" = "true")
+ ) DUPLICATE KEY(k1) DISTRIBUTED BY HASH(k1) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ // If we reach here, it means the table was created successfully - this is unexpected
+ tbl8CreateSuccess = true
+ def res8 = sql """ SHOW CREATE TABLE ${tbl8} """
+ logger.info("SHOW CREATE TABLE result: ${res8}")
+ } catch (Exception e) {
+ logger.info("Expected error for same tokenizer+filter combination: ${e.message}")
+ assertTrue(e.message.contains("multiple inverted indexes") || e.message.contains("same type") || e.message.contains("analyzer identity") || e.message.contains("already exists"))
+ }
+ // Verify: table creation should fail because both analyzers have same identity
+ assertFalse(tbl8CreateSuccess, "TC-DDL-008 FAILED: Table should NOT be created when two different analyzer names have same tokenizer+filter combination")
+
+ // TC-DDL-009: Drop one index should not affect other indexes on the same column
+ // This test case verifies the bug fix: when dropping one inverted index,
+ // other indexes on the same column should NOT be deleted
+ def tbl9 = "test_multi_analyzer_drop_one_index"
+ sql "DROP TABLE IF EXISTS ${tbl9}"
+ sql """CREATE TABLE ${tbl9} (
+ id INT,
+ title VARCHAR(512) NOT NULL,
+ INDEX idx_std (title) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}", "support_phrase" = "true"),
+ INDEX idx_kw (title) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}", "support_phrase" = "true")
+ ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+
+ // Insert some test data
+ sql """ INSERT INTO ${tbl9} VALUES (1, 'Hello World Test'); """
+ sql """ INSERT INTO ${tbl9} VALUES (2, 'Apache Doris Database'); """
+ sql """ INSERT INTO ${tbl9} VALUES (3, 'Inverted Index Feature'); """
+
+ // Verify both indexes exist
+ def res9_before = sql """ SHOW CREATE TABLE ${tbl9} """
+ def schema9_before = res9_before.collect { it[1].toString().toLowerCase() }.join("\n")
+ assertTrue(schema9_before.contains("idx_std"))
+ assertTrue(schema9_before.contains("idx_kw"))
+
+ // Drop one index (idx_std)
+ sql """ DROP INDEX idx_std ON ${tbl9}; """
+ wait_for_latest_op_on_table_finish(tbl9, timeout)
+
+ // Verify: idx_std should be dropped, but idx_kw should still exist
+ def res9_after = sql """ SHOW CREATE TABLE ${tbl9} """
+ def schema9_after = res9_after.collect { it[1].toString().toLowerCase() }.join("\n")
+ assertFalse(schema9_after.contains("idx_std"), "TC-DDL-009 FAILED: idx_std should be dropped")
+ assertTrue(schema9_after.contains("idx_kw"), "TC-DDL-009 FAILED: idx_kw should still exist after dropping idx_std")
+
+ // Verify: idx_kw index should still work for queries
+ def query_res = sql """ SELECT * FROM ${tbl9} WHERE title MATCH_PHRASE 'hello world'; """
+ logger.info("Query result using remaining index: ${query_res}")
+
+ // Cleanup
+ //sql "DROP TABLE IF EXISTS ${tbl9}"
+
+ // Cleanup
+ /*sql "DROP TABLE IF EXISTS ${tbl1}"
+ sql "DROP TABLE IF EXISTS ${tbl2}"
+ sql "DROP TABLE IF EXISTS ${tbl3}"
+ sql "DROP TABLE IF EXISTS ${tbl4}"
+ sql "DROP TABLE IF EXISTS ${tbl5}"
+ sql "DROP TABLE IF EXISTS ${tbl6}"
+ sql "DROP TABLE IF EXISTS ${tbl7}"
+ sql "DROP TABLE IF EXISTS ${tbl8}"*/
+}
diff --git a/regression-test/suites/inverted_index_p0/test_multi_analyzer_dml.groovy b/regression-test/suites/inverted_index_p0/test_multi_analyzer_dml.groovy
new file mode 100644
index 00000000000..3ae302f4cb7
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_multi_analyzer_dml.groovy
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_multi_analyzer_dml", "p0") {
+ // Test DML operations with multi-analyzer indexes
+
+ def analyzerStd = "multi_dml_std_analyzer"
+ def analyzerKw = "multi_dml_kw_analyzer"
+
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerStd} PROPERTIES ("tokenizer" = "standard", "token_filter" = "lowercase");"""
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerKw} PROPERTIES ("tokenizer" = "keyword", "token_filter" = "lowercase");"""
+ sleep(10000)
+
+ // Test INSERT with multi-analyzer indexes
+ def tbl1 = "test_multi_analyzer_dml_insert"
+ sql "DROP TABLE IF EXISTS ${tbl1}"
+ sql """CREATE TABLE ${tbl1} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """INSERT INTO ${tbl1} VALUES (1, 'alice'), (2, 'alice cooper'), (3, 'bob');"""
+ sql "sync"
+ qt_insert_std """SELECT id FROM ${tbl1} WHERE name MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_insert_kw """SELECT id FROM ${tbl1} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // Test UPDATE with multi-analyzer indexes (MOW table)
+ def tbl2 = "test_multi_analyzer_dml_update"
+ sql "DROP TABLE IF EXISTS ${tbl2}"
+ sql """CREATE TABLE ${tbl2} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) UNIQUE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1", "enable_unique_key_merge_on_write" = "true");"""
+ sql """INSERT INTO ${tbl2} VALUES (1, 'alice'), (2, 'bob');"""
+ sql "sync"
+ qt_before_update """SELECT id FROM ${tbl2} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+ sql """UPDATE ${tbl2} SET name = 'alice updated' WHERE id = 1;"""
+ sql "sync"
+ qt_after_update_old """SELECT id FROM ${tbl2} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+ qt_after_update_new """SELECT id FROM ${tbl2} WHERE name MATCH 'alice updated' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // Test DELETE with multi-analyzer indexes (MOW table)
+ def tbl3 = "test_multi_analyzer_dml_delete"
+ sql "DROP TABLE IF EXISTS ${tbl3}"
+ sql """CREATE TABLE ${tbl3} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) UNIQUE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1", "enable_unique_key_merge_on_write" = "true");"""
+ sql """INSERT INTO ${tbl3} VALUES (1, 'alice'), (2, 'bob'), (3, 'carol');"""
+ sql "sync"
+ qt_before_delete """SELECT id FROM ${tbl3} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+ sql """DELETE FROM ${tbl3} WHERE id = 1;"""
+ sql "sync"
+ qt_after_delete """SELECT id FROM ${tbl3} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // Test batch INSERT
+ def tbl4 = "test_multi_analyzer_dml_batch"
+ sql "DROP TABLE IF EXISTS ${tbl4}"
+ sql """CREATE TABLE ${tbl4} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ for (int i = 1; i <= 100; i++) {
+ sql """INSERT INTO ${tbl4} VALUES (${i}, 'alice_${i}');"""
+ }
+ sql "sync"
+ qt_batch_count """SELECT count(*) FROM ${tbl4} WHERE name MATCH 'alice' USING ANALYZER ${analyzerStd}"""
+
+ // Cleanup
+ sql "DROP TABLE IF EXISTS ${tbl1}"
+ sql "DROP TABLE IF EXISTS ${tbl2}"
+ sql "DROP TABLE IF EXISTS ${tbl3}"
+ sql "DROP TABLE IF EXISTS ${tbl4}"
+}
diff --git a/regression-test/suites/inverted_index_p0/test_multi_analyzer_index_version.groovy b/regression-test/suites/inverted_index_p0/test_multi_analyzer_index_version.groovy
new file mode 100644
index 00000000000..d2243e315c9
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_multi_analyzer_index_version.groovy
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_multi_analyzer_index_version", "p0") {
+ // Test multi-analyzer indexes with different index format versions
+
+ def analyzerStd = "multi_ver_std_analyzer"
+ def analyzerKw = "multi_ver_kw_analyzer"
+
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerStd} PROPERTIES ("tokenizer" = "standard", "token_filter" = "lowercase");"""
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerKw} PROPERTIES ("tokenizer" = "keyword", "token_filter" = "lowercase");"""
+ sleep(10000)
+
+ // TC-VER-001: Index format V1 should NOT support multi-analyzer indexes
+ def tblV1 = "test_multi_analyzer_v1"
+ sql "DROP TABLE IF EXISTS ${tblV1}"
+ try {
+ sql """CREATE TABLE ${tblV1} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1", "inverted_index_storage_format" = "V1");"""
+ assertTrue(false, "V1 format should not support multi-analyzer indexes")
+ } catch (Exception e) {
+ logger.info("Expected error for V1 format: ${e.message}")
+ assertTrue(e.message.contains("V1") || e.message.contains("not supported") || e.message.contains("format"))
+ }
+
+ // TC-VER-002: Index format V2 should support multi-analyzer indexes
+ def tblV2 = "test_multi_analyzer_v2"
+ sql "DROP TABLE IF EXISTS ${tblV2}"
+ sql """CREATE TABLE ${tblV2} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1", "inverted_index_storage_format" = "V2");"""
+ sql """INSERT INTO ${tblV2} VALUES (1, 'alice'), (2, 'alice cooper'), (3, 'bob');"""
+ sql "sync"
+ qt_v2_std """SELECT id FROM ${tblV2} WHERE name MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_v2_kw """SELECT id FROM ${tblV2} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // TC-VER-003: Index format V3 should support multi-analyzer indexes
+ def tblV3 = "test_multi_analyzer_v3"
+ sql "DROP TABLE IF EXISTS ${tblV3}"
+ sql """CREATE TABLE ${tblV3} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1", "inverted_index_storage_format" = "V3");"""
+ sql """INSERT INTO ${tblV3} VALUES (1, 'alice'), (2, 'alice cooper'), (3, 'bob');"""
+ sql "sync"
+ qt_v3_std """SELECT id FROM ${tblV3} WHERE name MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_v3_kw """SELECT id FROM ${tblV3} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // Cleanup
+ sql "DROP TABLE IF EXISTS ${tblV1}"
+ sql "DROP TABLE IF EXISTS ${tblV2}"
+ sql "DROP TABLE IF EXISTS ${tblV3}"
+}
diff --git a/regression-test/suites/inverted_index_p0/test_multi_analyzer_partition.groovy b/regression-test/suites/inverted_index_p0/test_multi_analyzer_partition.groovy
new file mode 100644
index 00000000000..5eac697478f
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_multi_analyzer_partition.groovy
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_multi_analyzer_partition", "p0") {
+ def analyzerStd = "multi_part_std_analyzer"
+ def analyzerKw = "multi_part_kw_analyzer"
+ def timeout = 60000
+ def delta_time = 1000
+
+ def wait_for_latest_op_on_table_finish = { table_name, OpTimeout ->
+ def useTime = 0
+ for (int t = delta_time; t <= OpTimeout; t += delta_time) {
+ def alter_res = sql """SHOW ALTER TABLE COLUMN WHERE TableName = "${table_name}" ORDER BY CreateTime DESC LIMIT 1;"""
+ if (alter_res.toString().contains("FINISHED")) {
+ sleep(3000)
+ break
+ }
+ useTime = t
+ sleep(delta_time)
+ }
+ assertTrue(useTime <= OpTimeout)
+ }
+
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerStd} PROPERTIES ("tokenizer" = "standard", "token_filter" = "lowercase");"""
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerKw} PROPERTIES ("tokenizer" = "keyword", "token_filter" = "lowercase");"""
+ sleep(10000)
+
+ // Manual partition addition should inherit index properties
+ def tbl1 = "test_multi_analyzer_part_manual"
+ sql "DROP TABLE IF EXISTS ${tbl1}"
+ sql """CREATE TABLE ${tbl1} (id INT, dt DATE, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id, dt) PARTITION BY RANGE(dt) (PARTITION p20230101 VALUES [("2023-01-01"), ("2023-02-01"))) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """ALTER TABLE ${tbl1} ADD PARTITION p20230201 VALUES [("2023-02-01"), ("2023-03-01"));"""
+ wait_for_latest_op_on_table_finish(tbl1, timeout)
+ sql """INSERT INTO ${tbl1} VALUES (1, '2023-01-15', 'alice'), (2, '2023-02-15', 'alice cooper');"""
+ sql "sync"
+ qt_manual_std """SELECT id FROM ${tbl1} WHERE name MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_manual_kw """SELECT id FROM ${tbl1} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // Dynamic partition should inherit index properties
+ def tbl2 = "test_multi_analyzer_part_dynamic"
+ sql "DROP TABLE IF EXISTS ${tbl2}"
+ sql """CREATE TABLE ${tbl2} (id INT, dt DATE, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id, dt) PARTITION BY RANGE(dt) () DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1", "dynamic_partition.enable" = "true", "dynamic_partition.time_unit" = "DAY", "dynamic_partition.start" = "-3", "dynamic_partition. [...]
+ sleep(5000)
+ def res = sql """ SHOW CREATE TABLE ${tbl2} """
+ def createStmt = res.collect { it[1].toString().toLowerCase() }.join("\n")
+ assertTrue(createStmt.contains(analyzerStd.toLowerCase()))
+ assertTrue(createStmt.contains(analyzerKw.toLowerCase()))
+
+ sql "DROP TABLE IF EXISTS ${tbl1}"
+ sql "DROP TABLE IF EXISTS ${tbl2}"
+}
diff --git a/regression-test/suites/inverted_index_p0/test_multi_analyzer_query.groovy b/regression-test/suites/inverted_index_p0/test_multi_analyzer_query.groovy
new file mode 100644
index 00000000000..ccd3d27cad4
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_multi_analyzer_query.groovy
@@ -0,0 +1,83 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_multi_analyzer_query", "p0") {
+ // Test USING ANALYZER query syntax for multi-analyzer indexes
+
+ def analyzerStd = "multi_query_test_std_analyzer"
+ def analyzerKw = "multi_query_test_kw_analyzer"
+ def tableName = "test_multi_analyzer_query_tbl"
+
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerStd} PROPERTIES ("tokenizer" = "standard", "token_filter" = "lowercase");"""
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerKw} PROPERTIES ("tokenizer" = "keyword", "token_filter" = "lowercase");"""
+ sleep(10000)
+
+ sql "DROP TABLE IF EXISTS ${tableName}"
+ sql """
+ CREATE TABLE ${tableName} (
+ id INT, first_name STRING,
+ INDEX idx_std (first_name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}", "support_phrase"="true"),
+ INDEX idx_kw (first_name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")
+ ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+ sleep(10000)
+
+ sql """INSERT INTO ${tableName} VALUES (1, 'alice'), (2, 'alice cooper'), (3, 'bob'), (4, 'allyson'), (5, NULL);"""
+ sql "sync"
+
+ // TC-QUERY-001: Default analyzer query (should use analyzed index)
+ qt_match_default """SELECT id FROM ${tableName} WHERE first_name MATCH 'alice' ORDER BY id"""
+
+ // TC-QUERY-002: USING ANALYZER keyword for exact match
+ qt_match_keyword """SELECT id FROM ${tableName} WHERE first_name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // TC-QUERY-003: USING ANALYZER standard for tokenized search
+ qt_match_standard """SELECT id FROM ${tableName} WHERE first_name MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+
+ // TC-QUERY-004: Non-existent analyzer should report error
+ try {
+ sql """SELECT id FROM ${tableName} WHERE first_name MATCH 'alice' USING ANALYZER not_exist_analyzer"""
+ assertTrue(false, "Should fail with non-existent analyzer")
+ } catch (Exception e) {
+ assertTrue(e.message.contains("No inverted index found") || e.message.contains("not found"))
+ }
+
+ // TC-QUERY-005: MATCH_ANY with USING ANALYZER
+ qt_match_any """SELECT id FROM ${tableName} WHERE first_name MATCH_ANY 'alice bob' USING ANALYZER ${analyzerStd} ORDER BY id"""
+
+ // TC-QUERY-006: MATCH_ALL with USING ANALYZER
+ qt_match_all """SELECT id FROM ${tableName} WHERE first_name MATCH_ALL 'alice cooper' USING ANALYZER ${analyzerStd} ORDER BY id"""
+
+ // TC-QUERY-007: MATCH_PHRASE with USING ANALYZER
+ qt_match_phrase """SELECT id FROM ${tableName} WHERE first_name MATCH_PHRASE 'alice cooper' USING ANALYZER ${analyzerStd} ORDER BY id"""
+
+ // TC-QUERY-008: MATCH_PHRASE_PREFIX with USING ANALYZER
+ qt_match_phrase_prefix """SELECT id FROM ${tableName} WHERE first_name MATCH_PHRASE_PREFIX 'al' USING ANALYZER ${analyzerStd} ORDER BY id"""
+
+ // TC-QUERY-009: MATCH_REGEXP with USING ANALYZER
+ qt_match_regexp """SELECT id FROM ${tableName} WHERE first_name MATCH_REGEXP 'ali.*' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // TC-QUERY-010: Case insensitive analyzer name
+ qt_match_case_insensitive """SELECT id FROM ${tableName} WHERE first_name MATCH 'alice' USING ANALYZER ${analyzerKw.toUpperCase()} ORDER BY id"""
+
+ // TC-QUERY-011: Query with NULL values
+ qt_match_null """SELECT id FROM ${tableName} WHERE first_name MATCH 'alice' OR first_name IS NULL ORDER BY id"""
+
+ // Cleanup
+ sql "DROP TABLE IF EXISTS ${tableName}"
+}
diff --git a/regression-test/suites/inverted_index_p0/test_multi_analyzer_show.groovy b/regression-test/suites/inverted_index_p0/test_multi_analyzer_show.groovy
new file mode 100644
index 00000000000..0c539b1b676
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_multi_analyzer_show.groovy
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_multi_analyzer_show", "p0") {
+ // Test observation commands for multi-analyzer indexes
+
+ def analyzerStd = "multi_show_std_analyzer"
+ def analyzerKw = "multi_show_kw_analyzer"
+ def timeout = 60000
+ def delta_time = 1000
+
+ def wait_for_latest_op_on_table_finish = { table_name, OpTimeout ->
+ def useTime = 0
+ for (int t = delta_time; t <= OpTimeout; t += delta_time) {
+ def alter_res = sql """SHOW ALTER TABLE COLUMN WHERE TableName = "${table_name}" ORDER BY CreateTime DESC LIMIT 1;"""
+ if (alter_res.toString().contains("FINISHED")) {
+ sleep(3000)
+ break
+ }
+ useTime = t
+ sleep(delta_time)
+ }
+ assertTrue(useTime <= OpTimeout)
+ }
+
+ def wait_for_build_index_on_partition_finish = { table_name, OpTimeout ->
+ def useTime = 0
+ for (int t = delta_time; t <= OpTimeout; t += delta_time) {
+ def alter_res = sql """SHOW BUILD INDEX WHERE TableName = "${table_name}";"""
+ def expected_finished_num = alter_res.size()
+ def finished_num = 0
+ for (int i = 0; i < expected_finished_num; i++) {
+ if (alter_res[i][7] == "FINISHED") {
+ ++finished_num
+ }
+ }
+ if (finished_num == expected_finished_num) {
+ break
+ }
+ useTime = t
+ sleep(delta_time)
+ }
+ assertTrue(useTime <= OpTimeout)
+ }
+
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerStd} PROPERTIES ("tokenizer" = "standard", "token_filter" = "lowercase");"""
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerKw} PROPERTIES ("tokenizer" = "keyword", "token_filter" = "lowercase");"""
+ sleep(10000)
+
+ def tableName = "test_multi_analyzer_show_tbl"
+ sql "DROP TABLE IF EXISTS ${tableName}"
+ sql """CREATE TABLE ${tableName} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+
+ // TC-SHOW-001: SHOW CREATE TABLE should contain all index definitions
+ def showCreate = sql """ SHOW CREATE TABLE ${tableName} """
+ def createStmt = showCreate.collect { it[1].toString() }.join("\n")
+ assertTrue(createStmt.contains("idx_std"))
+ assertTrue(createStmt.contains("idx_kw"))
+ assertTrue(createStmt.toLowerCase().contains(analyzerStd.toLowerCase()))
+ assertTrue(createStmt.toLowerCase().contains(analyzerKw.toLowerCase()))
+
+ // Verify SHOW CREATE TABLE output can be used to recreate table
+ def recreateTable = "test_multi_analyzer_show_recreate"
+ sql "DROP TABLE IF EXISTS ${recreateTable}"
+ def modifiedStmt = createStmt.replace(tableName, recreateTable)
+ sql "${modifiedStmt}"
+ def verifyCreate = sql """ SHOW CREATE TABLE ${recreateTable} """
+ def verifyStmt = verifyCreate.collect { it[1].toString().toLowerCase() }.join("\n")
+ assertTrue(verifyStmt.contains(analyzerStd.toLowerCase()))
+ assertTrue(verifyStmt.contains(analyzerKw.toLowerCase()))
+
+ // TC-SHOW-002: SHOW INDEX FROM should show all indexes
+ def showIndex = sql """ SHOW INDEX FROM ${tableName} """
+ def indexCount = showIndex.size()
+ assertTrue(indexCount >= 2, "Should have at least 2 indexes")
+ def indexNames = showIndex.collect { it[2].toString() }
+ assertTrue(indexNames.contains("idx_std"))
+ assertTrue(indexNames.contains("idx_kw"))
+
+ // TC-SHOW-003: SHOW BUILD INDEX after build
+ if (!isCloudMode()) {
+ def buildTable = "test_multi_analyzer_show_build"
+ sql "DROP TABLE IF EXISTS ${buildTable}"
+ sql """CREATE TABLE ${buildTable} (id INT, name STRING) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """INSERT INTO ${buildTable} VALUES (1, 'alice'), (2, 'bob');"""
+ sql """ ALTER TABLE ${buildTable} ADD INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"); """
+ wait_for_latest_op_on_table_finish(buildTable, timeout)
+ sql """ BUILD INDEX idx_std ON ${buildTable}; """
+ wait_for_build_index_on_partition_finish(buildTable, timeout)
+ def showBuild = sql """ SHOW BUILD INDEX WHERE TableName = "${buildTable}" """
+ assertTrue(showBuild.size() > 0, "Should have build index records")
+ sql "DROP TABLE IF EXISTS ${buildTable}"
+ }
+
+ // TC-SHOW-004: SHOW ALTER TABLE COLUMN after add index
+ def alterTable = "test_multi_analyzer_show_alter"
+ sql "DROP TABLE IF EXISTS ${alterTable}"
+ sql """CREATE TABLE ${alterTable} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """ ALTER TABLE ${alterTable} ADD INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}"); """
+ wait_for_latest_op_on_table_finish(alterTable, timeout)
+ def showAlter = sql """ SHOW ALTER TABLE COLUMN WHERE TableName = "${alterTable}" ORDER BY CreateTime DESC LIMIT 1; """
+ assertTrue(showAlter.size() > 0, "Should have alter table records")
+ assertTrue(showAlter[0].toString().contains("FINISHED"), "Alter should be finished")
+
+ // Cleanup
+ sql "DROP TABLE IF EXISTS ${tableName}"
+ sql "DROP TABLE IF EXISTS ${recreateTable}"
+ sql "DROP TABLE IF EXISTS ${alterTable}"
+}
diff --git a/regression-test/suites/inverted_index_p0/test_multi_analyzer_table_types.groovy b/regression-test/suites/inverted_index_p0/test_multi_analyzer_table_types.groovy
new file mode 100644
index 00000000000..2d918b3466a
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_multi_analyzer_table_types.groovy
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_multi_analyzer_table_types", "p0") {
+ // Test multi-analyzer indexes on different table types: DUP, MOW, MOR, AGG
+
+ def analyzerStd = "multi_tbl_type_std_analyzer"
+ def analyzerKw = "multi_tbl_type_kw_analyzer"
+
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerStd} PROPERTIES ("tokenizer" = "standard", "token_filter" = "lowercase");"""
+ sql """CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${analyzerKw} PROPERTIES ("tokenizer" = "keyword", "token_filter" = "lowercase");"""
+ sleep(10000)
+
+ // TC-TBL-001: DUPLICATE KEY table
+ def tblDup = "test_multi_analyzer_dup"
+ sql "DROP TABLE IF EXISTS ${tblDup}"
+ sql """CREATE TABLE ${tblDup} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1");"""
+ sql """INSERT INTO ${tblDup} VALUES (1, 'alice'), (2, 'bob'), (3, 'alice bob');"""
+ sql "sync"
+ qt_dup_std """SELECT id FROM ${tblDup} WHERE name MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_dup_kw """SELECT id FROM ${tblDup} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+
+ // TC-TBL-002: UNIQUE KEY (MOW) table
+ def tblMow = "test_multi_analyzer_mow"
+ sql "DROP TABLE IF EXISTS ${tblMow}"
+ sql """CREATE TABLE ${tblMow} (id INT, name STRING, INDEX idx_std (name) USING INVERTED PROPERTIES("analyzer"="${analyzerStd}"), INDEX idx_kw (name) USING INVERTED PROPERTIES("analyzer"="${analyzerKw}")) UNIQUE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_allocation" = "tag.location.default: 1", "enable_unique_key_merge_on_write" = "true");"""
+ sql """INSERT INTO ${tblMow} VALUES (1, 'alice'), (2, 'bob'), (3, 'alice bob');"""
+ sql "sync"
+ qt_mow_std """SELECT id FROM ${tblMow} WHERE name MATCH 'alice' USING ANALYZER ${analyzerStd} ORDER BY id"""
+ qt_mow_kw """SELECT id FROM ${tblMow} WHERE name MATCH 'alice' USING ANALYZER ${analyzerKw} ORDER BY id"""
+}
diff --git a/regression-test/suites/inverted_index_p0/test_multi_tokenize_index_not_built.groovy b/regression-test/suites/inverted_index_p0/test_multi_tokenize_index_not_built.groovy
new file mode 100644
index 00000000000..66bceecf804
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_multi_tokenize_index_not_built.groovy
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_multi_tokenize_index_not_built", "p0") {
+ // Test case for https://github.com/apache/doris/issues/XXXXX
+ // When a column has multiple tokenize indexes and one of them is not built,
+ // queries using that analyzer should NOT fallback to other indexes.
+ // Instead, they should use non-index matching to return correct results.
+
+ def tableName = "test_multi_tokenize_index_not_built"
+
+ sql "DROP TABLE IF EXISTS ${tableName}"
+
+ // Create table with one inverted index using standard analyzer (tokenizes by whitespace)
+ sql """
+ CREATE TABLE ${tableName} (
+ id INT,
+ title VARCHAR(512) NOT NULL,
+ INDEX idx_title_standard (title) USING INVERTED PROPERTIES("built_in_analyzer" = "standard", "support_phrase" = "true")
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+
+ // Insert test data
+ // Difference: standard analyzer tokenizes "hello world" -> ["hello", "world"]
+ // keyword analyzer keeps "hello world" as single token
+ sql """
+ INSERT INTO ${tableName} VALUES
+ (1, 'hello world'),
+ (2, 'hello'),
+ (3, 'world'),
+ (4, 'hello world test'),
+ (5, 'foo bar')
+ """
+
+ sql "sync"
+
+ // Query using standard analyzer - should use idx_title_standard
+ // Standard tokenizes text, so 'hello' matches rows containing 'hello' token
+ // Should match rows 1, 2, 4
+ qt_standard_before """
+ SELECT id FROM ${tableName}
+ WHERE title MATCH 'hello' USING ANALYZER standard
+ ORDER BY id
+ """
+
+ // Add new index with keyword analyzer (no tokenization) but DO NOT build it
+ sql """
+ ALTER TABLE ${tableName} ADD INDEX idx_title_keyword (title)
+ USING INVERTED PROPERTIES("built_in_analyzer" = "none")
+ """
+
+ // Wait for schema change to complete
+ def waitSchemaChange = { tbl ->
+ int maxRetry = 60
+ for (int i = 0; i < maxRetry; i++) {
+ def result = sql "SHOW ALTER TABLE COLUMN WHERE TableName='${tbl}' ORDER BY CreateTime DESC LIMIT 1"
+ if (result.size() > 0 && result[0][9] == "FINISHED") {
+ return true
+ }
+ sleep(1000)
+ }
+ return false
+ }
+ assertTrue(waitSchemaChange(tableName), "Schema change did not finish in time")
+
+ // Query using keyword/none analyzer - idx_title_keyword is NOT built
+ // This should NOT use idx_title_standard (which would give wrong results)
+ // Instead, it should use non-index matching path (downgrade to slow path)
+ // With none analyzer: 'hello' should only match exact 'hello' (row 2)
+ // If it wrongly uses standard index, it would match rows 1, 2, 4
+ qt_keyword_not_built """
+ SELECT id FROM ${tableName}
+ WHERE title MATCH 'hello' USING ANALYZER none
+ ORDER BY id
+ """
+
+ // Query using standard analyzer - should still work with idx_title_standard
+ qt_standard_after_add """
+ SELECT id FROM ${tableName}
+ WHERE title MATCH 'hello' USING ANALYZER standard
+ ORDER BY id
+ """
+
+ // Now build the keyword index
+ // Cloud mode: BUILD INDEX ON table (builds all indexes)
+ // Non-cloud mode: BUILD INDEX idx_name ON table (builds specific index)
+ if (isCloudMode()) {
+ sql "BUILD INDEX ON ${tableName}"
+ } else {
+ sql "BUILD INDEX idx_title_keyword ON ${tableName}"
+ }
+
+ // Wait for build index to complete
+ def waitBuildIndex = { tbl ->
+ int maxRetry = 60
+ for (int i = 0; i < maxRetry; i++) {
+ def result = sql "SHOW BUILD INDEX WHERE TableName='${tbl}' ORDER BY CreateTime DESC LIMIT 1"
+ if (result.size() > 0 && result[0][7] == "FINISHED") {
+ return true
+ }
+ sleep(1000)
+ }
+ return false
+ }
+ assertTrue(waitBuildIndex(tableName), "Build index did not finish in time")
+
+ // Query using keyword analyzer after build - should use idx_title_keyword
+ // 'hello' exact match only matches row 2
+ qt_keyword_after_build """
+ SELECT id FROM ${tableName}
+ WHERE title MATCH 'hello' USING ANALYZER none
+ ORDER BY id
+ """
+
+ // Additional test: search for full string with keyword analyzer
+ qt_full_string_keyword """
+ SELECT id FROM ${tableName}
+ WHERE title MATCH 'hello world' USING ANALYZER none
+ ORDER BY id
+ """
+
+ // Same search with standard analyzer (tokenizes, so matches 'hello' AND 'world')
+ qt_full_string_standard """
+ SELECT id FROM ${tableName}
+ WHERE title MATCH 'hello world' USING ANALYZER standard
+ ORDER BY id
+ """
+
+ // Clean up
+ sql "DROP TABLE IF EXISTS ${tableName}"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org