You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2025/10/24 11:53:04 UTC
[PR] [feature](search) add another two params for search [doris]
airborne12 opened a new pull request, #57312:
URL: https://github.com/apache/doris/pull/57312
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](search) add another two params for search [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #57312:
URL: https://github.com/apache/doris/pull/57312#issuecomment-3443894502
# FE Regression Coverage Report
Increment line coverage ` 5.81% (9/155)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/57312_b1bb6da43bc3aebbb99ed2b2d8a54b4e517acfc9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/57312_b1bb6da43bc3aebbb99ed2b2d8a54b4e517acfc9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](search) add another two params for search [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #57312:
URL: https://github.com/apache/doris/pull/57312#issuecomment-3442916468
<details>
<summary>TPC-DS: <b>Total hot run time: 189640 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1bb6da43bc3aebbb99ed2b2d8a54b4e517acfc9, data reload: false
query1 1060 438 420 420
query2 6577 1698 1677 1677
query3 6758 226 220 220
query4 26225 24034 23395 23395
query5 5781 639 471 471
query6 359 248 241 241
query7 4662 499 298 298
query8 324 286 280 280
query9 8736 2569 2603 2569
query10 550 362 297 297
query11 15235 14990 14948 14948
query12 185 125 124 124
query13 1759 590 457 457
query14 11906 9329 9290 9290
query15 254 186 182 182
query16 7826 668 488 488
query17 1598 760 628 628
query18 2041 502 355 355
query19 284 239 182 182
query20 151 133 130 130
query21 219 139 123 123
query22 4579 4555 4501 4501
query23 34904 33874 33823 33823
query24 8554 2507 2555 2507
query25 635 551 445 445
query26 1224 278 162 162
query27 3049 526 390 390
query28 4390 2199 2189 2189
query29 806 670 521 521
query30 349 240 207 207
query31 967 1036 760 760
query32 84 73 66 66
query33 588 389 337 337
query34 819 870 538 538
query35 813 929 797 797
query36 1007 1045 922 922
query37 130 105 86 86
query38 3543 3592 3519 3519
query39 1460 1402 1413 1402
query40 222 126 116 116
query41 59 56 57 56
query42 125 108 109 108
query43 492 510 453 453
query44 1206 730 729 729
query45 183 180 171 171
query46 872 985 643 643
query47 1786 1802 1732 1732
query48 399 427 316 316
query49 781 509 415 415
query50 648 693 415 415
query51 3853 3944 3861 3861
query52 109 101 97 97
query53 229 264 190 190
query54 609 592 521 521
query55 91 87 82 82
query56 338 314 332 314
query57 1164 1197 1112 1112
query58 286 291 274 274
query59 2536 2636 2505 2505
query60 350 351 334 334
query61 163 163 155 155
query62 781 741 667 667
query63 230 192 194 192
query64 4550 1171 861 861
query65 4049 3950 3949 3949
query66 1039 431 328 328
query67 15364 15260 14852 14852
query68 8309 924 590 590
query69 545 324 301 301
query70 1394 1262 1266 1262
query71 517 343 338 338
query72 5744 5015 4849 4849
query73 676 658 360 360
query74 8862 9081 8677 8677
query75 4153 3409 2894 2894
query76 3772 1153 736 736
query77 819 395 316 316
query78 9472 9572 8861 8861
query79 2015 837 591 591
query80 648 559 498 498
query81 491 268 227 227
query82 460 163 136 136
query83 272 281 244 244
query84 258 113 98 98
query85 876 476 429 429
query86 335 292 305 292
query87 3692 3723 3681 3681
query88 3657 2230 2220 2220
query89 395 327 290 290
query90 2046 215 218 215
query91 182 173 135 135
query92 88 73 65 65
query93 1480 984 641 641
query94 680 457 325 325
query95 399 325 317 317
query96 504 577 277 277
query97 2916 2995 2855 2855
query98 257 210 211 210
query99 1380 1418 1315 1315
Total cold run time: 280199 ms
Total hot run time: 189640 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](search) add another two params for search [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #57312:
URL: https://github.com/apache/doris/pull/57312#issuecomment-3442944021
<details>
<summary>ClickBench: <b>Total hot run time: 27.7 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b1bb6da43bc3aebbb99ed2b2d8a54b4e517acfc9, data reload: false
query1 0.05 0.06 0.05
query2 0.09 0.06 0.05
query3 0.26 0.09 0.09
query4 1.61 0.12 0.13
query5 0.28 0.26 0.25
query6 1.21 0.65 0.64
query7 0.03 0.03 0.03
query8 0.05 0.04 0.04
query9 0.64 0.52 0.52
query10 0.60 0.58 0.57
query11 0.16 0.11 0.11
query12 0.14 0.12 0.12
query13 0.63 0.61 0.60
query14 1.01 1.03 1.01
query15 0.85 0.83 0.87
query16 0.41 0.39 0.41
query17 1.03 1.02 1.02
query18 0.22 0.20 0.21
query19 1.90 1.83 1.79
query20 0.02 0.01 0.01
query21 15.50 0.20 0.13
query22 5.05 0.08 0.05
query23 15.66 0.27 0.10
query24 2.87 0.58 1.02
query25 0.08 0.06 0.06
query26 0.14 0.13 0.14
query27 0.06 0.04 0.05
query28 5.15 1.13 0.95
query29 12.63 3.94 3.27
query30 0.28 0.13 0.11
query31 2.82 0.59 0.37
query32 3.23 0.55 0.47
query33 3.01 3.09 3.01
query34 15.79 5.15 4.58
query35 4.58 4.58 4.55
query36 0.66 0.51 0.49
query37 0.11 0.07 0.06
query38 0.07 0.04 0.04
query39 0.04 0.03 0.03
query40 0.19 0.14 0.14
query41 0.10 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 99.29 s
Total hot run time: 27.7 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](search) add another two params for search [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #57312:
URL: https://github.com/apache/doris/pull/57312#issuecomment-3446446294
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](search) add another two params for search [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #57312:
URL: https://github.com/apache/doris/pull/57312#issuecomment-3446446317
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](search) add another two params for search [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #57312:
URL: https://github.com/apache/doris/pull/57312#discussion_r2460228062
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/SearchDslParser.java:
##########
@@ -107,11 +126,271 @@ public void syntaxError(org.antlr.v4.runtime.Recognizer<?, ?> recognizer,
return new QsPlan(root, bindings);
} catch (Exception e) {
- LOG.error("Failed to parse search DSL: '{}'", dsl, e);
+ LOG.error("Failed to parse search DSL: '{}' (expanded: '{}')", dsl, expandedDsl, e);
throw new RuntimeException("Invalid search DSL syntax: " + dsl + ". Error: " + e.getMessage(), e);
}
}
+ /**
+ * Normalize default operator to lowercase "and" or "or"
+ */
+ private static String normalizeDefaultOperator(String operator) {
+ if (operator == null || operator.trim().isEmpty()) {
+ return "or"; // Default to OR
+ }
+ String normalized = operator.trim().toLowerCase();
+ if ("and".equals(normalized) || "or".equals(normalized)) {
+ return normalized;
+ }
+ throw new IllegalArgumentException("Invalid default operator: " + operator
+ + ". Must be 'and' or 'or'");
+ }
+
+ /**
+ * Expand simplified DSL to full DSL format
+ * <p>
+ * Examples:
+ * - "foo bar" + field="tags" + operator="and" → "tags:ALL(foo bar)"
+ * - "foo* bar*" + field="tags" + operator="and" → "tags:foo* AND tags:bar*"
+ * - "foo OR bar" + field="tags" → "tags:foo OR tags:bar"
+ * - "EXACT(foo bar)" + field="tags" → "tags:EXACT(foo bar)"
+ *
+ * @param dsl Simple DSL string
+ * @param defaultField Default field name
+ * @param defaultOperator "and" or "or"
+ * @return Expanded full DSL
+ */
+ private static String expandSimplifiedDsl(String dsl, String defaultField, String defaultOperator) {
+ // 1. If DSL already contains field names (colon), return as-is
+ if (containsFieldReference(dsl)) {
+ return dsl;
+ }
+
+ // 2. Check if DSL starts with a function keyword (EXACT, ANY, ALL, IN)
+ if (startsWithFunction(dsl)) {
+ return defaultField + ":" + dsl;
+ }
+
+ // 3. Check for explicit boolean operators in DSL
+ if (containsExplicitOperators(dsl)) {
+ return addFieldPrefixToOperatorExpression(dsl, defaultField);
+ }
+
+ // 4. Tokenize and analyze terms
+ List<String> terms = tokenizeDsl(dsl);
+ if (terms.isEmpty()) {
+ return defaultField + ":" + dsl;
+ }
+
+ // 5. Single term - simple case
+ if (terms.size() == 1) {
+ return defaultField + ":" + terms.get(0);
+ }
+
+ // 6. Multiple terms - check for wildcards
+ boolean hasWildcard = terms.stream().anyMatch(SearchDslParser::containsWildcard);
+
+ if (hasWildcard) {
+ // Wildcards cannot be tokenized - must create separate field queries
+ String operator = "and".equals(defaultOperator) ? " AND " : " OR ";
+ return terms.stream()
+ .map(term -> defaultField + ":" + term)
+ .collect(java.util.stream.Collectors.joining(operator));
+ } else {
+ // Regular multi-term query - use ANY/ALL
+ String clauseType = "and".equals(defaultOperator) ? "ALL" : "ANY";
+ return defaultField + ":" + clauseType + "(" + dsl + ")";
+ }
+ }
+
+ /**
+ * Check if DSL contains field references (has colon not in quoted strings)
+ */
+ private static boolean containsFieldReference(String dsl) {
+ boolean inQuotes = false;
+ boolean inRegex = false;
+ for (int i = 0; i < dsl.length(); i++) {
+ char c = dsl.charAt(i);
+ if (c == '"' && (i == 0 || dsl.charAt(i - 1) != '\\')) {
+ inQuotes = !inQuotes;
+ } else if (c == '/' && !inQuotes) {
+ inRegex = !inRegex;
+ } else if (c == ':' && !inQuotes && !inRegex) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Check if DSL starts with function keywords
+ */
+ private static boolean startsWithFunction(String dsl) {
+ String upper = dsl.toUpperCase();
+ return upper.startsWith("EXACT(")
+ || upper.startsWith("ANY(")
+ || upper.startsWith("ALL(")
+ || upper.startsWith("IN(");
+ }
+
+ /**
+ * Check if DSL contains explicit boolean operators (AND/OR/NOT)
+ */
+ private static boolean containsExplicitOperators(String dsl) {
+ // Look for standalone AND/OR/NOT keywords (not part of field names)
+ String upper = dsl.toUpperCase();
+ return upper.matches(".*\\s+(AND|OR)\\s+.*")
+ || upper.matches("^NOT\\s+.*")
+ || upper.matches(".*\\s+NOT\\s+.*");
Review Comment:
Multiple regex matches on the same string can be inefficient. Consider using a single compiled Pattern or combining these patterns into one regex: `Pattern.matches(\"^NOT\\s+.*|.*\\s+(AND|OR|NOT)\\s+.*\", upper)` to improve performance for frequently called code.
##########
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/scalar/SearchDslParserTest.java:
##########
@@ -272,4 +272,309 @@ public void testQuotedFieldNames() {
Assertions.assertEquals(1, plan.fieldBindings.size());
Assertions.assertEquals("field name", plan.fieldBindings.get(0).fieldName);
}
+
+ // ============ Tests for Default Field and Operator Support ============
+
+ @Test
+ public void testDefaultFieldWithSimpleTerm() {
+ // Test: "foo" + field="tags" → "tags:foo"
+ String dsl = "foo";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", null);
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.TERM, plan.root.type);
+ Assertions.assertEquals("tags", plan.root.field);
+ Assertions.assertEquals("foo", plan.root.value);
+ Assertions.assertEquals(1, plan.fieldBindings.size());
+ Assertions.assertEquals("tags", plan.fieldBindings.get(0).fieldName);
+ }
+
+ @Test
+ public void testDefaultFieldWithMultiTermAnd() {
+ // Test: "foo bar" + field="tags" + operator="and" → "tags:ALL(foo bar)"
+ String dsl = "foo bar";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", "and");
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.ALL, plan.root.type);
+ Assertions.assertEquals("tags", plan.root.field);
+ Assertions.assertEquals("foo bar", plan.root.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithMultiTermOr() {
+ // Test: "foo bar" + field="tags" + operator="or" → "tags:ANY(foo bar)"
+ String dsl = "foo bar";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", "or");
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.ANY, plan.root.type);
+ Assertions.assertEquals("tags", plan.root.field);
+ Assertions.assertEquals("foo bar", plan.root.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithMultiTermDefaultOr() {
+ // Test: "foo bar" + field="tags" (no operator, defaults to OR) → "tags:ANY(foo bar)"
+ String dsl = "foo bar";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", null);
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.ANY, plan.root.type);
+ Assertions.assertEquals("tags", plan.root.field);
+ Assertions.assertEquals("foo bar", plan.root.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithWildcardSingleTerm() {
+ // Test: "foo*" + field="tags" → "tags:foo*"
+ String dsl = "foo*";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", null);
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.PREFIX, plan.root.type);
+ Assertions.assertEquals("tags", plan.root.field);
+ Assertions.assertEquals("foo*", plan.root.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithWildcardMultiTermAnd() {
+ // Test: "foo* bar*" + field="tags" + operator="and" → "tags:foo* AND tags:bar*"
+ String dsl = "foo* bar*";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", "and");
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.AND, plan.root.type);
+ Assertions.assertEquals(2, plan.root.children.size());
+
+ QsNode firstChild = plan.root.children.get(0);
+ Assertions.assertEquals(QsClauseType.PREFIX, firstChild.type);
+ Assertions.assertEquals("tags", firstChild.field);
+ Assertions.assertEquals("foo*", firstChild.value);
+
+ QsNode secondChild = plan.root.children.get(1);
+ Assertions.assertEquals(QsClauseType.PREFIX, secondChild.type);
+ Assertions.assertEquals("tags", secondChild.field);
+ Assertions.assertEquals("bar*", secondChild.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithWildcardMultiTermOr() {
+ // Test: "foo* bar*" + field="tags" + operator="or" → "tags:foo* OR tags:bar*"
+ String dsl = "foo* bar*";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", "or");
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.OR, plan.root.type);
+ Assertions.assertEquals(2, plan.root.children.size());
+ }
+
+ @Test
+ public void testDefaultFieldWithExplicitOperatorOverride() {
+ // Test: "foo OR bar" + field="tags" + operator="and" → explicit OR takes precedence
+ String dsl = "foo OR bar";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", "and");
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.OR, plan.root.type);
+ Assertions.assertEquals(2, plan.root.children.size());
+
+ QsNode firstChild = plan.root.children.get(0);
+ Assertions.assertEquals("tags", firstChild.field);
+ Assertions.assertEquals("foo", firstChild.value);
+
+ QsNode secondChild = plan.root.children.get(1);
+ Assertions.assertEquals("tags", secondChild.field);
+ Assertions.assertEquals("bar", secondChild.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithExplicitAndOperator() {
+ // Test: "foo AND bar" + field="tags" + operator="or" → explicit AND takes precedence
+ String dsl = "foo AND bar";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", "or");
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.AND, plan.root.type);
+ Assertions.assertEquals(2, plan.root.children.size());
+ }
+
+ @Test
+ public void testDefaultFieldWithExactFunction() {
+ // Test: "EXACT(foo bar)" + field="tags" → "tags:EXACT(foo bar)"
+ String dsl = "EXACT(foo bar)";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", null);
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.EXACT, plan.root.type);
+ Assertions.assertEquals("tags", plan.root.field);
+ Assertions.assertEquals("foo bar", plan.root.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithAnyFunction() {
+ // Test: "ANY(foo bar)" + field="tags" → "tags:ANY(foo bar)"
+ String dsl = "ANY(foo bar)";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", null);
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.ANY, plan.root.type);
+ Assertions.assertEquals("tags", plan.root.field);
+ Assertions.assertEquals("foo bar", plan.root.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithAllFunction() {
+ // Test: "ALL(foo bar)" + field="tags" → "tags:ALL(foo bar)"
+ String dsl = "ALL(foo bar)";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", null);
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.ALL, plan.root.type);
+ Assertions.assertEquals("tags", plan.root.field);
+ Assertions.assertEquals("foo bar", plan.root.value);
+ }
+
+ @Test
+ public void testDefaultFieldIgnoredWhenDslHasFieldReference() {
+ // Test: DSL with field reference should ignore default field
+ String dsl = "title:hello";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", "and");
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.TERM, plan.root.type);
+ Assertions.assertEquals("title", plan.root.field); // Should be "title", not "tags"
+ Assertions.assertEquals("hello", plan.root.value);
+ }
+
+ @Test
+ public void testInvalidDefaultOperator() {
+ // Test: invalid operator should throw exception
+ String dsl = "foo bar";
+
+ IllegalArgumentException exception = Assertions.assertThrows(IllegalArgumentException.class, () -> {
+ SearchDslParser.parseDsl(dsl, "tags", "invalid");
+ });
+
+ Assertions.assertTrue(exception.getMessage().contains("Invalid default operator"));
+ Assertions.assertTrue(exception.getMessage().contains("Must be 'and' or 'or'"));
+ }
+
+ @Test
+ public void testDefaultOperatorCaseInsensitive() {
+ // Test: operator should be case-insensitive
+ String dsl = "foo bar";
+
+ // Test "AND"
+ QsPlan plan1 = SearchDslParser.parseDsl(dsl, "tags", "AND");
+ Assertions.assertEquals(QsClauseType.ALL, plan1.root.type);
+
+ // Test "Or"
+ QsPlan plan2 = SearchDslParser.parseDsl(dsl, "tags", "Or");
+ Assertions.assertEquals(QsClauseType.ANY, plan2.root.type);
+
+ // Test "aNd"
+ QsPlan plan3 = SearchDslParser.parseDsl(dsl, "tags", "aNd");
+ Assertions.assertEquals(QsClauseType.ALL, plan3.root.type);
+ }
+
+ @Test
+ public void testDefaultFieldWithComplexWildcard() {
+ // Test: "*foo*" (middle wildcard) + field="tags"
+ String dsl = "*foo*";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", null);
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.WILDCARD, plan.root.type);
+ Assertions.assertEquals("tags", plan.root.field);
+ Assertions.assertEquals("*foo*", plan.root.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithMixedWildcards() {
+ // Test: "foo* *bar baz" (mixed wildcards and regular terms) + field="tags" + operator="and"
+ String dsl = "foo* bar baz";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", "and");
+
+ Assertions.assertNotNull(plan);
+ // Should create AND query because it contains wildcards
+ Assertions.assertEquals(QsClauseType.AND, plan.root.type);
+ Assertions.assertEquals(3, plan.root.children.size());
+ }
+
+ @Test
+ public void testDefaultFieldWithQuotedPhrase() {
+ // Test: quoted phrase should be treated as PHRASE
+ String dsl = "\"hello world\"";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", "and");
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.PHRASE, plan.root.type);
+ Assertions.assertEquals("tags", plan.root.field);
+ Assertions.assertEquals("hello world", plan.root.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithNotOperator() {
+ // Test: "NOT foo" + field="tags" → "NOT tags:foo"
+ String dsl = "NOT foo";
+ QsPlan plan = SearchDslParser.parseDsl(dsl, "tags", null);
+
+ Assertions.assertNotNull(plan);
+ Assertions.assertEquals(QsClauseType.NOT, plan.root.type);
+ Assertions.assertEquals(1, plan.root.children.size());
+
+ QsNode child = plan.root.children.get(0);
+ Assertions.assertEquals(QsClauseType.TERM, child.type);
+ Assertions.assertEquals("tags", child.field);
+ Assertions.assertEquals("foo", child.value);
+ }
+
+ @Test
+ public void testDefaultFieldWithEmptyString() {
+ // Test: empty default field should not expand DSL, causing parse error
+ // for incomplete DSL like "foo" (no field specified)
+ String dsl = "foo";
+
+ // This should throw an exception because "foo" alone is not valid DSL
Review Comment:
The comment is misleading. The test expects an exception not because `\"foo\"` is invalid DSL in general, but because with an empty default field, the expansion fails. Clarify that the exception occurs due to the empty default field preventing proper DSL expansion.
```suggestion
// This should throw an exception because with an empty default field, the parser cannot expand "foo" into a valid fielded query
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/SearchDslParser.java:
##########
@@ -107,11 +126,271 @@ public void syntaxError(org.antlr.v4.runtime.Recognizer<?, ?> recognizer,
return new QsPlan(root, bindings);
} catch (Exception e) {
- LOG.error("Failed to parse search DSL: '{}'", dsl, e);
+ LOG.error("Failed to parse search DSL: '{}' (expanded: '{}')", dsl, expandedDsl, e);
throw new RuntimeException("Invalid search DSL syntax: " + dsl + ". Error: " + e.getMessage(), e);
}
}
+ /**
+ * Normalize default operator to lowercase "and" or "or"
+ */
+ private static String normalizeDefaultOperator(String operator) {
+ if (operator == null || operator.trim().isEmpty()) {
+ return "or"; // Default to OR
+ }
+ String normalized = operator.trim().toLowerCase();
+ if ("and".equals(normalized) || "or".equals(normalized)) {
+ return normalized;
+ }
+ throw new IllegalArgumentException("Invalid default operator: " + operator
+ + ". Must be 'and' or 'or'");
+ }
Review Comment:
[nitpick] The null/empty check and trimming logic is duplicated. Consider extracting the trimming and null handling: `String trimmed = (operator == null) ? \"\" : operator.trim(); if (trimmed.isEmpty()) return \"or\";` This reduces redundancy and makes the intent clearer.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/SearchDslParser.java:
##########
@@ -107,11 +126,271 @@ public void syntaxError(org.antlr.v4.runtime.Recognizer<?, ?> recognizer,
return new QsPlan(root, bindings);
} catch (Exception e) {
- LOG.error("Failed to parse search DSL: '{}'", dsl, e);
+ LOG.error("Failed to parse search DSL: '{}' (expanded: '{}')", dsl, expandedDsl, e);
throw new RuntimeException("Invalid search DSL syntax: " + dsl + ". Error: " + e.getMessage(), e);
}
}
+ /**
+ * Normalize default operator to lowercase "and" or "or"
+ */
+ private static String normalizeDefaultOperator(String operator) {
+ if (operator == null || operator.trim().isEmpty()) {
+ return "or"; // Default to OR
+ }
+ String normalized = operator.trim().toLowerCase();
+ if ("and".equals(normalized) || "or".equals(normalized)) {
+ return normalized;
+ }
+ throw new IllegalArgumentException("Invalid default operator: " + operator
+ + ". Must be 'and' or 'or'");
+ }
+
+ /**
+ * Expand simplified DSL to full DSL format
+ * <p>
+ * Examples:
+ * - "foo bar" + field="tags" + operator="and" → "tags:ALL(foo bar)"
+ * - "foo* bar*" + field="tags" + operator="and" → "tags:foo* AND tags:bar*"
+ * - "foo OR bar" + field="tags" → "tags:foo OR tags:bar"
+ * - "EXACT(foo bar)" + field="tags" → "tags:EXACT(foo bar)"
+ *
+ * @param dsl Simple DSL string
+ * @param defaultField Default field name
+ * @param defaultOperator "and" or "or"
+ * @return Expanded full DSL
+ */
+ private static String expandSimplifiedDsl(String dsl, String defaultField, String defaultOperator) {
+ // 1. If DSL already contains field names (colon), return as-is
+ if (containsFieldReference(dsl)) {
+ return dsl;
+ }
+
+ // 2. Check if DSL starts with a function keyword (EXACT, ANY, ALL, IN)
+ if (startsWithFunction(dsl)) {
+ return defaultField + ":" + dsl;
+ }
+
+ // 3. Check for explicit boolean operators in DSL
+ if (containsExplicitOperators(dsl)) {
+ return addFieldPrefixToOperatorExpression(dsl, defaultField);
+ }
+
+ // 4. Tokenize and analyze terms
+ List<String> terms = tokenizeDsl(dsl);
+ if (terms.isEmpty()) {
+ return defaultField + ":" + dsl;
+ }
+
+ // 5. Single term - simple case
+ if (terms.size() == 1) {
+ return defaultField + ":" + terms.get(0);
+ }
+
+ // 6. Multiple terms - check for wildcards
+ boolean hasWildcard = terms.stream().anyMatch(SearchDslParser::containsWildcard);
+
+ if (hasWildcard) {
+ // Wildcards cannot be tokenized - must create separate field queries
+ String operator = "and".equals(defaultOperator) ? " AND " : " OR ";
+ return terms.stream()
+ .map(term -> defaultField + ":" + term)
+ .collect(java.util.stream.Collectors.joining(operator));
+ } else {
+ // Regular multi-term query - use ANY/ALL
+ String clauseType = "and".equals(defaultOperator) ? "ALL" : "ANY";
+ return defaultField + ":" + clauseType + "(" + dsl + ")";
+ }
+ }
+
+ /**
+ * Check if DSL contains field references (has colon not in quoted strings)
+ */
+ private static boolean containsFieldReference(String dsl) {
+ boolean inQuotes = false;
+ boolean inRegex = false;
+ for (int i = 0; i < dsl.length(); i++) {
+ char c = dsl.charAt(i);
+ if (c == '"' && (i == 0 || dsl.charAt(i - 1) != '\\')) {
+ inQuotes = !inQuotes;
+ } else if (c == '/' && !inQuotes) {
+ inRegex = !inRegex;
+ } else if (c == ':' && !inQuotes && !inRegex) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Check if DSL starts with function keywords
+ */
+ private static boolean startsWithFunction(String dsl) {
+ String upper = dsl.toUpperCase();
+ return upper.startsWith("EXACT(")
+ || upper.startsWith("ANY(")
+ || upper.startsWith("ALL(")
+ || upper.startsWith("IN(");
+ }
+
+ /**
+ * Check if DSL contains explicit boolean operators (AND/OR/NOT)
+ */
+ private static boolean containsExplicitOperators(String dsl) {
+ // Look for standalone AND/OR/NOT keywords (not part of field names)
+ String upper = dsl.toUpperCase();
+ return upper.matches(".*\\s+(AND|OR)\\s+.*")
+ || upper.matches("^NOT\\s+.*")
+ || upper.matches(".*\\s+NOT\\s+.*");
+ }
+
+ /**
+ * Add field prefix to expressions with explicit operators
+ * Example: "foo AND bar" → "field:foo AND field:bar"
+ */
+ private static String addFieldPrefixToOperatorExpression(String dsl, String defaultField) {
+ StringBuilder result = new StringBuilder();
+ StringBuilder currentTerm = new StringBuilder();
+ int i = 0;
+
+ while (i < dsl.length()) {
+ // Skip whitespace
+ while (i < dsl.length() && Character.isWhitespace(dsl.charAt(i))) {
+ i++;
+ }
+ if (i >= dsl.length()) {
+ break;
+ }
+
+ // Try to match operators
+ String remaining = dsl.substring(i);
+ String upperRemaining = remaining.toUpperCase();
+
+ if (upperRemaining.startsWith("AND ") || upperRemaining.startsWith("AND\t")
+ || (upperRemaining.equals("AND") && i + 3 >= dsl.length())) {
Review Comment:
[nitpick] The operator matching logic for AND, OR, and NOT (lines 269-317) contains duplicated patterns. Consider extracting a helper method `matchOperator(String remaining, String operator, int length)` to reduce code duplication and improve maintainability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](search) add another two params for search [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #57312:
URL: https://github.com/apache/doris/pull/57312#issuecomment-3442745785
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](search) add another two params for search [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 merged PR #57312:
URL: https://github.com/apache/doris/pull/57312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](search) add another two params for search [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #57312:
URL: https://github.com/apache/doris/pull/57312#issuecomment-3442745520
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org