You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2026/02/10 05:00:08 UTC
[PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
airborne12 opened a new pull request, #60635:
URL: https://github.com/apache/doris/pull/60635
## Summary
Refactor SearchDslParser from two-phase parsing to single-phase ANTLR parsing:
- **Eliminate hand-written preprocessing**: The previous two-phase approach used string manipulation to preprocess DSL before ANTLR parsing, which was mangling regex patterns like `/(\\d{1,2}:\\d{2} [AP]M)/`
- **Native bareQuery support**: Added `bareQuery` rule to ANTLR grammar to handle terms without field prefix directly
- **Implicit conjunction**: Made `AND` optional in grammar to support space-separated terms using `default_operator` (AND/OR)
- **Lucene mode semantics**: Added `luceneMode` parameter to multi-field expansion for proper `OCCUR_BOOLEAN` handling
- **Preserve explicit fields**: Multi-field expansion now correctly preserves fields not in the expansion list
## Test Plan
- [x] All 106 unit tests pass
- [x] All 18 search regression tests pass
- [x] Manual verification of regex pattern handling
🤖 Generated with [Claude Code](https://claude.ai/code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #60635:
URL: https://github.com/apache/doris/pull/60635#issuecomment-3875356704
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60635:
URL: https://github.com/apache/doris/pull/60635#issuecomment-3877247310
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60635:
URL: https://github.com/apache/doris/pull/60635#issuecomment-3875548875
<details>
<summary>TPC-H: <b>Total hot run time: 31163 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39b806cc71ed6a01299726b83286a6dcd28f74e5, data reload: false
------ Round 1 ----------------------------------
q1 17662 4432 4269 4269
q2 2011 393 257 257
q3 10109 1277 734 734
q4 10199 840 315 315
q5 7468 2118 2064 2064
q6 193 175 144 144
q7 882 725 614 614
q8 9269 1380 1200 1200
q9 4772 4666 4687 4666
q10 6789 1926 1545 1545
q11 551 301 286 286
q12 344 375 222 222
q13 17797 4132 3287 3287
q14 237 238 222 222
q15 874 805 819 805
q16 686 699 621 621
q17 697 791 554 554
q18 6622 6008 6337 6008
q19 1417 1044 658 658
q20 556 552 410 410
q21 2769 2068 1984 1984
q22 392 322 298 298
Total cold run time: 102296 ms
Total hot run time: 31163 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4572 4521 4660 4521
q2 276 352 250 250
q3 2332 2832 2348 2348
q4 1437 1873 1425 1425
q5 4797 4447 4795 4447
q6 231 188 144 144
q7 2052 1870 1690 1690
q8 2595 2528 2436 2436
q9 7630 7437 7425 7425
q10 2889 3098 2595 2595
q11 587 460 453 453
q12 730 801 633 633
q13 3986 4372 3579 3579
q14 282 311 285 285
q15 833 781 775 775
q16 639 686 647 647
q17 1090 1283 1293 1283
q18 7386 7473 7165 7165
q19 808 811 809 809
q20 1985 2054 1915 1915
q21 4466 4294 4119 4119
q22 558 560 503 503
Total cold run time: 52161 ms
Total hot run time: 49447 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60635:
URL: https://github.com/apache/doris/pull/60635#issuecomment-3877246984
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 merged PR #60635:
URL: https://github.com/apache/doris/pull/60635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60635:
URL: https://github.com/apache/doris/pull/60635#issuecomment-3875718808
<details>
<summary>ClickBench: <b>Total hot run time: 28.1 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c7d990cceec11eff1987d50668940410388bb4b0, data reload: false
query1 0.05 0.04 0.04
query2 0.10 0.05 0.04
query3 0.25 0.08 0.09
query4 1.61 0.12 0.11
query5 0.26 0.27 0.25
query6 1.17 0.66 0.67
query7 0.03 0.03 0.02
query8 0.06 0.04 0.04
query9 0.56 0.50 0.49
query10 0.56 0.55 0.55
query11 0.14 0.10 0.10
query12 0.14 0.10 0.10
query13 0.62 0.60 0.61
query14 1.05 1.05 1.06
query15 0.87 0.86 0.87
query16 0.42 0.42 0.39
query17 1.15 1.13 1.10
query18 0.22 0.20 0.20
query19 2.03 1.95 2.00
query20 0.01 0.01 0.02
query21 15.39 0.26 0.14
query22 5.18 0.06 0.05
query23 15.94 0.28 0.10
query24 2.43 0.27 0.18
query25 0.09 0.06 0.11
query26 0.15 0.13 0.14
query27 0.10 0.05 0.05
query28 4.07 1.17 0.96
query29 12.57 3.96 3.17
query30 0.28 0.14 0.12
query31 2.82 0.63 0.40
query32 3.23 0.59 0.49
query33 3.26 3.22 3.32
query34 16.08 5.45 4.77
query35 4.83 4.77 4.74
query36 0.64 0.49 0.49
query37 0.11 0.07 0.07
query38 0.07 0.04 0.04
query39 0.05 0.03 0.03
query40 0.20 0.16 0.15
query41 0.09 0.04 0.03
query42 0.04 0.03 0.04
query43 0.04 0.04 0.04
Total cold run time: 98.96 s
Total hot run time: 28.1 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #60635:
URL: https://github.com/apache/doris/pull/60635#issuecomment-3875349411
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60635:
URL: https://github.com/apache/doris/pull/60635#issuecomment-3876393945
# FE Regression Coverage Report
Increment line coverage ` 59.87% (185/309)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60635_c7d990cceec11eff1987d50668940410388bb4b0_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60635_c7d990cceec11eff1987d50668940410388bb4b0_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #60635:
URL: https://github.com/apache/doris/pull/60635#issuecomment-3875557856
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](search) Refactor SearchDslParser to single-phase ANTLR parsing [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60635:
URL: https://github.com/apache/doris/pull/60635#issuecomment-3875653849
<details>
<summary>TPC-H: <b>Total hot run time: 30771 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7d990cceec11eff1987d50668940410388bb4b0, data reload: false
------ Round 1 ----------------------------------
q1 17589 4361 4281 4281
q2 2026 354 236 236
q3 6984 1349 777 777
q4 10233 820 318 318
q5 8825 2198 2115 2115
q6 259 177 147 147
q7 901 743 616 616
q8 9276 1449 1219 1219
q9 4824 4631 4607 4607
q10 6865 1949 1570 1570
q11 491 310 295 295
q12 406 385 223 223
q13 14548 4080 3249 3249
q14 230 237 215 215
q15 905 831 811 811
q16 688 689 624 624
q17 694 814 566 566
q18 6577 5890 5809 5809
q19 1112 1000 609 609
q20 497 501 398 398
q21 2572 1806 1844 1806
q22 368 321 280 280
Total cold run time: 96870 ms
Total hot run time: 30771 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4387 4365 4381 4365
q2 254 343 261 261
q3 2104 2712 2202 2202
q4 1364 1796 1280 1280
q5 4331 4215 4278 4215
q6 214 175 135 135
q7 1851 1799 1676 1676
q8 2505 2348 2370 2348
q9 6982 7017 6997 6997
q10 2616 2815 2438 2438
q11 575 468 438 438
q12 636 709 559 559
q13 3627 4016 3246 3246
q14 282 302 267 267
q15 835 789 811 789
q16 641 703 654 654
q17 1092 1275 1275 1275
q18 7388 7231 7297 7231
q19 853 840 819 819
q20 1949 2119 1886 1886
q21 4662 4286 4094 4094
q22 581 556 499 499
Total cold run time: 49729 ms
Total hot run time: 47674 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org