You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "LiBinfeng-01 (via GitHub)" <gi...@apache.org> on 2025/04/11 07:20:30 UTC
[PR] [fix](Nereids) fix hint existing using problems [doris]
LiBinfeng-01 opened a new pull request, #49987:
URL: https://github.com/apache/doris/pull/49987
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
- add BRACKET to distribute hint inside leading hint
- distinguish different distribute hints in leading hint, showing in hint log
- when we have some syntax error in hint, we should not throw an exception. put it in hint log instead
- change BRACE in leading hint to paren
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2833787901
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2796133742
<details>
<summary>TPC-DS: <b>Total hot run time: 186245 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae8c7515292bb6c14bbbb45e6739659e31a14079, data reload: false
query1 1033 503 469 469
query2 6570 1910 1968 1910
query3 6751 222 216 216
query4 26132 23640 23542 23542
query5 4771 619 473 473
query6 316 217 198 198
query7 4639 490 274 274
query8 289 238 229 229
query9 8649 2580 2582 2580
query10 503 332 282 282
query11 15352 14999 14889 14889
query12 156 111 106 106
query13 1645 496 399 399
query14 9329 6105 6109 6105
query15 216 186 174 174
query16 7273 662 468 468
query17 1124 707 559 559
query18 1974 388 307 307
query19 185 186 156 156
query20 124 121 119 119
query21 218 119 100 100
query22 3957 4093 3997 3997
query23 33832 32989 32977 32977
query24 8421 2375 2384 2375
query25 529 450 383 383
query26 1230 265 147 147
query27 2759 503 326 326
query28 4336 2396 2398 2396
query29 761 538 423 423
query30 278 215 190 190
query31 942 846 762 762
query32 72 65 66 65
query33 564 372 311 311
query34 786 871 505 505
query35 778 822 726 726
query36 943 987 910 910
query37 122 101 80 80
query38 4200 4192 4080 4080
query39 1473 1395 1381 1381
query40 218 120 107 107
query41 57 54 51 51
query42 120 101 107 101
query43 504 516 487 487
query44 1310 811 808 808
query45 221 177 162 162
query46 829 1019 627 627
query47 1731 1784 1718 1718
query48 375 409 294 294
query49 775 482 415 415
query50 621 671 406 406
query51 4138 4084 4161 4084
query52 110 107 100 100
query53 221 263 182 182
query54 578 566 515 515
query55 86 86 88 86
query56 299 284 286 284
query57 1105 1125 1067 1067
query58 275 264 259 259
query59 2666 2669 2561 2561
query60 327 312 310 310
query61 129 130 128 128
query62 784 718 657 657
query63 219 184 186 184
query64 4373 1063 673 673
query65 4367 4207 4253 4207
query66 1137 417 306 306
query67 15746 15368 15219 15219
query68 4313 897 527 527
query69 465 304 263 263
query70 1169 1110 1064 1064
query71 406 316 306 306
query72 5739 4712 4679 4679
query73 616 585 342 342
query74 8952 9133 8765 8765
query75 3193 3228 2687 2687
query76 3220 1193 772 772
query77 502 382 299 299
query78 9916 10063 9264 9264
query79 1826 816 582 582
query80 673 555 453 453
query81 499 266 232 232
query82 196 129 99 99
query83 266 260 239 239
query84 265 109 90 90
query85 827 414 483 414
query86 377 311 310 310
query87 4395 4498 4369 4369
query88 2824 2263 2259 2259
query89 384 310 285 285
query90 1812 212 214 212
query91 144 141 115 115
query92 66 63 56 56
query93 1239 936 580 580
query94 633 424 279 279
query95 371 295 283 283
query96 482 579 272 272
query97 3131 3208 3106 3106
query98 228 218 204 204
query99 1325 1432 1251 1251
Total cold run time: 267007 ms
Total hot run time: 186245 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2796071537
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on code in PR #49987:
URL: https://github.com/apache/doris/pull/49987#discussion_r2041759430
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -3729,10 +3742,17 @@ private LogicalPlan withSelectHint(LogicalPlan logicalPlan, List<ParserRuleConte
break;
case "leading":
List<String> leadingParameters = new ArrayList<>();
+ int distributeIndex = 1;
for (HintAssignmentContext kv : hintStatement.parameters) {
if (kv.key != null) {
- String parameterName = visitIdentifierOrText(kv.key);
+ String parameterName = visitIdentifierOrTextOrParen(kv.key);
leadingParameters.add(parameterName);
+ } else if (kv.distributeHintType() != null) {
+ if (kv.distributeHintType().BROADCAST() != null) {
+ leadingParameters.add("[broadcast]_" + distributeIndex++);
Review Comment:
use LeadingHint.broadcastString
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2796145411
<details>
<summary>ClickBench: <b>Total hot run time: 30.83 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae8c7515292bb6c14bbbb45e6739659e31a14079, data reload: false
query1 0.03 0.04 0.03
query2 0.12 0.10 0.11
query3 0.25 0.19 0.19
query4 1.60 0.18 0.19
query5 0.60 0.58 0.58
query6 1.18 0.72 0.73
query7 0.02 0.02 0.02
query8 0.04 0.03 0.03
query9 0.58 0.53 0.53
query10 0.57 0.61 0.57
query11 0.16 0.11 0.11
query12 0.14 0.11 0.11
query13 0.61 0.60 0.59
query14 2.80 2.84 2.69
query15 0.92 0.85 0.85
query16 0.37 0.38 0.40
query17 1.00 1.01 1.00
query18 0.20 0.19 0.20
query19 1.95 1.97 1.83
query20 0.02 0.01 0.01
query21 15.35 0.94 0.55
query22 0.75 1.19 0.63
query23 15.05 1.36 0.64
query24 6.97 1.64 0.60
query25 0.50 0.15 0.08
query26 0.66 0.16 0.14
query27 0.05 0.05 0.05
query28 8.98 0.88 0.44
query29 12.57 3.91 3.27
query30 0.24 0.08 0.07
query31 2.82 0.58 0.38
query32 3.22 0.54 0.47
query33 3.08 3.02 3.08
query34 15.58 5.02 4.50
query35 4.52 4.52 4.48
query36 0.67 0.49 0.48
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.14 0.13
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 104.65 s
Total hot run time: 30.83 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow closed pull request #49987: [fix](Nereids) fix hint existing using problems
URL: https://github.com/apache/doris/pull/49987
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2796065593
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2796109825
<details>
<summary>TPC-H: <b>Total hot run time: 34963 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae8c7515292bb6c14bbbb45e6739659e31a14079, data reload: false
------ Round 1 ----------------------------------
q1 26033 5047 4988 4988
q2 2060 293 186 186
q3 10374 1237 689 689
q4 10237 1037 545 545
q5 7528 2331 2375 2331
q6 189 167 135 135
q7 942 749 618 618
q8 9331 1309 1170 1170
q9 6841 5111 5085 5085
q10 6809 2323 1886 1886
q11 481 281 280 280
q12 345 353 221 221
q13 17752 3679 3134 3134
q14 246 233 204 204
q15 536 490 487 487
q16 637 610 589 589
q17 580 858 375 375
q18 7573 7249 7036 7036
q19 1216 949 584 584
q20 344 325 234 234
q21 4407 3390 3219 3219
q22 1048 990 967 967
Total cold run time: 115509 ms
Total hot run time: 34963 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5169 5040 5026 5026
q2 238 332 248 248
q3 2104 2630 2307 2307
q4 1395 1818 1512 1512
q5 4542 4411 4345 4345
q6 212 166 124 124
q7 1944 1908 1761 1761
q8 2590 2630 2545 2545
q9 7111 7246 7044 7044
q10 3006 3192 2720 2720
q11 567 497 485 485
q12 657 772 617 617
q13 3551 3951 3253 3253
q14 273 300 263 263
q15 508 479 479 479
q16 647 696 666 666
q17 1119 1535 1348 1348
q18 7665 7489 7593 7489
q19 800 789 948 789
q20 1960 1931 1795 1795
q21 5124 4706 4720 4706
q22 1035 1020 988 988
Total cold run time: 52217 ms
Total hot run time: 50510 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2833861322
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2800020322
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2800704233
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2832927146
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #49987:
URL: https://github.com/apache/doris/pull/49987#issuecomment-2800375119
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on code in PR #49987:
URL: https://github.com/apache/doris/pull/49987#discussion_r2041760815
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -3729,10 +3742,17 @@ private LogicalPlan withSelectHint(LogicalPlan logicalPlan, List<ParserRuleConte
break;
case "leading":
List<String> leadingParameters = new ArrayList<>();
+ int distributeIndex = 1;
for (HintAssignmentContext kv : hintStatement.parameters) {
if (kv.key != null) {
- String parameterName = visitIdentifierOrText(kv.key);
+ String parameterName = visitIdentifierOrTextOrParen(kv.key);
leadingParameters.add(parameterName);
+ } else if (kv.distributeHintType() != null) {
+ if (kv.distributeHintType().BROADCAST() != null) {
+ leadingParameters.add("[broadcast]_" + distributeIndex++);
+ } else {
+ leadingParameters.add("[shuffle]_" + distributeIndex++);
Review Comment:
use LeadingHint.shuffleString
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org