You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "BiteTheDDDDt (via GitHub)" <gi...@apache.org> on 2026/05/21 09:35:55 UTC
[PR] [fix](be) Reject right anti hash mark join plans [doris]
BiteTheDDDDt opened a new pull request, #63482:
URL: https://github.com/apache/doris/pull/63482
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Hash mark join currently supports left semi/anti, null-aware left semi/anti, and right semi plans. Current FE rules do not generate right anti mark hash join from SQL, but a malformed or future plan could still reach BE hash join operators. The right anti mark path is not a supported hash mark join shape, so reject it during hash join build/probe operator initialization instead of allowing the unsupported plan to proceed.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- Added and ran `HashJoinProbeOperatorTest.RightAntiMarkJoinRejected`
- Ran `HashJoinProbeOperatorTest.RightSemiJoinMarkJoin`
- Behavior changed: No
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Reject right anti hash mark join plans [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63482:
URL: https://github.com/apache/doris/pull/63482#issuecomment-4507305813
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Reject right anti hash mark join plans [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63482:
URL: https://github.com/apache/doris/pull/63482#issuecomment-4507312417
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](be) Reject right anti hash mark join plans [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63482:
URL: https://github.com/apache/doris/pull/63482#issuecomment-4509662068
<details>
<summary>TPC-DS: <b>Total hot run time: 168022 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9d50a2b7fbc14fd693c5ac1e2fa6012e05d56e87, data reload: false
query5 4346 645 505 505
query6 347 228 199 199
query7 4322 539 304 304
query8 324 228 211 211
query9 8797 3953 3928 3928
query10 454 332 287 287
query11 5844 2424 2196 2196
query12 187 124 120 120
query13 1312 554 422 422
query14 5983 5377 5061 5061
query14_1 4367 4341 4360 4341
query15 217 202 183 183
query16 1046 470 370 370
query17 1149 745 605 605
query18 2635 487 364 364
query19 216 204 168 168
query20 139 133 128 128
query21 212 140 117 117
query22 13557 13532 13327 13327
query23 17102 16320 16109 16109
query23_1 16077 15984 16099 15984
query24 7440 1755 1305 1305
query24_1 1300 1297 1288 1288
query25 534 469 415 415
query26 1311 319 168 168
query27 2689 543 326 326
query28 4437 1939 1936 1936
query29 962 619 482 482
query30 297 229 197 197
query31 1137 1051 936 936
query32 89 73 74 73
query33 525 342 279 279
query34 1175 1116 622 622
query35 763 784 670 670
query36 1342 1341 1141 1141
query37 152 99 91 91
query38 3208 3124 3032 3032
query39 922 910 889 889
query39_1 860 862 874 862
query40 228 148 123 123
query41 68 63 68 63
query42 110 106 105 105
query43 327 321 277 277
query44
query45 204 199 191 191
query46 1069 1143 732 732
query47 2241 2349 2176 2176
query48 386 428 301 301
query49 623 484 374 374
query50 946 336 243 243
query51 4329 4329 4262 4262
query52 117 104 94 94
query53 256 280 217 217
query54 305 263 247 247
query55 94 89 83 83
query56 319 304 306 304
query57 1430 1406 1355 1355
query58 302 263 272 263
query59 1552 1607 1432 1432
query60 318 315 301 301
query61 161 152 146 146
query62 670 622 554 554
query63 245 196 205 196
query64 2399 796 612 612
query65
query66 1696 466 348 348
query67 29890 29882 29751 29751
query68
query69 456 349 307 307
query70 1054 1029 983 983
query71 300 274 263 263
query72 2911 2669 2369 2369
query73 824 736 424 424
query74 5062 4868 4727 4727
query75 2656 2582 2235 2235
query76 2293 1142 743 743
query77 391 398 331 331
query78 12111 12115 11499 11499
query79 1451 1045 753 753
query80 653 541 456 456
query81 453 282 244 244
query82 1253 162 126 126
query83 364 280 249 249
query84 272 146 117 117
query85 961 609 531 531
query86 386 372 311 311
query87 3390 3353 3283 3283
query88 3515 2684 2667 2667
query89 437 379 333 333
query90 1907 193 180 180
query91 196 187 157 157
query92 81 80 75 75
query93 1442 1361 884 884
query94 565 335 307 307
query95 679 381 368 368
query96 1008 781 320 320
query97 2688 2679 2543 2543
query98 235 222 219 219
query99 1104 1085 940 940
Total cold run time: 252136 ms
Total hot run time: 168022 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](be) Reject right anti hash mark join plans [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63482:
URL: https://github.com/apache/doris/pull/63482#issuecomment-4509564628
<details>
<summary>TPC-H: <b>Total hot run time: 30653 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9d50a2b7fbc14fd693c5ac1e2fa6012e05d56e87, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17610 3928 3897 3897
q2 q3 10798 1394 810 810
q4 4687 467 346 346
q5 7565 2209 2074 2074
q6 257 172 136 136
q7 963 745 636 636
q8 9375 1788 1491 1491
q9 6782 4927 4904 4904
q10 6312 2122 1806 1806
q11 448 279 239 239
q12 690 428 289 289
q13 18220 3354 2754 2754
q14 259 255 231 231
q15 q16 819 770 709 709
q17 993 964 1044 964
q18 7004 5663 5516 5516
q19 1194 1185 941 941
q20 502 410 260 260
q21 5836 2646 2342 2342
q22 418 357 308 308
Total cold run time: 100732 ms
Total hot run time: 30653 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4183 4160 4151 4151
q2 q3 4444 4871 4296 4296
q4 2078 2192 1379 1379
q5 4362 4224 4249 4224
q6 225 171 129 129
q7 2101 1874 1666 1666
q8 2472 2068 2019 2019
q9 7837 7811 7618 7618
q10 4506 4458 4080 4080
q11 567 396 391 391
q12 713 835 592 592
q13 3314 3553 3027 3027
q14 297 305 270 270
q15 q16 710 748 656 656
q17 1331 1294 1314 1294
q18 7800 7312 6806 6806
q19 1073 1110 1087 1087
q20 2236 2170 1928 1928
q21 5259 4621 4442 4442
q22 510 460 406 406
Total cold run time: 56018 ms
Total hot run time: 50461 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Reject right anti hash mark join plans [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63482:
URL: https://github.com/apache/doris/pull/63482#issuecomment-4506799688
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](be) Reject right anti hash mark join plans [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63482:
URL: https://github.com/apache/doris/pull/63482#issuecomment-4508784345
<details>
<summary>TPC-H: <b>Total hot run time: 31162 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9d50a2b7fbc14fd693c5ac1e2fa6012e05d56e87, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17595 3870 3836 3836
q2 q3 10789 1328 802 802
q4 4680 475 345 345
q5 7542 2323 2077 2077
q6 229 174 135 135
q7 958 779 640 640
q8 9342 1696 1710 1696
q9 5071 4863 4881 4863
q10 6409 2068 1793 1793
q11 434 277 244 244
q12 631 425 296 296
q13 18096 3346 2844 2844
q14 259 253 230 230
q15 q16 820 762 707 707
q17 1004 971 951 951
q18 6847 5643 5467 5467
q19 1320 1288 1029 1029
q20 538 407 263 263
q21 6239 2834 2610 2610
q22 467 369 334 334
Total cold run time: 99270 ms
Total hot run time: 31162 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4792 4532 4466 4466
q2 q3 4760 5321 4784 4784
q4 2127 2212 1409 1409
q5 4728 4789 4605 4605
q6 232 174 128 128
q7 1806 1759 1544 1544
q8 2424 2073 2173 2073
q9 7710 7715 7180 7180
q10 4506 4399 4006 4006
q11 525 377 350 350
q12 716 710 508 508
q13 2986 3368 2818 2818
q14 265 291 257 257
q15 q16 689 697 599 599
q17 1275 1255 1236 1236
q18 7256 6865 6864 6864
q19 1093 1089 1105 1089
q20 2202 2204 1935 1935
q21 5284 4707 4519 4519
q22 506 471 408 408
Total cold run time: 55882 ms
Total hot run time: 50778 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](be) Reject right anti hash mark join plans [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #63482:
URL: https://github.com/apache/doris/pull/63482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Reject right anti hash mark join plans [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #63482:
URL: https://github.com/apache/doris/pull/63482#discussion_r3280159649
##########
be/src/exec/operator/partitioned_hash_join_probe_operator.cpp:
##########
@@ -542,6 +542,9 @@ Status PartitionedHashJoinProbeOperatorX::init(const TPlanNode& tnode, RuntimeSt
// default repartition max depth; can be overridden from session variable
_repartition_max_depth = state->spill_repartition_max_depth();
RETURN_IF_ERROR(JoinProbeOperatorX::init(tnode, state));
+ if (_is_mark_join && _join_op == TJoinOp::RIGHT_ANTI_JOIN) {
+ return Status::InternalError("Hash join does not support right anti mark join");
Review Comment:
This new InternalError message doesn't include query/node context, which can make diagnosing malformed plans harder in production logs. Consider adding query id + node id (and/or to_string(_join_op) / debug_string) to the error so operators can be located quickly when this triggers.
##########
be/src/exec/operator/hashjoin_probe_operator.cpp:
##########
@@ -461,6 +461,9 @@ Status HashJoinProbeOperatorX::push(RuntimeState* state, Block* input_block, boo
Status HashJoinProbeOperatorX::init(const TPlanNode& tnode, RuntimeState* state) {
RETURN_IF_ERROR(JoinProbeOperatorX<HashJoinProbeLocalState>::init(tnode, state));
DCHECK(tnode.__isset.hash_join_node);
+ if (_is_mark_join && _join_op == TJoinOp::RIGHT_ANTI_JOIN) {
+ return Status::InternalError("Hash join does not support right anti mark join");
+ }
Review Comment:
The same right-anti mark-join validation is duplicated across multiple hash join operator init() implementations (build/probe, partitioned/unpartitioned). This duplication risks the checks diverging over time; consider factoring it into a shared helper (e.g., in JoinBuildSinkOperatorX/JoinProbeOperatorX base init) so all hash-join variants enforce the same supported mark-join shapes in one place.
##########
be/src/exec/operator/hashjoin_build_sink.cpp:
##########
@@ -700,6 +700,9 @@ HashJoinBuildSinkOperatorX::HashJoinBuildSinkOperatorX(ObjectPool* pool, int ope
Status HashJoinBuildSinkOperatorX::init(const TPlanNode& tnode, RuntimeState* state) {
RETURN_IF_ERROR(JoinBuildSinkOperatorX::init(tnode, state));
DCHECK(tnode.__isset.hash_join_node);
+ if (_is_mark_join && _join_op == TJoinOp::RIGHT_ANTI_JOIN) {
+ return Status::InternalError("Hash join does not support right anti mark join");
+ }
Review Comment:
New behavior is introduced here (rejecting right anti mark join) but the added unit test only asserts HashJoinProbeOperatorX::init() fails; the same check was also added to build-sink and partitioned hash-join operators. Please add/extend unit tests (e.g., in hashjoin_build_sink_test.cpp and partitioned_hash_join_*_operator_test.cpp) to cover these additional init() rejection paths so regressions are caught.
##########
be/test/exec/operator/hashjoin_probe_operator_test.cpp:
##########
@@ -919,6 +919,17 @@ TEST_F(HashJoinProbeOperatorTest, RightSemiJoinMarkJoin) {
Field::create_field<TYPE_BOOLEAN>(1), Field::create_field<TYPE_BOOLEAN>(0)});
}
+TEST_F(HashJoinProbeOperatorTest, RightAntiMarkJoinRejected) {
+ auto tnode = _helper.create_test_plan_node(TJoinOp::RIGHT_ANTI_JOIN, {TPrimitiveType::INT},
+ {false}, {false}, true, 1);
Review Comment:
This test constructs a mark-join plan with key_types.size()==mark_join_conjuncts_size, which (via HashJoinTestHelper::create_test_plan_node) results in eq_join_conjuncts being empty. To make the test less fragile and closer to real plans, consider ensuring at least one eq_join_conjunct remains (e.g., by using 2 key columns and mark_join_conjuncts_size=1) so the failure is clearly attributable to the right-anti mark-join rejection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](be) Reject right anti hash mark join plans [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63482:
URL: https://github.com/apache/doris/pull/63482#issuecomment-4509279686
# BE UT Coverage Report
Increment line coverage `62.50% (5/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9d50a2b7fbc14fd693c5ac1e2fa6012e05d56e87_9d50a2b7fbc14fd693c5ac1e2fa6012e05d56e87/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9d50a2b7fbc14fd693c5ac1e2fa6012e05d56e87_9d50a2b7fbc14fd693c5ac1e2fa6012e05d56e87/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.64% (20735/38658) |
| Line Coverage | 37.22% (196331/527474) |
| Region Coverage | 33.55% (153920/458743) |
| Branch Coverage | 34.57% (67073/194048) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](be) Reject right anti hash mark join plans [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63482:
URL: https://github.com/apache/doris/pull/63482#issuecomment-4508595764
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Chore](be) Reject right anti hash mark join plans [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63482:
URL: https://github.com/apache/doris/pull/63482#issuecomment-4514494532
# BE Regression && UT Coverage Report
Increment line coverage `62.50% (5/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63482_9d50a2b7fbc14fd693c5ac1e2fa6012e05d56e87_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63482_9d50a2b7fbc14fd693c5ac1e2fa6012e05d56e87_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.74% (27922/37864) |
| Line Coverage | 57.63% (303211/526128) |
| Region Coverage | 54.93% (254413/463172) |
| Branch Coverage | 56.39% (109828/194775) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org