You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2023/12/22 12:51:26 UTC
[PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
mrhhsg opened a new pull request, #28898:
URL: https://github.com/apache/doris/pull/28898
## Proposed changes
In a left anti join, if the right side is empty, all rows from the left side should be output, regardless of whether the row is null or not.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1867660536
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of null aware left anti join with empty build side [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1868170864
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.42 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17183822767 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1867653061
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1867700347
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1867658893
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1867706657
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of null aware left anti join with empty build side [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1868152643
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #28898:
URL: https://github.com/apache/doris/pull/28898#discussion_r1435037595
##########
be/src/vec/common/hash_table/hash_map.h:
##########
@@ -338,6 +341,26 @@ class JoinHashMapTable : public HashMapTable<Key, Cell, Hash, Grower, Allocator>
auto matched_cnt = 0;
const auto batch_size = max_batch_size;
+ if (_empty_table) {
Review Comment:
add a function to process case when left anti+empty
##########
be/src/vec/common/hash_table/hash_map.h:
##########
@@ -338,6 +341,26 @@ class JoinHashMapTable : public HashMapTable<Key, Cell, Hash, Grower, Allocator>
auto matched_cnt = 0;
const auto batch_size = max_batch_size;
+ if (_empty_table) {
+ if constexpr (JoinOpType == doris::TJoinOp::LEFT_ANTI_JOIN ||
+ JoinOpType == doris::TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN ||
+ JoinOpType == doris::TJoinOp::LEFT_SEMI_JOIN) {
+ while (probe_idx < probe_rows && matched_cnt < batch_size) {
+ probe_idxs[matched_cnt] = probe_idx++;
+ build_idxs[matched_cnt] = 0;
+ matched_cnt++;
+ }
+
+ if constexpr (!with_other_conjuncts) {
+ mark_column->resize_fill(matched_cnt,
+ JoinOpType != doris::TJoinOp::LEFT_SEMI_JOIN);
+ }
+ } else {
+ LOG(FATAL) << "CROSS JOIN SHOULD NOT BUILD HASH TABLE";
Review Comment:
do not log fatal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1867750985
TeamCity be ut coverage result:
Function Coverage: 36.54% (8540/23374)
Line Coverage: 28.62% (69437/242609)
Region Coverage: 27.64% (35931/129995)
Branch Coverage: 24.36% (18360/75362)
Coverage Report: http://coverage.selectdb-in.cc/coverage/82a270d716f272d55bc32092a42d02482820252b_82a270d716f272d55bc32092a42d02482820252b/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of null aware left anti join with empty build side [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1868163293
TeamCity be ut coverage result:
Function Coverage: 36.52% (8537/23374)
Line Coverage: 28.60% (69391/242596)
Region Coverage: 27.62% (35908/129995)
Branch Coverage: 24.36% (18357/75364)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b9824820d044258f8e35285296c9065ace010928_b9824820d044258f8e35285296c9065ace010928/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of null aware left anti join with empty build side [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #28898:
URL: https://github.com/apache/doris/pull/28898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #28898:
URL: https://github.com/apache/doris/pull/28898#discussion_r1435038222
##########
be/src/vec/common/hash_table/hash_map.h:
##########
@@ -338,6 +341,26 @@ class JoinHashMapTable : public HashMapTable<Key, Cell, Hash, Grower, Allocator>
auto matched_cnt = 0;
const auto batch_size = max_batch_size;
+ if (_empty_table) {
+ if constexpr (JoinOpType == doris::TJoinOp::LEFT_ANTI_JOIN ||
+ JoinOpType == doris::TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN ||
+ JoinOpType == doris::TJoinOp::LEFT_SEMI_JOIN) {
Review Comment:
does left semi need to match?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of null aware left anti join with empty build side [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1868166842
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit b9824820d044258f8e35285296c9065ace010928, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4735 4449 4427 4427
q2 373 141 159 141
q3 1455 1268 1240 1240
q4 1096 905 887 887
q5 3166 3134 3126 3126
q6 249 135 127 127
q7 979 488 494 488
q8 2189 2215 2171 2171
q9 6688 6668 6677 6668
q10 3217 3267 3238 3238
q11 309 182 180 180
q12 350 213 213 213
q13 4582 3762 3811 3762
q14 239 212 212 212
q15 565 522 517 517
q16 436 393 395 393
q17 1013 637 523 523
q18 7122 6809 6712 6712
q19 1537 1411 1451 1411
q20 545 324 316 316
q21 3045 2648 2630 2630
q22 347 283 291 283
Total cold run time: 44237 ms
Total hot run time: 39665 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4354 4363 4364 4363
q2 272 165 170 165
q3 3520 3490 3495 3490
q4 2391 2356 2360 2356
q5 5691 5691 5705 5691
q6 245 117 121 117
q7 2363 1852 1864 1852
q8 3507 3528 3528 3528
q9 8998 9003 8978 8978
q10 3935 3983 3976 3976
q11 480 366 356 356
q12 758 600 640 600
q13 4297 3573 3553 3553
q14 291 256 255 255
q15 565 523 518 518
q16 495 456 461 456
q17 1877 1838 1818 1818
q18 8459 8078 7991 7991
q19 1744 1773 1739 1739
q20 2241 1929 1925 1925
q21 6476 6100 6134 6100
q22 489 420 436 420
Total cold run time: 63448 ms
Total hot run time: 60247 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of null aware left anti join with empty build side [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1868156852
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of null aware left anti join with empty build side [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1868156855
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1867778033
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 82a270d716f272d55bc32092a42d02482820252b, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4670 4405 4418 4405
q2 370 146 158 146
q3 1452 1263 1197 1197
q4 1107 911 897 897
q5 3148 3157 3161 3157
q6 244 130 135 130
q7 1009 492 499 492
q8 2190 2239 2185 2185
q9 6665 6683 6665 6665
q10 3207 3283 3243 3243
q11 309 183 183 183
q12 350 205 205 205
q13 4568 3808 3789 3789
q14 244 212 212 212
q15 575 525 521 521
q16 435 381 383 381
q17 1021 572 558 558
q18 7066 6694 6862 6694
q19 1509 1472 1436 1436
q20 533 293 290 290
q21 3058 2722 2631 2631
q22 353 282 291 282
Total cold run time: 44083 ms
Total hot run time: 39699 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4397 4326 4342 4326
q2 274 165 178 165
q3 3499 3495 3486 3486
q4 2399 2368 2379 2368
q5 5711 5708 5709 5708
q6 241 122 120 120
q7 2349 1858 1866 1858
q8 3527 3523 3520 3520
q9 9045 8980 8975 8975
q10 3926 4017 3990 3990
q11 481 369 365 365
q12 766 596 585 585
q13 4299 3523 3569 3523
q14 285 243 260 243
q15 574 529 523 523
q16 492 451 470 451
q17 1904 1877 1834 1834
q18 8630 8141 8113 8113
q19 1743 1749 1764 1749
q20 2244 1964 1936 1936
q21 6508 6140 6136 6136
q22 496 424 419 419
Total cold run time: 63790 ms
Total hot run time: 60393 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1867780100
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.24 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.7 seconds inserted 10000000 Rows, about 336K ops/s
storage size: 17184151637 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](join) incorrect result of left semi/anti join with empty build side [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #28898:
URL: https://github.com/apache/doris/pull/28898#issuecomment-1868150948
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org