You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2023/11/20 03:20:54 UTC
[PR] [opt](nereids) penalty to shuffle join if data skew [doris]
englefly opened a new pull request, #27259:
URL: https://github.com/apache/doris/pull/27259
## Proposed changes
data skew reduces shuffle join parallelism, where broadcast join is a better choice.
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1818258842
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.31 seconds
stream load tsv: 589 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17099864926 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1829418780
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 4d6d4aff6c23ce4d8913a5cb105ca3e586234423, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4921 4645 4660 4645
q2 359 162 160 160
q3 2053 1875 1918 1875
q4 1398 1259 1226 1226
q5 3964 3925 4005 3925
q6 264 132 135 132
q7 1362 855 835 835
q8 2842 2826 2777 2777
q9 9605 10325 9427 9427
q10 3430 3534 3505 3505
q11 388 256 245 245
q12 445 295 298 295
q13 4533 3812 3784 3784
q14 315 295 286 286
q15 578 540 521 521
q16 486 461 452 452
q17 1148 979 953 953
q18 7939 7576 7310 7310
q19 1670 1686 1672 1672
q20 573 318 296 296
q21 4445 4013 4012 4012
q22 467 369 381 369
Total cold run time: 53185 ms
Total hot run time: 48702 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4567 4544 4581 4544
q2 348 229 275 229
q3 4035 4021 4018 4018
q4 2718 2697 2709 2697
q5 9635 9579 9638 9579
q6 253 126 126 126
q7 3001 2504 2541 2504
q8 4457 4462 4462 4462
q9 12934 12806 12863 12806
q10 4056 4183 4180 4180
q11 820 672 632 632
q12 979 808 808 808
q13 4308 3587 3536 3536
q14 376 353 354 353
q15 577 510 516 510
q16 621 574 572 572
q17 3877 3853 3893 3853
q18 9616 9014 9063 9014
q19 1834 1789 1793 1789
q20 2403 2104 2059 2059
q21 8919 8652 8506 8506
q22 888 801 783 783
Total cold run time: 81222 ms
Total hot run time: 77560 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly closed pull request #27259: [opt](nereids) penalty to shuffle join if data skew
URL: https://github.com/apache/doris/pull/27259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1827065590
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.65 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 27 seconds loaded 2358488459 Bytes, about 83 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17099208323 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1818297679
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 66a015ac0bbc7b21b70e57dfd2e8514ba6b651d0, data reload: true
run tpch-sf100 query with default conf and session variables
q1 6557 4605 4602 4602
q2 417 136 127 127
q3 2104 1900 1869 1869
q4 3510 1161 1230 1161
q5 4063 3998 3975 3975
q6 241 125 127 125
q7 1376 893 877 877
q8 4091 2824 2758 2758
q9 10006 9753 9624 9624
q10 10096 3556 3548 3548
q11 377 237 230 230
q12 447 284 291 284
q13 11589 3846 3775 3775
q14 316 274 284 274
q15 569 528 514 514
q16 654 585 571 571
q17 1155 997 952 952
q18 7866 7455 7446 7446
q19 1720 1674 1682 1674
q20 522 322 305 305
q21 4434 3964 3984 3964
q22 471 371 365 365
Total cold run time: 72581 ms
Total hot run time: 49020 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4562 4585 4596 4585
q2 339 252 247 247
q3 4030 3978 3991 3978
q4 2703 2704 2691 2691
q5 9700 9585 9673 9585
q6 239 126 120 120
q7 3037 2493 2633 2493
q8 4666 4741 4579 4579
q9 13304 13149 13255 13149
q10 4114 4221 4197 4197
q11 793 651 652 651
q12 979 821 822 821
q13 4400 3552 3596 3552
q14 373 331 337 331
q15 581 524 533 524
q16 732 676 674 674
q17 3878 3885 3890 3885
q18 9610 9035 9085 9035
q19 1833 1784 1796 1784
q20 2403 2080 2056 2056
q21 8764 8452 8442 8442
q22 901 799 767 767
Total cold run time: 81941 ms
Total hot run time: 78146 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1829383966
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.57 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 27 seconds loaded 2358488459 Bytes, about 83 MB/s
stream load orc: 70 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17098467949 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #27259:
URL: https://github.com/apache/doris/pull/27259#discussion_r1398635841
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java:
##########
@@ -299,12 +303,41 @@ public Cost visitPhysicalHashJoin(
0
);
}
- return CostV1.of(context.getSessionVariable(), leftRowCount + rightRowCount + outputRowCount,
+ return CostV1.of(context.getSessionVariable(),
Review Comment:
The tpcds 1T perf data would be better collected and considered, especially q4/q11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1826981990
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #27259:
URL: https://github.com/apache/doris/pull/27259#discussion_r1399931982
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java:
##########
@@ -299,12 +303,41 @@ public Cost visitPhysicalHashJoin(
0
);
}
- return CostV1.of(context.getSessionVariable(), leftRowCount + rightRowCount + outputRowCount,
+ return CostV1.of(context.getSessionVariable(),
+ (leftRowCount + rightRowCount + outputRowCount)
+ * skewPenalty(physicalHashJoin, probeStats, buildStats, totalInstanceNumber),
rightRowCount,
0
);
}
+ private double skewPenalty(PhysicalHashJoin<? extends Plan, ? extends Plan> physicalHashJoin,
+ Statistics probeStats,
+ Statistics buildStats,
+ int totalInstanceNumber) {
+ double penalty = 1.0;
+ for (Expression conj : physicalHashJoin.getHashJoinConjuncts()) {
+ EqualTo eq = (EqualTo) JoinUtils.swapEqualToForChildrenOrder(
+ (EqualTo) conj, physicalHashJoin.left().getOutputSet());
+ ColumnStatistic leftColStats = probeStats.findColumnStatistics(eq.left());
+ if (leftColStats != null && !leftColStats.isUnKnown()) {
+ if (leftColStats.ndv < beNumber) {
+ penalty = totalInstanceNumber / Math.max(1.0, leftColStats.ndv);
Review Comment:
1. optimizer does not know the number of instances assigned to this PlanNode. totalInstanceNumber is the upper bound.
2. it is more important to balance workload between BEs than that of between instances, that is why penalty is triggered when "ndv < beNumber"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1818166954
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #27259:
URL: https://github.com/apache/doris/pull/27259#discussion_r1398634610
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java:
##########
@@ -299,12 +303,41 @@ public Cost visitPhysicalHashJoin(
0
);
}
- return CostV1.of(context.getSessionVariable(), leftRowCount + rightRowCount + outputRowCount,
+ return CostV1.of(context.getSessionVariable(),
+ (leftRowCount + rightRowCount + outputRowCount)
+ * skewPenalty(physicalHashJoin, probeStats, buildStats, totalInstanceNumber),
rightRowCount,
0
);
}
+ private double skewPenalty(PhysicalHashJoin<? extends Plan, ? extends Plan> physicalHashJoin,
+ Statistics probeStats,
+ Statistics buildStats,
+ int totalInstanceNumber) {
+ double penalty = 1.0;
+ for (Expression conj : physicalHashJoin.getHashJoinConjuncts()) {
+ EqualTo eq = (EqualTo) JoinUtils.swapEqualToForChildrenOrder(
+ (EqualTo) conj, physicalHashJoin.left().getOutputSet());
+ ColumnStatistic leftColStats = probeStats.findColumnStatistics(eq.left());
+ if (leftColStats != null && !leftColStats.isUnKnown()) {
+ if (leftColStats.ndv < beNumber) {
+ penalty = totalInstanceNumber / Math.max(1.0, leftColStats.ndv);
+ break;
+ }
+ }
+
+ ColumnStatistic rightColStats = buildStats.findColumnStatistics(eq.right());
+ if (rightColStats != null && !rightColStats.isUnKnown()) {
+ if (rightColStats.ndv < beNumber) {
+ penalty = totalInstanceNumber / Math.max(1.0, leftColStats.ndv);
Review Comment:
right side also depends on leftColStats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #27259:
URL: https://github.com/apache/doris/pull/27259#discussion_r1398634384
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java:
##########
@@ -299,12 +303,41 @@ public Cost visitPhysicalHashJoin(
0
);
}
- return CostV1.of(context.getSessionVariable(), leftRowCount + rightRowCount + outputRowCount,
+ return CostV1.of(context.getSessionVariable(),
+ (leftRowCount + rightRowCount + outputRowCount)
Review Comment:
please add comments to explain why the factor is necessary and why the factor is on the original total cost, not on the cost respectively
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #27259:
URL: https://github.com/apache/doris/pull/27259#discussion_r1399933480
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java:
##########
@@ -299,12 +303,41 @@ public Cost visitPhysicalHashJoin(
0
);
}
- return CostV1.of(context.getSessionVariable(), leftRowCount + rightRowCount + outputRowCount,
+ return CostV1.of(context.getSessionVariable(),
+ (leftRowCount + rightRowCount + outputRowCount)
+ * skewPenalty(physicalHashJoin, probeStats, buildStats, totalInstanceNumber),
rightRowCount,
0
);
}
+ private double skewPenalty(PhysicalHashJoin<? extends Plan, ? extends Plan> physicalHashJoin,
+ Statistics probeStats,
+ Statistics buildStats,
+ int totalInstanceNumber) {
+ double penalty = 1.0;
+ for (Expression conj : physicalHashJoin.getHashJoinConjuncts()) {
+ EqualTo eq = (EqualTo) JoinUtils.swapEqualToForChildrenOrder(
+ (EqualTo) conj, physicalHashJoin.left().getOutputSet());
+ ColumnStatistic leftColStats = probeStats.findColumnStatistics(eq.left());
+ if (leftColStats != null && !leftColStats.isUnKnown()) {
+ if (leftColStats.ndv < beNumber) {
+ penalty = totalInstanceNumber / Math.max(1.0, leftColStats.ndv);
+ break;
+ }
+ }
+
+ ColumnStatistic rightColStats = buildStats.findColumnStatistics(eq.right());
+ if (rightColStats != null && !rightColStats.isUnKnown()) {
+ if (rightColStats.ndv < beNumber) {
+ penalty = totalInstanceNumber / Math.max(1.0, leftColStats.ndv);
Review Comment:
if right.ndv=1, then only one instance build hash table with full data, and others build empty hash table. All the workload is concentrated on a single node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #27259:
URL: https://github.com/apache/doris/pull/27259#discussion_r1398635327
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java:
##########
@@ -299,12 +303,41 @@ public Cost visitPhysicalHashJoin(
0
);
}
- return CostV1.of(context.getSessionVariable(), leftRowCount + rightRowCount + outputRowCount,
+ return CostV1.of(context.getSessionVariable(),
+ (leftRowCount + rightRowCount + outputRowCount)
+ * skewPenalty(physicalHashJoin, probeStats, buildStats, totalInstanceNumber),
rightRowCount,
0
);
}
+ private double skewPenalty(PhysicalHashJoin<? extends Plan, ? extends Plan> physicalHashJoin,
+ Statistics probeStats,
+ Statistics buildStats,
+ int totalInstanceNumber) {
+ double penalty = 1.0;
+ for (Expression conj : physicalHashJoin.getHashJoinConjuncts()) {
+ EqualTo eq = (EqualTo) JoinUtils.swapEqualToForChildrenOrder(
+ (EqualTo) conj, physicalHashJoin.left().getOutputSet());
+ ColumnStatistic leftColStats = probeStats.findColumnStatistics(eq.left());
+ if (leftColStats != null && !leftColStats.isUnKnown()) {
+ if (leftColStats.ndv < beNumber) {
+ penalty = totalInstanceNumber / Math.max(1.0, leftColStats.ndv);
Review Comment:
consider the most commonly used case, 3 be and leftColStats is 2 (meet < beNumber), the totalInstanceNumber is 24, then the penalty will be 12 on the original total cost for shuffle join, which seems too big changes...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1831078850
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 797aeb6121a571d28a9939f30b953a7dde930699, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4895 4651 4646 4646
q2 376 177 157 157
q3 1507 1311 1278 1278
q4 1154 971 942 942
q5 3239 3221 3229 3221
q6 250 132 136 132
q7 984 516 511 511
q8 2243 2241 2226 2226
q9 6966 6943 6986 6943
q10 3298 3370 3358 3358
q11 335 218 209 209
q12 354 213 229 213
q13 4638 3912 4167 3912
q14 255 209 221 209
q15 589 542 521 521
q16 430 401 379 379
q17 1025 646 635 635
q18 7967 7347 7392 7347
q19 1538 1507 1558 1507
q20 588 331 323 323
q21 3417 2979 2936 2936
q22 371 297 307 297
Total cold run time: 46419 ms
Total hot run time: 41902 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4628 4577 4584 4577
q2 316 199 216 199
q3 3748 3740 3720 3720
q4 2558 2543 2525 2525
q5 6178 6174 6186 6174
q6 249 125 125 125
q7 2566 2015 1968 1968
q8 3709 3917 3696 3696
q9 9386 9321 9336 9321
q10 4042 4114 4162 4114
q11 635 514 525 514
q12 777 623 614 614
q13 4392 3656 3589 3589
q14 280 257 248 248
q15 593 532 529 529
q16 519 500 483 483
q17 2078 2070 2072 2070
q18 9594 9001 9066 9001
q19 1817 1746 1748 1746
q20 2302 1991 1980 1980
q21 7441 6874 6886 6874
q22 715 555 562 555
Total cold run time: 68523 ms
Total hot run time: 64622 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1830989471
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1818183997
Pls add the case and skew data here to describe the necessary and background of this improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1829569812
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1829230298
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1831108286
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.78 seconds
stream load tsv: 585 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17100959591 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) penalty to shuffle join if data skew [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27259:
URL: https://github.com/apache/doris/pull/27259#issuecomment-1827058766
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit cb7efb4e89f68e477f47f31580a19632f1b4300b, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4974 4640 4671 4640
q2 359 167 160 160
q3 2067 1943 1906 1906
q4 1394 1257 1244 1244
q5 3963 3928 4023 3928
q6 251 130 134 130
q7 1435 881 885 881
q8 2795 2786 2808 2786
q9 9840 9445 9424 9424
q10 3482 3489 3523 3489
q11 392 243 252 243
q12 436 288 284 284
q13 4589 3789 3806 3789
q14 313 282 278 278
q15 576 535 535 535
q16 660 594 578 578
q17 1134 969 934 934
q18 7907 7589 7381 7381
q19 1664 1683 1684 1683
q20 552 324 301 301
q21 4481 3986 4049 3986
q22 478 375 368 368
Total cold run time: 53742 ms
Total hot run time: 48948 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4590 4580 4560 4560
q2 346 229 241 229
q3 4037 4023 4010 4010
q4 2744 2738 2703 2703
q5 9731 9673 9767 9673
q6 249 127 123 123
q7 3039 2470 2480 2470
q8 4491 4449 4468 4449
q9 13101 12771 12772 12771
q10 4066 4127 4146 4127
q11 763 701 649 649
q12 972 803 819 803
q13 4309 3592 3584 3584
q14 380 357 346 346
q15 580 512 527 512
q16 748 666 686 666
q17 3928 3855 3825 3825
q18 9561 9138 9099 9099
q19 1790 1797 1768 1768
q20 2410 2077 2044 2044
q21 8962 8577 8476 8476
q22 900 760 856 760
Total cold run time: 81697 ms
Total hot run time: 77647 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org