You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaijchen (via GitHub)" <gi...@apache.org> on 2024/03/26 09:59:19 UTC
[PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
kaijchen opened a new pull request, #32845:
URL: https://github.com/apache/doris/pull/32845
## Proposed changes
Single replica load is not compatible with memtable on sink node.
Enabling single replica load when move memtable will cause missing location in OlapTableSink plan.
Ignore single replica load config when memtable on sink node is enabled.
Namely `enable_single_replica_load` config in fe.conf and `enable_single_replica_insert` in session variable.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2053033169
<details>
<summary>TPC-DS: <b>Total hot run time: 184712 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd41d181c19a64bd05c524d0a969eaab59ad6899, data reload: false
query1 895 1120 1118 1118
query2 6169 2845 2494 2494
query3 6639 201 202 201
query4 36901 21645 21457 21457
query5 4141 393 389 389
query6 239 195 179 179
query7 4045 282 285 282
query8 214 166 166 166
query9 5800 2312 2299 2299
query10 369 235 249 235
query11 14511 14326 14176 14176
query12 140 92 87 87
query13 1000 358 357 357
query14 11058 6800 6890 6800
query15 199 181 177 177
query16 6794 255 248 248
query17 1671 570 544 544
query18 1366 293 273 273
query19 198 159 150 150
query20 100 84 89 84
query21 196 133 131 131
query22 5018 4907 4811 4811
query23 33673 33066 33550 33066
query24 11524 3007 3071 3007
query25 544 399 390 390
query26 1074 159 157 157
query27 3285 349 365 349
query28 7108 2077 2110 2077
query29 895 642 628 628
query30 306 172 178 172
query31 944 738 731 731
query32 63 57 55 55
query33 589 255 256 255
query34 935 475 479 475
query35 881 711 707 707
query36 1083 952 950 950
query37 138 69 74 69
query38 3730 3587 3702 3587
query39 1638 1601 1583 1583
query40 174 133 126 126
query41 47 44 42 42
query42 109 102 98 98
query43 613 575 571 571
query44 1425 771 754 754
query45 297 279 281 279
query46 1110 743 750 743
query47 2014 1969 2031 1969
query48 364 321 300 300
query49 798 385 386 385
query50 786 400 421 400
query51 7007 6933 6889 6889
query52 103 93 87 87
query53 345 281 275 275
query54 246 237 214 214
query55 72 72 69 69
query56 240 216 220 216
query57 1199 1111 1132 1111
query58 212 200 198 198
query59 3488 3448 3338 3338
query60 256 231 239 231
query61 94 93 90 90
query62 605 458 461 458
query63 307 277 274 274
query64 4070 4007 3933 3933
query65 3055 3045 3018 3018
query66 766 321 326 321
query67 15778 15278 14816 14816
query68 7117 531 535 531
query69 590 310 309 309
query70 1216 1230 1161 1161
query71 450 280 265 265
query72 6403 2599 2486 2486
query73 819 313 312 312
query74 7074 6526 6399 6399
query75 3250 2388 2327 2327
query76 4211 1088 1139 1088
query77 624 244 247 244
query78 11054 10194 10261 10194
query79 6717 523 521 521
query80 1949 433 419 419
query81 540 249 235 235
query82 1585 98 90 90
query83 347 178 166 166
query84 262 83 85 83
query85 1154 265 260 260
query86 456 289 274 274
query87 3745 3520 3501 3501
query88 6012 2277 2289 2277
query89 506 370 372 370
query90 1850 175 176 175
query91 124 95 93 93
query92 60 48 46 46
query93 5621 512 506 506
query94 1083 182 242 182
query95 376 291 295 291
query96 594 256 256 256
query97 2721 2470 2476 2470
query98 236 216 222 216
query99 1164 844 851 844
Total cold run time: 301563 ms
Total hot run time: 184712 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #32845:
URL: https://github.com/apache/doris/pull/32845#discussion_r1568143895
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java:
##########
@@ -1023,13 +1023,16 @@ private DataSink createDataSink() throws AnalysisException {
}
if (targetTable instanceof OlapTable) {
OlapTableSink sink;
+ final boolean enableSingleReplicaLoad =
+ analyzer.getContext().getSessionVariable().isEnableMemtableOnSinkNode()
+ ? false : analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert();
if (isGroupCommitStreamLoadSql) {
sink = new GroupCommitBlockSink((OlapTable) targetTable, olapTuple,
- targetPartitionIds, analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert(),
+ targetPartitionIds, enableSingleReplicaLoad,
ConnectContext.get().getSessionVariable().getGroupCommit(), 0);
} else {
sink = new OlapTableSink((OlapTable) targetTable, olapTuple, targetPartitionIds,
- analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert());
+ enableSingleReplicaLoad);
Review Comment:
Currently there is no `single_replica_load` option in nereids.
This PR is for fixing a problem.
If there is a need, we can create a new PR for that feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2020356991
<details>
<summary>TPC-H: <b>Total hot run time: 38373 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f078c7dfe79db7d87e359f816c0e648e93b03ba, data reload: false
------ Round 1 ----------------------------------
q1 17628 4453 4215 4215
q2 2171 163 151 151
q3 11141 1148 1237 1148
q4 10789 780 780 780
q5 8264 3090 3097 3090
q6 210 130 126 126
q7 1060 627 589 589
q8 9532 2083 2031 2031
q9 7466 6853 6685 6685
q10 8474 3508 3528 3508
q11 440 230 217 217
q12 447 199 199 199
q13 17797 2863 2929 2863
q14 232 203 217 203
q15 516 469 472 469
q16 485 374 372 372
q17 971 635 601 601
q18 7242 6529 6362 6362
q19 1562 1391 1470 1391
q20 536 253 250 250
q21 3584 2908 2825 2825
q22 348 298 304 298
Total cold run time: 110895 ms
Total hot run time: 38373 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4110 4073 4092 4073
q2 327 231 232 231
q3 2987 2878 2856 2856
q4 1897 1578 1614 1578
q5 5290 5374 5375 5374
q6 203 118 119 118
q7 2317 1927 1933 1927
q8 3184 3323 3264 3264
q9 8764 8747 8771 8747
q10 3793 3769 3848 3769
q11 540 446 445 445
q12 716 571 552 552
q13 15854 2854 2845 2845
q14 275 259 272 259
q15 494 463 458 458
q16 471 431 423 423
q17 1731 1511 1469 1469
q18 7418 7294 7131 7131
q19 1647 1558 1528 1528
q20 1891 1732 1720 1720
q21 4936 4658 4777 4658
q22 568 461 433 433
Total cold run time: 69413 ms
Total hot run time: 53858 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2022676807
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2020305580
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #32845:
URL: https://github.com/apache/doris/pull/32845#discussion_r1558066309
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java:
##########
@@ -1023,13 +1023,16 @@ private DataSink createDataSink() throws AnalysisException {
}
if (targetTable instanceof OlapTable) {
OlapTableSink sink;
+ final boolean enableSingleReplicaLoad =
+ analyzer.getContext().getSessionVariable().isEnableMemtableOnSinkNode()
+ ? false : analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert();
if (isGroupCommitStreamLoadSql) {
sink = new GroupCommitBlockSink((OlapTable) targetTable, olapTuple,
- targetPartitionIds, analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert(),
+ targetPartitionIds, enableSingleReplicaLoad,
ConnectContext.get().getSessionVariable().getGroupCommit(), 0);
} else {
sink = new OlapTableSink((OlapTable) targetTable, olapTuple, targetPartitionIds,
- analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert());
+ enableSingleReplicaLoad);
Review Comment:
This is a parameter configured by the user, in this case through session variables.
What do you mean "add this feature to nereids"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #32845:
URL: https://github.com/apache/doris/pull/32845#discussion_r1557342473
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java:
##########
@@ -1023,13 +1023,16 @@ private DataSink createDataSink() throws AnalysisException {
}
if (targetTable instanceof OlapTable) {
OlapTableSink sink;
+ final boolean enableSingleReplicaLoad =
+ analyzer.getContext().getSessionVariable().isEnableMemtableOnSinkNode()
+ ? false : analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert();
if (isGroupCommitStreamLoadSql) {
sink = new GroupCommitBlockSink((OlapTable) targetTable, olapTuple,
- targetPartitionIds, analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert(),
+ targetPartitionIds, enableSingleReplicaLoad,
ConnectContext.get().getSessionVariable().getGroupCommit(), 0);
} else {
sink = new OlapTableSink((OlapTable) targetTable, olapTuple, targetPartitionIds,
- analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert());
+ enableSingleReplicaLoad);
Review Comment:
do we need add this feature to nereids?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2053011450
<details>
<summary>TPC-H: <b>Total hot run time: 38555 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd41d181c19a64bd05c524d0a969eaab59ad6899, data reload: false
------ Round 1 ----------------------------------
q1 17601 4262 4194 4194
q2 2006 188 194 188
q3 10458 1161 1208 1161
q4 10193 867 801 801
q5 7519 2688 2649 2649
q6 216 132 133 132
q7 1011 607 596 596
q8 9208 2045 2032 2032
q9 8639 6579 6536 6536
q10 8592 3508 3563 3508
q11 462 234 243 234
q12 452 225 230 225
q13 18486 2952 2992 2952
q14 273 231 230 230
q15 523 465 476 465
q16 509 409 398 398
q17 965 756 654 654
q18 7394 6869 6806 6806
q19 8691 1544 1489 1489
q20 1050 325 307 307
q21 3492 2777 2692 2692
q22 358 311 306 306
Total cold run time: 118098 ms
Total hot run time: 38555 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4381 4241 4183 4183
q2 505 264 266 264
q3 3020 2815 2731 2731
q4 1841 1594 1562 1562
q5 5293 5313 5307 5307
q6 210 125 126 125
q7 2252 1899 1875 1875
q8 3209 3331 3306 3306
q9 8632 8575 8915 8575
q10 4085 4040 3919 3919
q11 596 489 497 489
q12 782 645 672 645
q13 17134 3183 3073 3073
q14 311 283 309 283
q15 530 479 465 465
q16 512 455 461 455
q17 1838 1534 1497 1497
q18 8017 7848 7705 7705
q19 1702 1624 1541 1541
q20 1978 1837 1862 1837
q21 5278 5075 4954 4954
q22 559 471 483 471
Total cold run time: 72665 ms
Total hot run time: 55262 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2053043568
<details>
<summary>ClickBench: <b>Total hot run time: 30.59 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bd41d181c19a64bd05c524d0a969eaab59ad6899, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.04
query3 0.23 0.04 0.04
query4 1.67 0.06 0.07
query5 0.47 0.48 0.47
query6 1.12 0.66 0.65
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.56 0.50 0.50
query10 0.53 0.56 0.55
query11 0.16 0.12 0.12
query12 0.14 0.12 0.11
query13 0.61 0.60 0.58
query14 0.75 0.78 0.78
query15 0.83 0.81 0.82
query16 0.36 0.37 0.38
query17 1.02 1.02 1.02
query18 0.19 0.26 0.24
query19 1.84 1.71 1.67
query20 0.02 0.01 0.01
query21 15.41 0.66 0.65
query22 4.86 6.48 2.24
query23 18.33 1.34 1.23
query24 1.72 0.25 0.25
query25 0.15 0.08 0.08
query26 0.26 0.17 0.16
query27 0.08 0.08 0.08
query28 13.34 0.99 0.97
query29 12.60 3.23 3.24
query30 0.26 0.06 0.06
query31 2.88 0.40 0.38
query32 3.23 0.47 0.46
query33 2.85 2.82 2.88
query34 17.20 4.42 4.41
query35 4.44 4.45 4.46
query36 0.66 0.45 0.46
query37 0.18 0.16 0.15
query38 0.15 0.16 0.14
query39 0.04 0.03 0.04
query40 0.17 0.14 0.13
query41 0.10 0.05 0.04
query42 0.05 0.04 0.04
query43 0.04 0.04 0.04
Total cold run time: 109.69 s
Total hot run time: 30.59 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2053048988
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit bd41d181c19a64bd05c524d0a969eaab59ad6899 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select: 13.4 seconds inserted 10000000 Rows, about 746K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #32845:
URL: https://github.com/apache/doris/pull/32845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2020395291
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 6f078c7dfe79db7d87e359f816c0e648e93b03ba with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 20.9 seconds inserted 10000000 Rows, about 478K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2020013454
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2022676867
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2052978313
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2058418387
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2051855297
run fe_ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2020387835
<details>
<summary>TPC-DS: <b>Total hot run time: 181411 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f078c7dfe79db7d87e359f816c0e648e93b03ba, data reload: false
query1 935 366 360 360
query2 6556 2045 1741 1741
query3 6710 222 217 217
query4 31732 21260 21361 21260
query5 4277 390 385 385
query6 290 183 180 180
query7 4640 285 288 285
query8 234 170 167 167
query9 9448 2234 2254 2234
query10 555 265 265 265
query11 14701 14320 14215 14215
query12 135 92 90 90
query13 1651 426 421 421
query14 9802 7948 7582 7582
query15 263 197 197 197
query16 8179 263 267 263
query17 1992 577 554 554
query18 2085 281 270 270
query19 347 147 151 147
query20 98 89 88 88
query21 204 123 123 123
query22 5033 4856 4920 4856
query23 33564 32714 32516 32516
query24 12061 2865 2838 2838
query25 628 364 374 364
query26 1804 153 150 150
query27 2967 348 358 348
query28 7744 1848 1841 1841
query29 1007 640 601 601
query30 301 147 144 144
query31 948 749 726 726
query32 87 56 59 56
query33 747 244 244 244
query34 1006 487 474 474
query35 814 638 600 600
query36 1005 916 891 891
query37 265 66 66 66
query38 3577 3426 3418 3418
query39 1464 1455 1421 1421
query40 290 113 114 113
query41 49 48 49 48
query42 108 100 99 99
query43 485 447 462 447
query44 1187 736 756 736
query45 274 267 269 267
query46 1126 695 736 695
query47 1914 1874 1872 1872
query48 464 369 359 359
query49 1222 352 351 351
query50 757 379 377 377
query51 6684 6551 6602 6551
query52 102 97 92 92
query53 346 273 275 273
query54 325 233 242 233
query55 82 87 81 81
query56 247 229 230 229
query57 1214 1149 1140 1140
query58 226 217 210 210
query59 2767 2636 2631 2631
query60 270 249 251 249
query61 114 112 147 112
query62 654 454 451 451
query63 304 274 279 274
query64 6265 4073 4120 4073
query65 3081 3013 3034 3013
query66 1424 385 364 364
query67 15135 14996 15090 14996
query68 6628 532 525 525
query69 605 378 384 378
query70 1233 1199 1159 1159
query71 484 278 271 271
query72 6936 2761 2538 2538
query73 710 319 325 319
query74 8317 6461 6372 6372
query75 3577 2214 2192 2192
query76 4350 874 845 845
query77 695 266 253 253
query78 11083 10155 10119 10119
query79 8402 525 518 518
query80 1429 375 367 367
query81 524 211 216 211
query82 1592 88 82 82
query83 206 149 167 149
query84 285 85 78 78
query85 1499 320 318 318
query86 470 316 298 298
query87 3725 3563 3538 3538
query88 5009 2369 2356 2356
query89 531 366 357 357
query90 2069 174 171 171
query91 174 138 137 137
query92 60 47 47 47
query93 7112 500 479 479
query94 1193 175 173 173
query95 443 325 333 325
query96 628 277 279 277
query97 2690 2499 2462 2462
query98 229 205 212 205
query99 1200 958 973 958
Total cold run time: 311060 ms
Total hot run time: 181411 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #32845:
URL: https://github.com/apache/doris/pull/32845#discussion_r1558066309
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java:
##########
@@ -1023,13 +1023,16 @@ private DataSink createDataSink() throws AnalysisException {
}
if (targetTable instanceof OlapTable) {
OlapTableSink sink;
+ final boolean enableSingleReplicaLoad =
+ analyzer.getContext().getSessionVariable().isEnableMemtableOnSinkNode()
+ ? false : analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert();
if (isGroupCommitStreamLoadSql) {
sink = new GroupCommitBlockSink((OlapTable) targetTable, olapTuple,
- targetPartitionIds, analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert(),
+ targetPartitionIds, enableSingleReplicaLoad,
ConnectContext.get().getSessionVariable().getGroupCommit(), 0);
} else {
sink = new OlapTableSink((OlapTable) targetTable, olapTuple, targetPartitionIds,
- analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert());
+ enableSingleReplicaLoad);
Review Comment:
This is a parameter configured by the user, in this case through session variables.
What do you mean "add this feature to nereids"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2019994755
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2051054128
run cloud_p1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](move-memtable) ignore single replica load when move memtable [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #32845:
URL: https://github.com/apache/doris/pull/32845#issuecomment-2051095237
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org