You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hust-hhb (via GitHub)" <gi...@apache.org> on 2024/09/09 11:40:37 UTC
[PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout on broker load [doris]
hust-hhb opened a new pull request, #40562:
URL: https://github.com/apache/doris/pull/40562
Add retry when calculating delete bitmap timeout on broker load , like stream load doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2343758393
<details>
<summary>TPC-DS: <b>Total hot run time: 192739 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3ccbe272044546a4d5d606a72ca07f4196c3003, data reload: false
query1 914 379 366 366
query2 6453 1959 1920 1920
query3 6642 207 214 207
query4 30391 23110 23463 23110
query5 4188 524 510 510
query6 252 161 157 157
query7 4570 302 290 290
query8 270 221 220 220
query9 8660 2507 2512 2507
query10 456 299 287 287
query11 17945 15254 15231 15231
query12 164 97 96 96
query13 1630 380 356 356
query14 9976 7007 6976 6976
query15 293 168 174 168
query16 8027 438 437 437
query17 1565 561 554 554
query18 2123 283 281 281
query19 230 150 138 138
query20 121 111 110 110
query21 210 103 100 100
query22 4586 4065 4047 4047
query23 34347 33612 33846 33612
query24 10904 2940 2876 2876
query25 655 376 379 376
query26 1164 153 159 153
query27 2394 270 275 270
query28 7340 2039 2014 2014
query29 808 404 405 404
query30 308 162 151 151
query31 1025 777 797 777
query32 95 59 57 57
query33 749 302 290 290
query34 953 462 488 462
query35 885 737 722 722
query36 1135 919 963 919
query37 167 90 87 87
query38 4082 3869 3820 3820
query39 1444 1448 1399 1399
query40 203 120 113 113
query41 48 50 45 45
query42 114 97 96 96
query43 533 475 481 475
query44 1314 800 741 741
query45 196 164 167 164
query46 1108 730 752 730
query47 1877 1790 1825 1790
query48 365 293 292 292
query49 1077 434 481 434
query50 811 410 421 410
query51 7005 6910 6892 6892
query52 99 88 84 84
query53 253 181 182 181
query54 830 444 455 444
query55 75 76 78 76
query56 282 260 252 252
query57 1228 1096 1064 1064
query58 257 237 235 235
query59 3032 2971 2857 2857
query60 300 272 266 266
query61 104 100 103 100
query62 834 662 647 647
query63 234 201 197 197
query64 4184 664 637 637
query65 3281 3157 3164 3157
query66 1088 347 341 341
query67 16101 15307 15521 15307
query68 3124 845 844 844
query69 432 325 325 325
query70 1199 1181 1175 1175
query71 351 340 347 340
query72 6172 3493 3463 3463
query73 584 595 589 589
query74 9006 8921 8877 8877
query75 3151 3328 2953 2953
query76 1812 871 878 871
query77 504 424 411 411
query78 9436 9254 9638 9254
query79 907 895 865 865
query80 899 820 797 797
query81 468 264 261 261
query82 269 268 262 262
query83 195 194 194 194
query84 239 109 107 107
query85 637 403 380 380
query86 348 322 315 315
query87 4333 4355 4314 4314
query88 4261 4027 4010 4010
query89 384 375 378 375
query90 1310 317 319 317
query91 126 124 127 124
query92 77 75 77 75
query93 960 924 900 900
query94 458 399 387 387
query95 425 410 412 410
query96 467 464 464 464
query97 3098 3108 3124 3108
query98 229 228 220 220
query99 1415 1273 1268 1268
Total cold run time: 282230 ms
Total hot run time: 192739 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on code in PR #40562:
URL: https://github.com/apache/doris/pull/40562#discussion_r1759686614
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/BrokerLoadJob.java:
##########
@@ -351,26 +347,48 @@ private void onLoadingTaskFinished(BrokerLoadingTaskAttachment attachment) {
cancelJobWithoutCheck(new FailMsg(FailMsg.CancelType.LOAD_RUN_FAIL, e.getMessage()), true, true);
return;
}
- try {
- LOG.info(new LogBuilder(LogKey.LOAD_JOB, id)
- .add("txn_id", transactionId)
- .add("msg", "Load job try to commit txn")
- .build());
- Env.getCurrentGlobalTransactionMgr().commitTransaction(
- dbId, tableList, transactionId, commitInfos, getLoadJobFinalOperation());
- afterLoadingTaskCommitTransaction(tableList);
- afterCommit();
- } catch (UserException e) {
- LOG.warn(new LogBuilder(LogKey.LOAD_JOB, id)
- .add("database_id", dbId)
- .add("error_msg", "Failed to commit txn with error:" + e.getMessage())
- .build(), e);
- cancelJobWithoutCheck(new FailMsg(FailMsg.CancelType.LOAD_RUN_FAIL, e.getMessage()), true, true);
- } finally {
- if (Config.isCloudMode()) {
- MetaLockUtils.commitUnlockTables(tableList);
- } else {
- MetaLockUtils.writeUnlockTables(tableList);
+ int retryTimes = 0;
+ boolean finishJob = false;
+ while (!finishJob) {
+ try {
+ if (Config.isCloudMode()) {
+ MetaLockUtils.commitLockTables(tableList);
+ } else {
+ MetaLockUtils.writeLockTablesOrMetaException(tableList);
+ }
+ LOG.info(new LogBuilder(LogKey.LOAD_JOB, id)
+ .add("txn_id", transactionId)
+ .add("msg", "Load job try to commit txn")
+ .build());
+ Env.getCurrentGlobalTransactionMgr().commitTransaction(
+ dbId, tableList, transactionId, commitInfos, getLoadJobFinalOperation());
+ afterLoadingTaskCommitTransaction(tableList);
+ afterCommit();
+ finishJob = true;
Review Comment:
use return instead of break, unlock table will do in finally code before return
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2350914612
<details>
<summary>ClickBench: <b>Total hot run time: 31.53 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2261e978c0024b2d9de5e720d0847a2d39e10556, data reload: false
query1 0.05 0.04 0.04
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.67 0.09 0.08
query5 0.52 0.51 0.49
query6 1.13 0.72 0.72
query7 0.02 0.02 0.02
query8 0.06 0.05 0.05
query9 0.56 0.54 0.52
query10 0.56 0.58 0.58
query11 0.16 0.12 0.12
query12 0.16 0.13 0.13
query13 0.62 0.61 0.61
query14 1.46 1.47 1.48
query15 0.90 0.88 0.88
query16 0.36 0.35 0.37
query17 1.06 1.03 1.02
query18 0.16 0.16 0.17
query19 1.94 1.77 1.81
query20 0.01 0.01 0.01
query21 15.39 0.66 0.65
query22 4.30 8.23 1.48
query23 18.22 1.46 1.49
query24 2.26 0.23 0.22
query25 0.19 0.09 0.08
query26 0.30 0.18 0.19
query27 0.09 0.08 0.08
query28 13.18 1.14 1.10
query29 12.63 3.35 3.35
query30 0.24 0.06 0.06
query31 2.86 0.42 0.41
query32 3.23 0.50 0.49
query33 3.08 3.06 3.07
query34 15.45 4.34 4.36
query35 4.42 4.34 4.34
query36 0.68 0.49 0.49
query37 0.21 0.18 0.16
query38 0.17 0.15 0.15
query39 0.05 0.04 0.04
query40 0.18 0.14 0.14
query41 0.10 0.05 0.05
query42 0.06 0.05 0.04
query43 0.05 0.05 0.05
Total cold run time: 109.05 s
Total hot run time: 31.53 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2360190663
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout on broker load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2337999550
<details>
<summary>ClickBench: <b>Total hot run time: 31.24 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86efeaf20e73bb10d90f9e57b39c71eb6616cc83, data reload: false
query1 0.05 0.04 0.04
query2 0.08 0.04 0.04
query3 0.22 0.05 0.05
query4 1.68 0.09 0.08
query5 0.49 0.50 0.48
query6 1.13 0.73 0.73
query7 0.02 0.02 0.02
query8 0.05 0.05 0.05
query9 0.53 0.47 0.48
query10 0.53 0.56 0.53
query11 0.15 0.12 0.12
query12 0.16 0.12 0.11
query13 0.60 0.59 0.59
query14 1.39 1.40 1.41
query15 0.83 0.83 0.83
query16 0.39 0.37 0.36
query17 1.06 1.02 1.01
query18 0.22 0.20 0.21
query19 1.82 1.75 1.82
query20 0.02 0.01 0.01
query21 15.41 0.67 0.67
query22 4.51 7.07 1.79
query23 18.25 1.48 1.26
query24 2.05 0.23 0.22
query25 0.15 0.08 0.08
query26 0.28 0.19 0.17
query27 0.07 0.08 0.07
query28 13.28 1.02 1.00
query29 12.64 3.37 3.30
query30 0.24 0.05 0.05
query31 2.86 0.42 0.40
query32 3.24 0.48 0.49
query33 2.98 3.01 2.99
query34 17.08 4.40 4.40
query35 4.40 4.46 4.42
query36 0.66 0.48 0.48
query37 0.18 0.16 0.15
query38 0.15 0.14 0.14
query39 0.04 0.04 0.04
query40 0.15 0.13 0.13
query41 0.09 0.04 0.05
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 110.23 s
Total hot run time: 31.24 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2350928599
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2359913612
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2343725660
<details>
<summary>TPC-H: <b>Total hot run time: 38366 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3ccbe272044546a4d5d606a72ca07f4196c3003, data reload: false
------ Round 1 ----------------------------------
q1 17667 4462 4381 4381
q2 2012 190 199 190
q3 10441 1154 1190 1154
q4 10128 796 645 645
q5 7746 2888 2815 2815
q6 231 140 140 140
q7 972 618 614 614
q8 9322 2087 2084 2084
q9 7259 6567 6556 6556
q10 7060 2206 2123 2123
q11 473 240 243 240
q12 397 223 230 223
q13 18826 3111 3140 3111
q14 286 234 242 234
q15 539 487 480 480
q16 540 434 436 434
q17 991 729 733 729
q18 7476 6874 6990 6874
q19 1412 1133 1070 1070
q20 701 335 333 333
q21 4575 3008 2935 2935
q22 1101 1001 1056 1001
Total cold run time: 110155 ms
Total hot run time: 38366 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4487 4333 4349 4333
q2 420 286 279 279
q3 2868 2676 2666 2666
q4 1910 1671 1698 1671
q5 5467 5391 5463 5391
q6 223 133 132 132
q7 2091 1719 1729 1719
q8 3219 3374 3390 3374
q9 8464 8475 8462 8462
q10 3475 3204 3232 3204
q11 593 527 508 508
q12 782 616 606 606
q13 15890 3159 3109 3109
q14 300 272 272 272
q15 525 478 493 478
q16 516 487 476 476
q17 1814 1512 1474 1474
q18 7707 7490 7568 7490
q19 1693 1515 1546 1515
q20 2088 1816 1829 1816
q21 5605 5406 5308 5308
q22 1119 1046 1043 1043
Total cold run time: 71256 ms
Total hot run time: 55326 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2342531416
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2350900010
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2343772504
<details>
<summary>ClickBench: <b>Total hot run time: 31.38 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d3ccbe272044546a4d5d606a72ca07f4196c3003, data reload: false
query1 0.05 0.04 0.04
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.68 0.09 0.08
query5 0.50 0.49 0.49
query6 1.12 0.75 0.73
query7 0.02 0.02 0.02
query8 0.05 0.05 0.04
query9 0.54 0.49 0.48
query10 0.56 0.57 0.54
query11 0.15 0.12 0.12
query12 0.15 0.13 0.12
query13 0.60 0.59 0.60
query14 1.38 1.41 1.39
query15 0.85 0.82 0.82
query16 0.38 0.36 0.38
query17 1.04 0.98 1.03
query18 0.17 0.17 0.19
query19 1.86 1.91 1.74
query20 0.02 0.01 0.01
query21 15.39 0.66 0.65
query22 4.33 6.66 1.87
query23 18.28 1.45 1.34
query24 2.12 0.24 0.22
query25 0.16 0.08 0.08
query26 0.27 0.18 0.18
query27 0.08 0.08 0.08
query28 13.21 1.03 1.01
query29 12.58 3.40 3.43
query30 0.24 0.05 0.05
query31 2.88 0.41 0.38
query32 3.27 0.49 0.48
query33 2.98 3.01 3.03
query34 16.88 4.42 4.37
query35 4.46 4.45 4.41
query36 0.65 0.47 0.51
query37 0.18 0.15 0.16
query38 0.15 0.16 0.14
query39 0.05 0.04 0.03
query40 0.16 0.12 0.12
query41 0.09 0.05 0.04
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.94 s
Total hot run time: 31.38 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #40562:
URL: https://github.com/apache/doris/pull/40562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #40562:
URL: https://github.com/apache/doris/pull/40562#discussion_r1766111349
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java:
##########
@@ -656,21 +658,50 @@ public void updateLoadingStatus() throws UserException {
}
private void tryCommitJob() throws UserException {
- LOG.info(new LogBuilder(LogKey.LOAD_JOB, id).add("txn_id", transactionId)
- .add("msg", "Load job try to commit txn").build());
- Database db = getDb();
- List<Table> tableList = db.getTablesOnIdOrderOrThrowException(
- Lists.newArrayList(tableToLoadPartitions.keySet()));
- MetaLockUtils.writeLockTablesOrMetaException(tableList);
- try {
- Env.getCurrentGlobalTransactionMgr().commitTransaction(
- dbId, tableList, transactionId, commitInfos,
- new LoadJobFinalOperation(id, loadingStatus, progress, loadStartTimestamp,
- finishTimestamp, state, failMsg));
- } catch (TabletQuorumFailedException e) {
- // retry in next loop
- } finally {
- MetaLockUtils.writeUnlockTables(tableList);
+ int retryTimes = 0;
+ while (true) {
+ Database db = getDb();
+ List<Table> tableList = db.getTablesOnIdOrderOrThrowException(
+ Lists.newArrayList(tableToLoadPartitions.keySet()));
+ if (Config.isCloudMode()) {
+ MetaLockUtils.commitLockTables(tableList);
+ } else {
+ MetaLockUtils.writeLockTablesOrMetaException(tableList);
+ }
+ try {
+ LOG.info(new LogBuilder(LogKey.LOAD_JOB, id).add("txn_id", transactionId)
+ .add("msg", "Load job try to commit txn").build());
+ Env.getCurrentGlobalTransactionMgr().commitTransaction(
+ dbId, tableList, transactionId, commitInfos,
+ new LoadJobFinalOperation(id, loadingStatus, progress, loadStartTimestamp,
+ finishTimestamp, state, failMsg));
+ return;
+ } catch (UserException e) {
+ LOG.warn(new LogBuilder(LogKey.LOAD_JOB, id)
+ .add("txn_id", transactionId)
+ .add("database_id", dbId)
+ .add("retry_times", retryTimes)
+ .add("error_msg", "Failed to commit txn with error:" + e.getMessage())
+ .build(), e);
+ if (e.getErrorCode() == InternalErrorCode.DELETE_BITMAP_LOCK_ERR) {
+ retryTimes++;
+ if (retryTimes >= Config.mow_calculate_delete_bitmap_retry_times) {
+ LOG.warn("cancelJob {} because up to max retry time", id);
Review Comment:
log with the exception error message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2350913132
<details>
<summary>TPC-DS: <b>Total hot run time: 195605 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2261e978c0024b2d9de5e720d0847a2d39e10556, data reload: false
query1 933 384 394 384
query2 6465 1799 1765 1765
query3 6659 214 221 214
query4 26100 24172 24164 24164
query5 5560 560 534 534
query6 269 170 163 163
query7 4582 315 324 315
query8 286 224 238 224
query9 8484 2610 2617 2610
query10 488 287 274 274
query11 15936 15432 15523 15432
query12 156 102 107 102
query13 1702 408 388 388
query14 11410 6833 7187 6833
query15 214 178 176 176
query16 7617 509 527 509
query17 1545 603 575 575
query18 2071 300 296 296
query19 195 149 152 149
query20 125 114 117 114
query21 217 109 108 108
query22 4550 4336 4295 4295
query23 34608 34013 33695 33695
query24 9585 3094 3144 3094
query25 678 418 418 418
query26 1171 163 163 163
query27 2355 302 292 292
query28 6557 2108 2082 2082
query29 952 436 434 434
query30 293 164 159 159
query31 995 787 806 787
query32 107 59 65 59
query33 749 309 307 307
query34 923 483 486 483
query35 904 756 749 749
query36 1082 910 944 910
query37 169 86 83 83
query38 4103 3968 3906 3906
query39 1453 1420 1431 1420
query40 219 123 122 122
query41 51 51 50 50
query42 125 97 99 97
query43 497 451 441 441
query44 1243 811 781 781
query45 202 178 174 174
query46 1115 847 809 809
query47 1911 1775 1838 1775
query48 384 301 297 297
query49 1109 487 460 460
query50 911 453 442 442
query51 6955 6912 6916 6912
query52 109 94 92 92
query53 264 187 190 187
query54 835 477 484 477
query55 84 88 81 81
query56 298 288 292 288
query57 1229 1117 1079 1079
query58 271 245 255 245
query59 2967 2659 2472 2472
query60 313 285 296 285
query61 125 238 103 103
query62 910 679 672 672
query63 224 187 211 187
query64 4600 707 683 683
query65 3298 3160 3147 3147
query66 1443 304 306 304
query67 16182 15637 15427 15427
query68 3333 873 864 864
query69 439 327 328 327
query70 1166 1144 1164 1144
query71 347 344 341 341
query72 6139 3440 3365 3365
query73 589 584 580 580
query74 9429 9179 9207 9179
query75 3168 3065 2985 2985
query76 1926 843 845 843
query77 431 405 397 397
query78 9439 9331 9291 9291
query79 917 890 871 871
query80 914 830 810 810
query81 456 269 274 269
query82 267 264 260 260
query83 193 189 190 189
query84 243 109 105 105
query85 649 423 404 404
query86 320 311 330 311
query87 4378 4515 4385 4385
query88 4185 4096 4100 4096
query89 385 377 368 368
query90 1262 323 317 317
query91 125 127 125 125
query92 79 76 75 75
query93 1053 1079 1047 1047
query94 610 403 362 362
query95 443 423 422 422
query96 475 474 476 474
query97 3137 3131 3159 3131
query98 230 230 225 225
query99 1553 1294 1289 1289
Total cold run time: 278402 ms
Total hot run time: 195605 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2342575129
<details>
<summary>ClickBench: <b>Total hot run time: 31.82 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2fa6753d8d043c75d7366fcd4b4a31f16edb700, data reload: false
query1 0.05 0.05 0.04
query2 0.07 0.04 0.04
query3 0.23 0.05 0.04
query4 1.67 0.10 0.09
query5 0.50 0.48 0.49
query6 1.14 0.74 0.73
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.54 0.49 0.50
query10 0.54 0.57 0.53
query11 0.17 0.12 0.12
query12 0.16 0.13 0.12
query13 0.61 0.60 0.59
query14 1.41 1.41 1.40
query15 0.84 0.82 0.82
query16 0.35 0.38 0.37
query17 1.04 1.06 1.00
query18 0.21 0.20 0.20
query19 1.94 1.78 1.85
query20 0.01 0.01 0.02
query21 15.66 0.67 0.66
query22 4.55 6.53 2.24
query23 18.24 1.32 1.41
query24 2.07 0.23 0.23
query25 0.14 0.09 0.08
query26 0.26 0.17 0.17
query27 0.07 0.07 0.08
query28 13.25 1.02 1.00
query29 12.65 3.38 3.32
query30 0.25 0.06 0.07
query31 2.84 0.42 0.40
query32 3.24 0.47 0.50
query33 2.97 2.98 3.08
query34 17.12 4.39 4.40
query35 4.51 4.49 4.46
query36 0.65 0.50 0.46
query37 0.19 0.17 0.16
query38 0.17 0.14 0.14
query39 0.05 0.04 0.04
query40 0.16 0.13 0.14
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 110.8 s
Total hot run time: 31.82 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout on broker load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2337882807
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout on broker load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2337988218
<details>
<summary>TPC-DS: <b>Total hot run time: 192142 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86efeaf20e73bb10d90f9e57b39c71eb6616cc83, data reload: false
query1 1255 880 876 876
query2 6317 1947 1898 1898
query3 10600 3982 3806 3806
query4 59274 26722 23192 23192
query5 5356 531 540 531
query6 412 166 164 164
query7 5821 289 294 289
query8 324 226 226 226
query9 8878 2498 2503 2498
query10 491 295 274 274
query11 17999 15045 15548 15045
query12 158 109 113 109
query13 1570 427 400 400
query14 11376 6846 7248 6846
query15 248 179 173 173
query16 7590 452 459 452
query17 1130 574 581 574
query18 2078 317 310 310
query19 298 153 149 149
query20 117 123 120 120
query21 208 109 104 104
query22 4570 4378 4464 4378
query23 34513 33558 33533 33533
query24 5956 2836 2840 2836
query25 523 389 397 389
query26 685 154 155 154
query27 1785 274 276 274
query28 3738 2024 1999 1999
query29 661 413 405 405
query30 269 150 152 150
query31 914 759 788 759
query32 81 56 55 55
query33 500 300 292 292
query34 890 496 466 466
query35 853 720 717 717
query36 1076 935 950 935
query37 144 79 88 79
query38 3982 3790 3914 3790
query39 1440 1420 1375 1375
query40 198 118 114 114
query41 48 44 45 44
query42 119 100 96 96
query43 530 486 484 484
query44 1122 736 743 736
query45 198 174 171 171
query46 1087 752 765 752
query47 1875 1756 1809 1756
query48 380 296 310 296
query49 785 474 445 445
query50 827 413 436 413
query51 6963 6966 6885 6885
query52 99 87 85 85
query53 257 185 183 183
query54 563 447 458 447
query55 76 73 72 72
query56 268 253 266 253
query57 1193 1029 1087 1029
query58 216 224 264 224
query59 3234 3120 2899 2899
query60 290 270 276 270
query61 99 97 104 97
query62 784 665 646 646
query63 228 192 189 189
query64 2802 680 688 680
query65 3216 3125 3149 3125
query66 676 334 330 330
query67 15451 15274 15159 15159
query68 2960 572 587 572
query69 385 287 284 284
query70 1202 1131 1118 1118
query71 357 295 281 281
query72 5280 3976 4052 3976
query73 773 318 318 318
query74 9115 8872 8782 8782
query75 3333 2723 2691 2691
query76 1492 1004 978 978
query77 557 343 334 334
query78 9812 9058 9058 9058
query79 1041 536 546 536
query80 693 515 510 510
query81 474 232 232 232
query82 240 143 139 139
query83 171 152 150 150
query84 247 76 76 76
query85 681 299 290 290
query86 313 313 291 291
query87 4428 4370 4241 4241
query88 2923 2279 2274 2274
query89 379 293 290 290
query90 1980 195 190 190
query91 125 100 160 100
query92 62 48 49 48
query93 1071 538 537 537
query94 738 292 297 292
query95 322 251 257 251
query96 585 261 262 261
query97 3184 3054 3095 3054
query98 223 210 206 206
query99 1511 1311 1280 1280
Total cold run time: 304321 ms
Total hot run time: 192142 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2359950382
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout on broker load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2337960715
<details>
<summary>TPC-H: <b>Total hot run time: 38123 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 86efeaf20e73bb10d90f9e57b39c71eb6616cc83, data reload: false
------ Round 1 ----------------------------------
q1 17615 4411 4310 4310
q2 2016 195 187 187
q3 11727 1008 1115 1008
q4 10510 698 721 698
q5 7718 2830 2816 2816
q6 225 140 137 137
q7 945 622 604 604
q8 9338 2065 2056 2056
q9 7125 6492 6499 6492
q10 6987 2200 2237 2200
q11 467 248 252 248
q12 391 232 234 232
q13 17766 3071 3083 3071
q14 286 241 247 241
q15 550 489 505 489
q16 538 447 431 431
q17 987 666 722 666
q18 7259 6827 6905 6827
q19 1397 1142 1106 1106
q20 677 331 333 331
q21 4192 3121 2972 2972
q22 1110 1001 1013 1001
Total cold run time: 109826 ms
Total hot run time: 38123 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4585 4302 4335 4302
q2 386 279 275 275
q3 2852 2637 2656 2637
q4 1978 1784 1705 1705
q5 5660 5688 5748 5688
q6 223 142 142 142
q7 2228 1861 1840 1840
q8 3302 3414 3463 3414
q9 8856 8892 8823 8823
q10 3572 3382 3357 3357
q11 625 505 534 505
q12 840 656 674 656
q13 13846 3272 3304 3272
q14 321 279 293 279
q15 557 506 496 496
q16 562 495 508 495
q17 1836 1522 1549 1522
q18 8125 7875 7901 7875
q19 1761 1644 1610 1610
q20 2153 1894 1906 1894
q21 5790 5529 5564 5529
q22 1129 1050 1054 1050
Total cold run time: 71187 ms
Total hot run time: 57366 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2342560976
<details>
<summary>TPC-H: <b>Total hot run time: 38292 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2fa6753d8d043c75d7366fcd4b4a31f16edb700, data reload: false
------ Round 1 ----------------------------------
q1 17602 4396 4373 4373
q2 2017 185 188 185
q3 11699 977 1142 977
q4 10512 746 811 746
q5 7719 2868 2859 2859
q6 229 139 138 138
q7 949 643 606 606
q8 9297 2074 2102 2074
q9 7270 6581 6541 6541
q10 6989 2245 2228 2228
q11 468 248 250 248
q12 400 229 233 229
q13 17756 3130 3058 3058
q14 275 240 237 237
q15 527 481 485 481
q16 523 430 428 428
q17 981 664 689 664
q18 7356 6975 6821 6821
q19 1383 1060 1094 1060
q20 710 330 340 330
q21 4045 3067 2970 2970
q22 1134 1048 1039 1039
Total cold run time: 109841 ms
Total hot run time: 38292 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4418 4317 4300 4300
q2 390 283 280 280
q3 2878 2677 2654 2654
q4 1967 1613 1631 1613
q5 5668 5700 5742 5700
q6 232 143 140 140
q7 2260 1882 1792 1792
q8 3320 3430 3458 3430
q9 8901 8927 8890 8890
q10 3609 3431 3397 3397
q11 602 533 512 512
q12 827 661 648 648
q13 13944 3223 3323 3223
q14 325 286 287 286
q15 548 486 481 481
q16 536 487 504 487
q17 1847 1573 1579 1573
q18 8077 7742 7778 7742
q19 1749 1600 1508 1508
q20 2166 1901 1957 1901
q21 5735 5514 5526 5514
q22 1165 1068 1034 1034
Total cold run time: 71164 ms
Total hot run time: 57105 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2350928603
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on code in PR #40562:
URL: https://github.com/apache/doris/pull/40562#discussion_r1766138305
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java:
##########
@@ -656,21 +658,50 @@ public void updateLoadingStatus() throws UserException {
}
private void tryCommitJob() throws UserException {
- LOG.info(new LogBuilder(LogKey.LOAD_JOB, id).add("txn_id", transactionId)
- .add("msg", "Load job try to commit txn").build());
- Database db = getDb();
- List<Table> tableList = db.getTablesOnIdOrderOrThrowException(
- Lists.newArrayList(tableToLoadPartitions.keySet()));
- MetaLockUtils.writeLockTablesOrMetaException(tableList);
- try {
- Env.getCurrentGlobalTransactionMgr().commitTransaction(
- dbId, tableList, transactionId, commitInfos,
- new LoadJobFinalOperation(id, loadingStatus, progress, loadStartTimestamp,
- finishTimestamp, state, failMsg));
- } catch (TabletQuorumFailedException e) {
- // retry in next loop
- } finally {
- MetaLockUtils.writeUnlockTables(tableList);
+ int retryTimes = 0;
+ while (true) {
+ Database db = getDb();
+ List<Table> tableList = db.getTablesOnIdOrderOrThrowException(
+ Lists.newArrayList(tableToLoadPartitions.keySet()));
+ if (Config.isCloudMode()) {
+ MetaLockUtils.commitLockTables(tableList);
+ } else {
+ MetaLockUtils.writeLockTablesOrMetaException(tableList);
+ }
+ try {
+ LOG.info(new LogBuilder(LogKey.LOAD_JOB, id).add("txn_id", transactionId)
+ .add("msg", "Load job try to commit txn").build());
+ Env.getCurrentGlobalTransactionMgr().commitTransaction(
+ dbId, tableList, transactionId, commitInfos,
+ new LoadJobFinalOperation(id, loadingStatus, progress, loadStartTimestamp,
+ finishTimestamp, state, failMsg));
+ return;
+ } catch (UserException e) {
+ LOG.warn(new LogBuilder(LogKey.LOAD_JOB, id)
+ .add("txn_id", transactionId)
+ .add("database_id", dbId)
+ .add("retry_times", retryTimes)
+ .add("error_msg", "Failed to commit txn with error:" + e.getMessage())
+ .build(), e);
+ if (e.getErrorCode() == InternalErrorCode.DELETE_BITMAP_LOCK_ERR) {
+ retryTimes++;
+ if (retryTimes >= Config.mow_calculate_delete_bitmap_retry_times) {
+ LOG.warn("cancelJob {} because up to max retry time", id);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout on broker load [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2337882957
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2350889075
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2350909423
<details>
<summary>TPC-H: <b>Total hot run time: 43493 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2261e978c0024b2d9de5e720d0847a2d39e10556, data reload: false
------ Round 1 ----------------------------------
q1 17581 7388 7237 7237
q2 2050 198 187 187
q3 10521 1418 1430 1418
q4 10317 993 1071 993
q5 7744 3237 3184 3184
q6 245 155 161 155
q7 1056 645 636 636
q8 9475 2046 2086 2046
q9 6825 6365 6369 6365
q10 7043 2596 2564 2564
q11 439 256 263 256
q12 405 234 236 234
q13 17780 3029 3027 3027
q14 292 250 264 250
q15 593 545 524 524
q16 526 436 429 429
q17 1023 991 975 975
q18 7446 6830 7049 6830
q19 1381 1275 1258 1258
q20 606 350 332 332
q21 3961 3638 3598 3598
q22 1105 1040 995 995
Total cold run time: 108414 ms
Total hot run time: 43493 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7231 7190 7214 7190
q2 356 243 230 230
q3 2905 2954 2970 2954
q4 1967 1990 1993 1990
q5 5493 5593 5454 5454
q6 238 148 146 146
q7 2047 1717 1656 1656
q8 3272 3359 3353 3353
q9 8513 8447 8548 8447
q10 3437 3503 3492 3492
q11 581 459 468 459
q12 798 602 586 586
q13 8635 3077 3075 3075
q14 294 268 286 268
q15 586 522 524 522
q16 484 458 460 458
q17 1771 1737 1748 1737
q18 7996 7616 7504 7504
q19 1741 1738 1725 1725
q20 2054 1807 1810 1807
q21 5821 5441 5355 5355
q22 1111 1003 973 973
Total cold run time: 67331 ms
Total hot run time: 59381 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2343679167
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2343767037
TeamCity be ut coverage result:
Function Coverage: 36.88% (9452/25626)
Line Coverage: 28.24% (77734/275287)
Region Coverage: 27.63% (40119/145186)
Branch Coverage: 24.25% (20386/84076)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d3ccbe272044546a4d5d606a72ca07f4196c3003_d3ccbe272044546a4d5d606a72ca07f4196c3003/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2342570504
<details>
<summary>TPC-DS: <b>Total hot run time: 197246 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2fa6753d8d043c75d7366fcd4b4a31f16edb700, data reload: false
query1 1267 884 928 884
query2 6396 1957 1891 1891
query3 10604 4179 4011 4011
query4 60095 25065 23209 23209
query5 5085 516 524 516
query6 424 172 166 166
query7 5617 306 297 297
query8 308 220 220 220
query9 7751 2535 2525 2525
query10 437 311 263 263
query11 17302 15308 15442 15308
query12 159 107 102 102
query13 1456 409 391 391
query14 10794 7645 7370 7370
query15 213 185 184 184
query16 6698 483 484 483
query17 1088 583 563 563
query18 1448 295 291 291
query19 204 145 150 145
query20 120 112 107 107
query21 220 107 107 107
query22 4742 4361 4414 4361
query23 34439 33839 33518 33518
query24 5995 2928 2996 2928
query25 511 412 422 412
query26 601 163 156 156
query27 1567 280 285 280
query28 3745 2053 2035 2035
query29 672 440 434 434
query30 239 160 154 154
query31 976 757 740 740
query32 74 53 53 53
query33 414 291 288 288
query34 869 490 474 474
query35 835 725 702 702
query36 1070 945 943 943
query37 146 87 86 86
query38 4051 3923 3921 3921
query39 1422 1390 1404 1390
query40 191 114 116 114
query41 48 47 45 45
query42 120 96 93 93
query43 517 461 462 461
query44 1111 764 747 747
query45 195 165 166 165
query46 1087 784 771 771
query47 1906 1796 1795 1795
query48 373 306 299 299
query49 771 436 467 436
query50 835 430 431 430
query51 6910 6967 6864 6864
query52 99 88 88 88
query53 260 178 181 178
query54 571 471 468 468
query55 76 76 75 75
query56 271 263 260 260
query57 1205 1082 1044 1044
query58 220 270 232 232
query59 3030 2748 2723 2723
query60 333 274 280 274
query61 103 97 102 97
query62 767 662 653 653
query63 218 189 187 187
query64 1350 687 691 687
query65 3213 3131 3214 3131
query66 654 337 341 337
query67 15661 15403 15300 15300
query68 1686 848 842 842
query69 419 330 383 330
query70 1223 1100 1173 1100
query71 350 356 344 344
query72 4597 3596 2468 2468
query73 598 587 585 585
query74 9018 8997 8954 8954
query75 3080 2998 3019 2998
query76 952 885 867 867
query77 449 425 417 417
query78 9302 9389 9217 9217
query79 904 886 868 868
query80 822 845 822 822
query81 455 258 262 258
query82 264 266 266 266
query83 196 192 193 192
query84 201 108 109 108
query85 587 410 397 397
query86 304 315 303 303
query87 4293 4314 4273 4273
query88 4333 4133 4144 4133
query89 374 369 361 361
query90 796 321 317 317
query91 127 127 132 127
query92 79 79 82 79
query93 924 919 916 916
query94 412 362 380 362
query95 438 427 426 426
query96 474 474 474 474
query97 3102 3080 3104 3080
query98 246 237 233 233
query99 1316 1273 1267 1267
Total cold run time: 294470 ms
Total hot run time: 197246 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #40562:
URL: https://github.com/apache/doris/pull/40562#discussion_r1759660704
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/BrokerLoadJob.java:
##########
@@ -351,26 +347,48 @@ private void onLoadingTaskFinished(BrokerLoadingTaskAttachment attachment) {
cancelJobWithoutCheck(new FailMsg(FailMsg.CancelType.LOAD_RUN_FAIL, e.getMessage()), true, true);
return;
}
- try {
- LOG.info(new LogBuilder(LogKey.LOAD_JOB, id)
- .add("txn_id", transactionId)
- .add("msg", "Load job try to commit txn")
- .build());
- Env.getCurrentGlobalTransactionMgr().commitTransaction(
- dbId, tableList, transactionId, commitInfos, getLoadJobFinalOperation());
- afterLoadingTaskCommitTransaction(tableList);
- afterCommit();
- } catch (UserException e) {
- LOG.warn(new LogBuilder(LogKey.LOAD_JOB, id)
- .add("database_id", dbId)
- .add("error_msg", "Failed to commit txn with error:" + e.getMessage())
- .build(), e);
- cancelJobWithoutCheck(new FailMsg(FailMsg.CancelType.LOAD_RUN_FAIL, e.getMessage()), true, true);
- } finally {
- if (Config.isCloudMode()) {
- MetaLockUtils.commitUnlockTables(tableList);
- } else {
- MetaLockUtils.writeUnlockTables(tableList);
+ int retryTimes = 0;
+ boolean finishJob = false;
+ while (!finishJob) {
+ try {
+ if (Config.isCloudMode()) {
+ MetaLockUtils.commitLockTables(tableList);
+ } else {
+ MetaLockUtils.writeLockTablesOrMetaException(tableList);
+ }
+ LOG.info(new LogBuilder(LogKey.LOAD_JOB, id)
+ .add("txn_id", transactionId)
+ .add("msg", "Load job try to commit txn")
+ .build());
+ Env.getCurrentGlobalTransactionMgr().commitTransaction(
+ dbId, tableList, transactionId, commitInfos, getLoadJobFinalOperation());
+ afterLoadingTaskCommitTransaction(tableList);
+ afterCommit();
+ finishJob = true;
Review Comment:
just break instead of using finishJob.
```suggestion
finishJob = true;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2359994658
TeamCity be ut coverage result:
Function Coverage: 37.33% (9585/25676)
Line Coverage: 28.71% (79212/275885)
Region Coverage: 28.19% (41029/145522)
Branch Coverage: 24.82% (20902/84222)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0b1426545ef083873de900caf90b98e2ee785ac3_0b1426545ef083873de900caf90b98e2ee785ac3/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2359978900
<details>
<summary>TPC-H: <b>Total hot run time: 41265 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0b1426545ef083873de900caf90b98e2ee785ac3, data reload: false
------ Round 1 ----------------------------------
q1 17577 8116 7298 7298
q2 2052 162 160 160
q3 10583 1055 1154 1055
q4 10538 737 743 737
q5 7760 3069 3089 3069
q6 237 155 152 152
q7 1015 618 607 607
q8 9451 2053 2053 2053
q9 6816 6402 6449 6402
q10 7032 2282 2308 2282
q11 445 263 254 254
q12 405 215 225 215
q13 17784 3000 2982 2982
q14 242 218 226 218
q15 575 537 527 527
q16 702 615 618 615
q17 983 772 789 772
q18 7230 6583 6624 6583
q19 1398 929 1064 929
q20 569 296 287 287
q21 4079 3275 3069 3069
q22 1111 1009 999 999
Total cold run time: 108584 ms
Total hot run time: 41265 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7276 7298 7312 7298
q2 329 236 230 230
q3 3020 2954 2966 2954
q4 2032 1849 1843 1843
q5 5569 5565 5622 5565
q6 244 155 151 151
q7 2234 1834 1819 1819
q8 3328 3448 3445 3445
q9 8735 9020 8778 8778
q10 3504 3457 3507 3457
q11 591 477 490 477
q12 816 639 629 629
q13 9876 3182 3149 3149
q14 293 279 284 279
q15 559 522 552 522
q16 739 683 671 671
q17 1819 1632 1573 1573
q18 8198 7798 7809 7798
q19 1729 1542 1584 1542
q20 2184 1887 1885 1885
q21 5726 5321 5466 5321
q22 1145 1088 1074 1074
Total cold run time: 69946 ms
Total hot run time: 60460 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40562:
URL: https://github.com/apache/doris/pull/40562#issuecomment-2350917935
TeamCity be ut coverage result:
Function Coverage: 36.91% (9473/25663)
Line Coverage: 28.28% (77907/275491)
Region Coverage: 27.68% (40231/145323)
Branch Coverage: 24.29% (20442/84154)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2261e978c0024b2d9de5e720d0847a2d39e10556_2261e978c0024b2d9de5e720d0847a2d39e10556/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org