You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hust-hhb (via GitHub)" <gi...@apache.org> on 2024/09/03 14:30:05 UTC
[PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
hust-hhb opened a new pull request, #40340:
URL: https://github.com/apache/doris/pull/40340
mow table may get table lock timeout, add some log to print the queue length of committing txn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2326756552
<details>
<summary>TPC-H: <b>Total hot run time: 38158 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0164e777ca8b0136c9f51cfb0f02de5fbf8cdf77, data reload: false
------ Round 1 ----------------------------------
q1 18041 4762 4582 4582
q2 2737 184 178 178
q3 11483 1160 1126 1126
q4 11154 801 726 726
q5 8293 2910 2842 2842
q6 229 142 141 141
q7 956 624 604 604
q8 9344 2040 2034 2034
q9 7178 6574 6502 6502
q10 6990 2136 2215 2136
q11 482 240 241 240
q12 396 231 225 225
q13 17772 3081 3101 3081
q14 284 237 235 235
q15 525 494 490 490
q16 602 500 519 500
q17 974 599 793 599
q18 7564 6847 6765 6765
q19 1400 1028 1137 1028
q20 677 334 336 334
q21 3882 3052 2793 2793
q22 1066 1007 997 997
Total cold run time: 112029 ms
Total hot run time: 38158 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4401 4307 4271 4271
q2 386 276 267 267
q3 2871 2650 2642 2642
q4 1914 1669 1593 1593
q5 5392 5399 5399 5399
q6 226 132 135 132
q7 2100 1736 1720 1720
q8 3184 3349 3381 3349
q9 8466 8488 8432 8432
q10 3469 3243 3238 3238
q11 608 513 502 502
q12 809 618 619 618
q13 10454 3138 3077 3077
q14 305 274 282 274
q15 519 480 482 480
q16 596 546 573 546
q17 1781 1505 1473 1473
q18 7875 7546 7577 7546
q19 1674 1509 1526 1509
q20 2023 1836 1812 1812
q21 5544 5259 5283 5259
q22 1117 1024 1045 1024
Total cold run time: 65714 ms
Total hot run time: 55163 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2367113902
run fe_ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #40340:
URL: https://github.com/apache/doris/pull/40340#discussion_r1759658824
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java:
##########
@@ -959,7 +961,19 @@ public boolean commitAndPublishTransaction(DatabaseIf db, long transactionId,
public boolean commitAndPublishTransaction(DatabaseIf db, List<Table> tableList, long transactionId,
List<TabletCommitInfo> tabletCommitInfos, long timeoutMillis,
TxnCommitAttachment txnCommitAttachment) throws UserException {
+ for (int i = 0; i < tableList.size(); i++) {
+ long tableId = tableList.get(i).getId();
+ LOG.info("start commit txn=" + transactionId + ",table=" + tableId);
+ }
+ for (Map.Entry<Long, AtomicInteger> entry : waitToCommitTxnCountMap.entrySet()) {
+ if (entry.getValue().get() > 5) {
+ LOG.info("now table {} commitAndPublishTransaction queue is {}", entry.getKey(),
+ entry.getValue().get());
+ }
+ }
+ increaseCount(tableList);
Review Comment:
increase_waiting_lock_count()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #40340:
URL: https://github.com/apache/doris/pull/40340#discussion_r1759661112
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java:
##########
@@ -959,7 +961,19 @@ public boolean commitAndPublishTransaction(DatabaseIf db, long transactionId,
public boolean commitAndPublishTransaction(DatabaseIf db, List<Table> tableList, long transactionId,
List<TabletCommitInfo> tabletCommitInfos, long timeoutMillis,
TxnCommitAttachment txnCommitAttachment) throws UserException {
+ for (int i = 0; i < tableList.size(); i++) {
+ long tableId = tableList.get(i).getId();
+ LOG.info("start commit txn=" + transactionId + ",table=" + tableId);
+ }
+ for (Map.Entry<Long, AtomicInteger> entry : waitToCommitTxnCountMap.entrySet()) {
+ if (entry.getValue().get() > 5) {
+ LOG.info("now table {} commitAndPublishTransaction queue is {}", entry.getKey(),
+ entry.getValue().get());
+ }
+ }
+ increaseCount(tableList);
Review Comment:
increaseWaitingLockCount
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2350895794
<details>
<summary>TPC-DS: <b>Total hot run time: 199230 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4a92ac912885fe39fc2b287c2ff5dd79f3cc6b33, data reload: false
query1 1241 887 893 887
query2 6366 1795 1782 1782
query3 10675 3927 3823 3823
query4 55155 25394 23943 23943
query5 5055 542 518 518
query6 343 175 176 175
query7 5649 308 303 303
query8 288 233 236 233
query9 6098 2581 2574 2574
query10 401 300 282 282
query11 16107 15644 15509 15509
query12 157 105 99 99
query13 1446 401 389 389
query14 10342 6904 6585 6585
query15 211 176 184 176
query16 6971 471 477 471
query17 1157 607 577 577
query18 1866 292 298 292
query19 200 152 149 149
query20 116 110 115 110
query21 204 108 104 104
query22 4592 4514 4466 4466
query23 34867 33941 33617 33617
query24 6173 3111 3137 3111
query25 521 393 393 393
query26 603 153 164 153
query27 1592 279 286 279
query28 2838 2072 2063 2063
query29 660 426 428 426
query30 222 154 169 154
query31 941 791 781 781
query32 77 54 57 54
query33 459 286 296 286
query34 895 483 474 474
query35 835 713 735 713
query36 1025 869 909 869
query37 144 87 85 85
query38 4019 3955 3996 3955
query39 1476 1425 1404 1404
query40 203 113 113 113
query41 48 47 46 46
query42 127 97 104 97
query43 501 462 455 455
query44 1208 784 771 771
query45 204 171 168 168
query46 1089 841 806 806
query47 1876 1761 1802 1761
query48 385 300 297 297
query49 732 448 434 434
query50 928 443 466 443
query51 6943 6896 6843 6843
query52 99 88 88 88
query53 257 181 184 181
query54 564 491 469 469
query55 79 74 78 74
query56 297 267 269 267
query57 1179 1053 1061 1053
query58 227 243 240 240
query59 2739 2610 2521 2521
query60 299 274 279 274
query61 106 102 103 102
query62 793 670 669 669
query63 220 191 192 191
query64 1607 699 675 675
query65 3233 3150 3225 3150
query66 697 310 317 310
query67 15902 15658 15462 15462
query68 2014 887 884 884
query69 443 346 340 340
query70 1167 1161 1147 1147
query71 366 361 359 359
query72 4607 3428 3367 3367
query73 596 582 586 582
query74 9207 9069 9263 9069
query75 3092 3011 3024 3011
query76 969 878 884 878
query77 497 421 414 414
query78 9374 9255 9311 9255
query79 941 890 895 890
query80 843 858 844 844
query81 449 270 268 268
query82 268 267 269 267
query83 191 191 194 191
query84 205 107 109 107
query85 640 417 405 405
query86 310 318 309 309
query87 4387 4441 4333 4333
query88 4250 4173 4251 4173
query89 382 379 370 370
query90 1305 320 314 314
query91 129 126 130 126
query92 79 78 81 78
query93 1078 1067 1062 1062
query94 528 394 408 394
query95 464 424 434 424
query96 481 480 478 478
query97 3134 3133 3151 3133
query98 229 233 235 233
query99 1346 1309 1301 1301
Total cold run time: 288500 ms
Total hot run time: 199230 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2326678564
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2326679032
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2342884402
<details>
<summary>ClickBench: <b>Total hot run time: 30.93 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0cbe9fd363773e1611b3356acd2623c31621354e, data reload: false
query1 0.05 0.05 0.04
query2 0.07 0.04 0.04
query3 0.22 0.05 0.05
query4 1.66 0.08 0.09
query5 0.52 0.48 0.51
query6 1.12 0.74 0.74
query7 0.02 0.01 0.02
query8 0.05 0.04 0.05
query9 0.55 0.49 0.50
query10 0.54 0.57 0.55
query11 0.15 0.12 0.11
query12 0.15 0.13 0.12
query13 0.59 0.59 0.59
query14 1.43 1.43 1.42
query15 0.84 0.82 0.81
query16 0.35 0.38 0.38
query17 0.98 1.01 0.97
query18 0.21 0.20 0.20
query19 1.83 1.73 1.72
query20 0.01 0.01 0.01
query21 15.40 0.67 0.66
query22 4.47 6.61 1.46
query23 18.30 1.36 1.27
query24 2.03 0.21 0.22
query25 0.16 0.08 0.08
query26 0.26 0.18 0.18
query27 0.08 0.08 0.08
query28 13.35 1.03 0.99
query29 12.61 3.38 3.37
query30 0.24 0.05 0.05
query31 2.88 0.40 0.40
query32 3.24 0.48 0.48
query33 2.94 3.02 3.00
query34 16.99 4.38 4.39
query35 4.47 4.48 4.43
query36 0.66 0.49 0.46
query37 0.18 0.16 0.15
query38 0.15 0.14 0.14
query39 0.04 0.03 0.04
query40 0.17 0.14 0.13
query41 0.09 0.05 0.06
query42 0.06 0.05 0.04
query43 0.04 0.04 0.04
Total cold run time: 110.15 s
Total hot run time: 30.93 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2342789815
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2367114313
run fe_ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2326796738
<details>
<summary>ClickBench: <b>Total hot run time: 32.15 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0164e777ca8b0136c9f51cfb0f02de5fbf8cdf77, data reload: false
query1 0.05 0.04 0.04
query2 0.08 0.04 0.04
query3 0.22 0.05 0.05
query4 1.68 0.09 0.08
query5 0.50 0.49 0.49
query6 1.14 0.74 0.73
query7 0.02 0.02 0.01
query8 0.06 0.05 0.04
query9 0.54 0.49 0.49
query10 0.54 0.54 0.54
query11 0.15 0.12 0.12
query12 0.15 0.13 0.12
query13 0.61 0.59 0.58
query14 2.03 2.06 2.09
query15 0.93 0.83 0.82
query16 0.34 0.36 0.36
query17 0.99 1.04 0.98
query18 0.21 0.20 0.20
query19 1.82 1.75 1.80
query20 0.02 0.01 0.01
query21 15.44 0.67 0.66
query22 4.54 6.47 2.12
query23 18.26 1.36 1.27
query24 2.17 0.21 0.22
query25 0.15 0.08 0.08
query26 0.27 0.18 0.18
query27 0.08 0.08 0.08
query28 13.24 1.02 1.01
query29 12.66 3.34 3.31
query30 0.25 0.05 0.05
query31 2.88 0.39 0.39
query32 3.25 0.47 0.47
query33 3.05 2.98 3.04
query34 16.65 4.36 4.39
query35 4.48 4.42 4.44
query36 0.66 0.48 0.47
query37 0.18 0.14 0.15
query38 0.17 0.15 0.15
query39 0.06 0.03 0.04
query40 0.15 0.13 0.13
query41 0.09 0.05 0.04
query42 0.06 0.05 0.05
query43 0.04 0.05 0.04
Total cold run time: 110.86 s
Total hot run time: 32.15 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2350880977
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2360213895
run fe_ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2350809804
<details>
<summary>TPC-H: <b>Total hot run time: 42685 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6880357d4ce8561cbf7f0d063fe0093a15a9e286, data reload: false
------ Round 1 ----------------------------------
q1 17578 7215 7204 7204
q2 2029 188 180 180
q3 10470 1229 1352 1229
q4 10308 1043 1017 1017
q5 7718 3143 3121 3121
q6 238 155 150 150
q7 1037 634 608 608
q8 9453 1990 2001 1990
q9 6831 6293 6334 6293
q10 7028 2511 2512 2511
q11 456 253 246 246
q12 403 223 229 223
q13 17769 3027 3046 3027
q14 299 254 257 254
q15 593 531 539 531
q16 567 429 421 421
q17 995 929 959 929
q18 7431 6801 6687 6687
q19 1369 1235 1238 1235
q20 600 350 331 331
q21 3893 3526 3490 3490
q22 1039 1008 1018 1008
Total cold run time: 108104 ms
Total hot run time: 42685 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7127 7132 7098 7098
q2 334 233 233 233
q3 3103 3103 3111 3103
q4 2071 2087 2101 2087
q5 5612 5600 5671 5600
q6 243 151 147 147
q7 2180 1796 1764 1764
q8 3362 3425 3431 3425
q9 8764 8929 8768 8768
q10 3523 3562 3569 3562
q11 583 484 481 481
q12 826 597 618 597
q13 10661 3235 3194 3194
q14 328 287 290 287
q15 584 532 554 532
q16 512 483 476 476
q17 1792 1725 1721 1721
q18 8412 7996 8063 7996
q19 1741 1732 1743 1732
q20 2099 1874 1872 1872
q21 5985 5533 5662 5533
q22 1154 1058 1005 1005
Total cold run time: 70996 ms
Total hot run time: 61213 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2359913308
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2342874465
<details>
<summary>TPC-DS: <b>Total hot run time: 196898 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0cbe9fd363773e1611b3356acd2623c31621354e, data reload: false
query1 1250 895 861 861
query2 6445 1861 1843 1843
query3 10648 3870 3786 3786
query4 59188 25664 23315 23315
query5 5000 490 494 490
query6 396 169 168 168
query7 5638 294 280 280
query8 301 217 210 210
query9 7653 2481 2460 2460
query10 401 260 264 260
query11 16929 15065 15229 15065
query12 147 105 105 105
query13 1451 384 380 380
query14 11035 7375 7559 7375
query15 218 180 179 179
query16 6855 493 506 493
query17 1085 559 546 546
query18 1535 283 293 283
query19 196 149 146 146
query20 121 106 115 106
query21 208 101 102 101
query22 4688 4559 4591 4559
query23 34420 33319 33638 33319
query24 5977 2900 2849 2849
query25 517 401 380 380
query26 620 152 151 151
query27 1627 271 273 271
query28 3676 2045 2018 2018
query29 656 399 404 399
query30 234 157 144 144
query31 925 783 785 783
query32 75 50 49 49
query33 420 279 275 275
query34 891 466 459 459
query35 849 734 701 701
query36 1061 946 954 946
query37 139 85 84 84
query38 4055 3816 3956 3816
query39 1451 1399 1397 1397
query40 193 116 114 114
query41 49 46 48 46
query42 116 97 100 97
query43 520 461 457 457
query44 1069 747 740 740
query45 197 167 166 166
query46 1087 755 736 736
query47 1937 1781 1831 1781
query48 379 285 293 285
query49 791 457 454 454
query50 833 407 404 404
query51 7108 7501 6929 6929
query52 101 87 87 87
query53 261 178 177 177
query54 559 462 458 458
query55 75 77 74 74
query56 285 255 272 255
query57 1223 1088 1080 1080
query58 227 237 233 233
query59 2952 2721 2810 2721
query60 314 282 270 270
query61 124 125 121 121
query62 751 673 684 673
query63 225 189 186 186
query64 1473 744 731 731
query65 3236 3199 3145 3145
query66 647 346 359 346
query67 16103 15348 15397 15348
query68 2017 670 547 547
query69 410 289 277 277
query70 1189 1148 1094 1094
query71 336 268 275 268
query72 4728 4097 4167 4097
query73 741 320 327 320
query74 9269 8903 8847 8847
query75 3343 2692 2728 2692
query76 1297 978 935 935
query77 562 320 313 313
query78 9844 9170 9627 9170
query79 1220 881 860 860
query80 1089 818 798 798
query81 543 260 260 260
query82 1358 266 265 265
query83 237 187 182 182
query84 259 104 107 104
query85 703 427 387 387
query86 316 316 324 316
query87 4383 4545 4407 4407
query88 4659 4049 4032 4032
query89 391 364 364 364
query90 1969 304 305 304
query91 121 119 120 119
query92 81 80 76 76
query93 959 926 918 918
query94 788 363 383 363
query95 437 405 401 401
query96 466 466 468 466
query97 3135 3138 3069 3069
query98 232 234 226 226
query99 1500 1310 1255 1255
Total cold run time: 300398 ms
Total hot run time: 196898 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2326784198
<details>
<summary>TPC-DS: <b>Total hot run time: 188871 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0164e777ca8b0136c9f51cfb0f02de5fbf8cdf77, data reload: false
query1 905 389 373 373
query2 6474 2103 2068 2068
query3 6655 204 214 204
query4 34121 23213 23242 23213
query5 4187 499 506 499
query6 254 179 167 167
query7 4597 302 284 284
query8 262 208 202 202
query9 8650 2490 2476 2476
query10 443 278 260 260
query11 17982 15016 14991 14991
query12 145 97 95 95
query13 1618 388 363 363
query14 9872 7528 7280 7280
query15 291 179 171 171
query16 7857 442 457 442
query17 1605 582 553 553
query18 1910 279 280 279
query19 330 140 143 140
query20 120 109 106 106
query21 209 102 102 102
query22 4591 4128 4151 4128
query23 34036 33897 33194 33194
query24 11160 2833 2877 2833
query25 615 384 378 378
query26 1092 157 157 157
query27 2328 277 272 272
query28 7244 2037 2028 2028
query29 769 407 421 407
query30 335 161 154 154
query31 990 783 798 783
query32 96 53 60 53
query33 755 289 282 282
query34 974 462 488 462
query35 834 739 717 717
query36 1114 925 948 925
query37 163 98 97 97
query38 3986 3982 3918 3918
query39 1462 1452 1534 1452
query40 211 122 113 113
query41 50 45 47 45
query42 117 98 101 98
query43 529 497 466 466
query44 1292 741 732 732
query45 202 166 173 166
query46 1096 763 760 760
query47 1855 1777 1789 1777
query48 385 291 296 291
query49 1005 455 461 455
query50 812 419 416 416
query51 7019 6980 6818 6818
query52 103 86 86 86
query53 262 185 186 185
query54 929 452 452 452
query55 78 74 74 74
query56 270 250 270 250
query57 1176 1075 1100 1075
query58 232 232 247 232
query59 3034 2959 3027 2959
query60 291 267 265 265
query61 105 99 101 99
query62 805 674 634 634
query63 225 187 182 182
query64 4256 674 644 644
query65 3208 3164 3129 3129
query66 726 332 344 332
query67 15673 15248 15333 15248
query68 3196 597 576 576
query69 393 286 284 284
query70 1170 1061 1131 1061
query71 347 277 279 277
query72 6473 4188 4154 4154
query73 744 332 340 332
query74 9005 8851 8990 8851
query75 3415 2694 2693 2693
query76 1998 980 1043 980
query77 538 339 342 339
query78 9556 9509 9332 9332
query79 1845 541 556 541
query80 1134 556 550 550
query81 597 244 236 236
query82 977 150 151 150
query83 238 159 163 159
query84 236 86 79 79
query85 1140 351 351 351
query86 308 312 305 305
query87 4362 4238 4320 4238
query88 3120 2446 2328 2328
query89 395 292 298 292
query90 1590 194 194 194
query91 140 127 116 116
query92 60 53 54 53
query93 1178 540 533 533
query94 544 315 298 298
query95 354 265 264 264
query96 585 267 265 265
query97 3178 3070 3097 3070
query98 218 218 198 198
query99 1578 1290 1281 1281
Total cold run time: 287589 ms
Total hot run time: 188871 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2350890724
<details>
<summary>TPC-H: <b>Total hot run time: 42689 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4a92ac912885fe39fc2b287c2ff5dd79f3cc6b33, data reload: false
------ Round 1 ----------------------------------
q1 17599 7334 7184 7184
q2 2026 184 178 178
q3 10489 1262 1409 1262
q4 10518 974 989 974
q5 7719 3182 3145 3145
q6 239 154 150 150
q7 1030 639 615 615
q8 9461 2039 2003 2003
q9 6850 6283 6302 6283
q10 7012 2504 2528 2504
q11 424 246 255 246
q12 406 227 230 227
q13 17758 3021 3042 3021
q14 280 243 254 243
q15 577 517 528 517
q16 518 421 423 421
q17 969 924 931 924
q18 7315 6740 6762 6740
q19 1378 1233 1232 1232
q20 613 332 329 329
q21 3874 3514 3494 3494
q22 1073 997 1010 997
Total cold run time: 108128 ms
Total hot run time: 42689 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7132 7101 7135 7101
q2 342 229 234 229
q3 3033 3103 3046 3046
q4 2022 2118 2085 2085
q5 5638 5588 5639 5588
q6 240 153 146 146
q7 2169 1751 1768 1751
q8 3358 3429 3401 3401
q9 8783 8981 8784 8784
q10 3531 3522 3586 3522
q11 590 489 487 487
q12 875 607 595 595
q13 9439 3193 3150 3150
q14 304 292 276 276
q15 576 532 554 532
q16 526 452 461 452
q17 1800 1752 1720 1720
q18 8505 8057 7969 7969
q19 1792 1735 1734 1734
q20 2145 1881 1884 1881
q21 5888 5521 5609 5521
q22 1112 1045 1020 1020
Total cold run time: 69800 ms
Total hot run time: 60990 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2350897227
<details>
<summary>ClickBench: <b>Total hot run time: 30.74 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4a92ac912885fe39fc2b287c2ff5dd79f3cc6b33, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.04 0.03
query3 0.22 0.05 0.04
query4 1.68 0.06 0.06
query5 0.50 0.50 0.50
query6 1.14 0.73 0.73
query7 0.02 0.01 0.01
query8 0.05 0.04 0.05
query9 0.56 0.53 0.52
query10 0.56 0.59 0.56
query11 0.15 0.12 0.12
query12 0.15 0.12 0.12
query13 0.61 0.61 0.61
query14 1.49 1.45 1.49
query15 0.89 0.87 0.87
query16 0.35 0.36 0.37
query17 1.02 1.00 1.02
query18 0.16 0.16 0.17
query19 1.93 1.76 1.79
query20 0.01 0.01 0.01
query21 15.39 0.66 0.65
query22 4.23 8.98 1.12
query23 17.79 1.31 1.31
query24 2.28 0.23 0.22
query25 0.19 0.07 0.09
query26 0.30 0.18 0.18
query27 0.08 0.07 0.09
query28 13.18 1.12 1.10
query29 12.53 3.36 3.33
query30 0.24 0.05 0.06
query31 2.88 0.43 0.42
query32 3.23 0.49 0.49
query33 3.01 3.06 3.02
query34 15.51 4.32 4.31
query35 4.33 4.33 4.33
query36 0.69 0.50 0.49
query37 0.19 0.16 0.16
query38 0.15 0.16 0.15
query39 0.04 0.03 0.04
query40 0.17 0.13 0.14
query41 0.10 0.05 0.05
query42 0.06 0.04 0.04
query43 0.05 0.04 0.03
Total cold run time: 108.23 s
Total hot run time: 30.74 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on code in PR #40340:
URL: https://github.com/apache/doris/pull/40340#discussion_r1759682800
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java:
##########
@@ -959,7 +961,19 @@ public boolean commitAndPublishTransaction(DatabaseIf db, long transactionId,
public boolean commitAndPublishTransaction(DatabaseIf db, List<Table> tableList, long transactionId,
List<TabletCommitInfo> tabletCommitInfos, long timeoutMillis,
TxnCommitAttachment txnCommitAttachment) throws UserException {
+ for (int i = 0; i < tableList.size(); i++) {
+ long tableId = tableList.get(i).getId();
+ LOG.info("start commit txn=" + transactionId + ",table=" + tableId);
+ }
+ for (Map.Entry<Long, AtomicInteger> entry : waitToCommitTxnCountMap.entrySet()) {
+ if (entry.getValue().get() > 5) {
+ LOG.info("now table {} commitAndPublishTransaction queue is {}", entry.getKey(),
+ entry.getValue().get());
+ }
+ }
+ increaseCount(tableList);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #40340:
URL: https://github.com/apache/doris/pull/40340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2348296723
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2348296781
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2350884502
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2342851300
<details>
<summary>TPC-H: <b>Total hot run time: 38198 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0cbe9fd363773e1611b3356acd2623c31621354e, data reload: false
------ Round 1 ----------------------------------
q1 17607 5358 4313 4313
q2 2015 189 189 189
q3 11789 990 1156 990
q4 10518 806 732 732
q5 7723 2857 2783 2783
q6 230 141 143 141
q7 953 620 622 620
q8 9314 2098 2199 2098
q9 7261 6570 6566 6566
q10 7012 2222 2226 2222
q11 457 241 244 241
q12 404 230 233 230
q13 17768 3057 3081 3057
q14 280 236 251 236
q15 543 503 483 483
q16 538 431 436 431
q17 981 696 749 696
q18 7456 6769 6901 6769
q19 1396 1025 1077 1025
q20 681 337 345 337
q21 3902 3065 3031 3031
q22 1079 1053 1008 1008
Total cold run time: 109907 ms
Total hot run time: 38198 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4411 4282 4345 4282
q2 379 282 279 279
q3 2896 2692 2678 2678
q4 1977 1677 1662 1662
q5 5695 5702 5798 5702
q6 241 138 139 138
q7 2235 1845 1844 1844
q8 3318 3466 3523 3466
q9 8885 8899 8840 8840
q10 3662 3433 3356 3356
q11 615 523 502 502
q12 826 676 674 674
q13 14888 3302 3340 3302
q14 314 292 290 290
q15 560 482 501 482
q16 545 476 511 476
q17 1834 1588 1556 1556
q18 8073 7874 7883 7874
q19 1808 1517 1486 1486
q20 2200 1905 1958 1905
q21 5854 5741 5515 5515
q22 1133 1073 1030 1030
Total cold run time: 72349 ms
Total hot run time: 57339 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #40340:
URL: https://github.com/apache/doris/pull/40340#discussion_r1759643723
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java:
##########
@@ -959,7 +961,19 @@ public boolean commitAndPublishTransaction(DatabaseIf db, long transactionId,
public boolean commitAndPublishTransaction(DatabaseIf db, List<Table> tableList, long transactionId,
List<TabletCommitInfo> tabletCommitInfos, long timeoutMillis,
TxnCommitAttachment txnCommitAttachment) throws UserException {
+ for (int i = 0; i < tableList.size(); i++) {
+ long tableId = tableList.get(i).getId();
+ LOG.info("start commit txn=" + transactionId + ",table=" + tableId);
+ }
+ for (Map.Entry<Long, AtomicInteger> entry : countMap.entrySet()) {
+ if (entry.getValue().get() > 5) {
+ LOG.info("now table {} commitAndPublishTransaction queue is {}", entry.getKey(),
Review Comment:
also print queue length?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](cloud-mow) print the queue length of committing txn on FE [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #40340:
URL: https://github.com/apache/doris/pull/40340#issuecomment-2350796932
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org