You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Jibing-Li (via GitHub)" <gi...@apache.org> on 2024/01/08 11:52:11 UTC
[PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Jibing-Li opened a new pull request, #29685:
URL: https://github.com/apache/doris/pull/29685
Only write editlog for manual analyze task, don't need to do so for auto tasks to reduce writing editlog.
Add error message to job info while task failed.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29685:
URL: https://github.com/apache/doris/pull/29685#issuecomment-1882467795
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29685:
URL: https://github.com/apache/doris/pull/29685#issuecomment-1882384260
<details>
<summary>TPC-DS: <b>Total hot run time: 184189 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 36e90b4a52331004f70821334f22d814787f6c48, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 929 349 336 336
query2 6659 1971 2023 1971
query3 6741 226 222 222
query4 27088 22387 22406 22387
query5 5798 563 605 563
query6 338 199 200 199
query7 4624 311 292 292
query8 260 232 222 222
query9 8845 2910 2931 2910
query10 533 308 318 308
query11 15767 15734 15695 15695
query12 184 99 99 99
query13 1684 326 346 326
query14 13415 7577 7697 7577
query15 277 269 223 223
query16 6429 328 327 327
query17 1718 539 520 520
query18 1957 297 310 297
query19 240 158 164 158
query20 106 100 104 100
query21 188 101 95 95
query22 4934 4897 4849 4849
query23 32188 31792 31244 31244
query24 11850 2924 2953 2924
query25 598 361 379 361
query26 1808 170 179 170
query27 2765 313 318 313
query28 6652 2101 2082 2082
query29 2056 415 413 413
query30 289 148 151 148
query31 1023 825 812 812
query32 132 88 91 88
query33 845 338 344 338
query34 851 473 469 469
query35 988 906 908 906
query36 1395 1373 1329 1329
query37 216 94 93 93
query38 3459 3330 3351 3330
query39 1500 1306 1311 1306
query40 307 91 87 87
query41 38 43 37 37
query42 111 104 110 104
query43 562 532 506 506
query44 1279 815 820 815
query45 215 209 207 207
query46 1061 714 709 709
query47 1735 1667 1608 1608
query48 371 294 292 292
query49 1205 311 326 311
query50 725 334 334 334
query51 5449 5372 5244 5244
query52 117 95 100 95
query53 246 175 178 175
query54 1493 666 720 666
query55 114 95 97 95
query56 270 271 257 257
query57 1022 991 1000 991
query58 320 273 289 273
query59 3038 2966 2882 2882
query60 313 300 282 282
query61 132 144 148 144
query62 589 465 481 465
query63 202 184 188 184
query64 5898 1779 1718 1718
query65 3409 3355 3366 3355
query66 1368 397 371 371
query67 15645 15436 15069 15069
query68 11834 539 547 539
query69 619 335 304 304
query70 1828 1643 1674 1643
query71 604 308 298 298
query72 5253 3444 3449 3444
query73 2451 333 336 333
query74 7175 6453 6508 6453
query75 4864 2321 2300 2300
query76 6389 1175 1278 1175
query77 768 306 308 306
query78 9049 8576 8544 8544
query79 1072 554 534 534
query80 551 361 352 352
query81 462 216 214 214
query82 236 106 105 105
query83 167 141 140 140
query84 256 59 60 59
query85 933 276 271 271
query86 429 449 418 418
query87 3628 3497 3463 3463
query88 2991 2504 2519 2504
query89 366 297 295 295
query90 1888 256 258 256
query91 127 92 96 92
query92 85 69 70 69
query93 1021 483 437 437
query94 839 246 253 246
query95 548 509 499 499
query96 658 332 340 332
query97 4338 4226 4217 4217
query98 244 217 212 212
query99 1177 857 865 857
Total cold run time: 297413 ms
Total hot run time: 184189 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29685:
URL: https://github.com/apache/doris/pull/29685#issuecomment-1880888473
<details>
<summary>TPC-H: <b>Total hot run time: 38653 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 180df019c583399163e2bfa769249a9af93797cc, data reload: false
------ Round 1 ----------------------------------
q1 17730 4923 4964 4923
q2 2027 155 144 144
q3 10799 1111 1114 1111
q4 10518 822 788 788
q5 7800 2929 2990 2929
q6 201 123 122 122
q7 913 513 489 489
q8 9516 1999 2018 1999
q9 8338 6406 6404 6404
q10 8311 3041 3033 3033
q11 428 226 205 205
q12 386 226 227 226
q13 18009 3374 3351 3351
q14 239 212 207 207
q15 549 508 497 497
q16 493 403 397 397
q17 953 717 577 577
q18 7276 6762 6715 6715
q19 1577 1537 1529 1529
q20 705 331 305 305
q21 2735 2396 2407 2396
q22 352 306 320 306
Total cold run time: 109855 ms
Total hot run time: 38653 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4937 4960 4924 4924
q2 341 234 232 232
q3 3280 3325 3251 3251
q4 2106 2014 1987 1987
q5 5797 5770 5752 5752
q6 204 116 114 114
q7 2339 1867 1897 1867
q8 3378 3441 3436 3436
q9 8772 8734 8752 8734
q10 3797 3861 3848 3848
q11 563 432 436 432
q12 786 631 616 616
q13 3804 3158 3154 3154
q14 300 269 258 258
q15 547 508 497 497
q16 535 507 519 507
q17 1858 1857 1867 1857
q18 8582 8385 8317 8317
q19 1597 1643 1628 1628
q20 2197 1953 1952 1952
q21 5622 5310 5167 5167
q22 507 437 435 435
Total cold run time: 61849 ms
Total hot run time: 58965 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #29685:
URL: https://github.com/apache/doris/pull/29685#issuecomment-1882343919
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29685:
URL: https://github.com/apache/doris/pull/29685#issuecomment-1882467763
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #29685:
URL: https://github.com/apache/doris/pull/29685#issuecomment-1880861663
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29685:
URL: https://github.com/apache/doris/pull/29685#issuecomment-1880908568
<details>
<summary>TPC-DS: <b>Total hot run time: 183122 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 180df019c583399163e2bfa769249a9af93797cc, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 934 342 340 340
query2 6550 2008 1964 1964
query3 6713 225 215 215
query4 26897 22467 22499 22467
query5 5220 592 566 566
query6 334 213 210 210
query7 4626 313 296 296
query8 252 223 215 215
query9 8662 2920 2933 2920
query10 541 332 292 292
query11 16355 15740 15582 15582
query12 177 99 97 97
query13 1678 349 333 333
query14 13256 7475 7576 7475
query15 244 252 215 215
query16 6230 342 337 337
query17 1621 521 518 518
query18 1972 313 297 297
query19 276 159 159 159
query20 106 104 103 103
query21 187 88 93 88
query22 5135 4917 4855 4855
query23 32663 30831 31375 30831
query24 11720 2941 2963 2941
query25 509 351 366 351
query26 1498 166 165 165
query27 2780 314 318 314
query28 6832 2157 2105 2105
query29 959 426 410 410
query30 290 144 149 144
query31 1039 818 839 818
query32 126 73 74 73
query33 804 346 353 346
query34 858 468 486 468
query35 954 894 841 841
query36 1416 1234 1313 1234
query37 208 81 85 81
query38 3464 3358 3329 3329
query39 1343 1293 1306 1293
query40 304 92 85 85
query41 39 36 38 36
query42 114 103 113 103
query43 577 523 527 523
query44 1144 797 764 764
query45 216 203 202 202
query46 1062 688 670 670
query47 1644 1604 1571 1571
query48 345 282 287 282
query49 1125 315 305 305
query50 774 355 323 323
query51 5494 5275 5467 5275
query52 107 98 98 98
query53 251 164 159 159
query54 1411 664 740 664
query55 106 95 95 95
query56 255 239 247 239
query57 1012 963 1008 963
query58 294 287 266 266
query59 3090 2875 2926 2875
query60 315 288 288 288
query61 149 147 161 147
query62 578 491 471 471
query63 196 180 169 169
query64 4648 1741 1747 1741
query65 3416 3340 3330 3330
query66 1236 381 375 375
query67 15587 15123 15095 15095
query68 13787 537 562 537
query69 615 303 316 303
query70 2115 1604 1624 1604
query71 612 298 270 270
query72 6094 3422 3474 3422
query73 3449 346 345 345
query74 6965 6496 6419 6419
query75 6237 2292 2281 2281
query76 6704 1244 1218 1218
query77 797 352 333 333
query78 9104 8624 8579 8579
query79 2613 522 520 520
query80 580 342 363 342
query81 466 215 214 214
query82 241 111 102 102
query83 306 144 145 144
query84 262 56 61 56
query85 884 295 268 268
query86 469 433 444 433
query87 3633 3383 3452 3383
query88 3118 2498 2496 2496
query89 374 284 281 281
query90 2156 260 252 252
query91 125 94 99 94
query92 77 79 78 78
query93 1300 442 440 440
query94 943 246 238 238
query95 582 502 496 496
query96 643 340 347 340
query97 4373 4171 4256 4171
query98 246 224 228 224
query99 1129 837 845 837
Total cold run time: 301917 ms
Total hot run time: 183122 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29685:
URL: https://github.com/apache/doris/pull/29685#issuecomment-1882376583
<details>
<summary>TPC-H: <b>Total hot run time: 38349 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 36e90b4a52331004f70821334f22d814787f6c48, data reload: false
------ Round 1 ----------------------------------
q1 17706 4988 4894 4894
q2 2011 157 144 144
q3 10576 1130 1194 1130
q4 10449 772 811 772
q5 7802 2993 2898 2898
q6 203 124 124 124
q7 911 499 501 499
q8 9242 1995 1972 1972
q9 6460 6392 6318 6318
q10 8214 3014 2984 2984
q11 421 217 230 217
q12 394 233 228 228
q13 18013 3431 3395 3395
q14 243 213 202 202
q15 550 515 493 493
q16 453 411 395 395
q17 954 722 610 610
q18 7324 6682 6648 6648
q19 1568 1507 1501 1501
q20 731 327 301 301
q21 2755 2352 2312 2312
q22 356 312 323 312
Total cold run time: 107336 ms
Total hot run time: 38349 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4934 4948 4950 4948
q2 336 233 253 233
q3 3278 3277 3276 3276
q4 2126 2014 2030 2014
q5 5815 5744 5770 5744
q6 202 117 116 116
q7 2298 1905 1921 1905
q8 3395 3463 3433 3433
q9 8791 8786 8746 8746
q10 3792 3834 3844 3834
q11 547 442 466 442
q12 801 636 646 636
q13 7846 3179 3176 3176
q14 285 259 254 254
q15 545 503 508 503
q16 528 509 489 489
q17 1838 1866 1861 1861
q18 8633 8328 8204 8204
q19 1578 1624 1616 1616
q20 2179 1991 1961 1961
q21 5579 5316 5377 5316
q22 535 451 450 450
Total cold run time: 65861 ms
Total hot run time: 59157 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29685:
URL: https://github.com/apache/doris/pull/29685#issuecomment-1880920804
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.79 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.1 seconds inserted 10000000 Rows, about 369K ops/s
storage size: 17183905442 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #29685:
URL: https://github.com/apache/doris/pull/29685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Only write editlog for manual analyze task, don't need to do so for auto tasks. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29685:
URL: https://github.com/apache/doris/pull/29685#issuecomment-1882383340
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.52 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.4 seconds inserted 10000000 Rows, about 364K ops/s
storage size: 17183693162 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org