You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zy-kkk (via GitHub)" <gi...@apache.org> on 2025/11/20 03:15:09 UTC
[PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
zy-kkk opened a new pull request, #58166:
URL: https://github.com/apache/doris/pull/58166
### What problem does this PR solve?
For Hive tables with massive partitions (10K+), INSERT operations are extremely slow because:
- FE fetches all partition metadata from HMS directly (expensive RPC calls)
- Full table cache invalidation after each insert (unnecessary)
Problem Summary:
1. **Use cache for partition metadata in INSERT**
- FE now fetches partition info from cache instead of directly querying HMS when preparing INSERT
- Avoid expensive HMS RPC calls for every INSERT operation
2. **Selective cache refresh after commit**
- Only invalidate affected partitions instead of full table cache
- Based on partition update info from BE (NEW/APPEND/OVERWRITE)
- Significantly reduces cache invalidation overhead
3. **Handle cache inconsistency gracefully**
- When BE marks partition as NEW but it already exists in HMS (cache miss)
- FE detects this by checking HMS and treats it as APPEND instead of failing
- Prevents `AlreadyExistsException` errors
For tables with partitions:
- **Before**: HMS calls per INSERT + full cache invalidation
- **After**: cache lookup + selective partition refresh
- Expected speedup: 10x-100x for partition metadata fetching phas
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3555738836
<details>
<summary>TPC-H: <b>Total hot run time: 34362 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f37442a801d36436009ff8fdb6ea6bd9a9f56a09, data reload: false
------ Round 1 ----------------------------------
q1 17606 5039 4916 4916
q2 2037 320 200 200
q3 10266 1285 700 700
q4 10225 903 364 364
q5 7483 2400 2342 2342
q6 183 166 135 135
q7 950 767 626 626
q8 9354 1362 1113 1113
q9 7179 5364 5332 5332
q10 6900 2232 1792 1792
q11 494 292 280 280
q12 373 366 234 234
q13 17790 3661 3073 3073
q14 239 237 223 223
q15 581 522 505 505
q16 990 1007 943 943
q17 597 863 371 371
q18 7909 7277 7211 7211
q19 1101 947 546 546
q20 366 352 232 232
q21 3716 2547 2262 2262
q22 1078 1024 962 962
Total cold run time: 107417 ms
Total hot run time: 34362 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4968 4945 4943 4943
q2 338 420 313 313
q3 2202 2673 2314 2314
q4 1373 1780 1341 1341
q5 4220 4380 4529 4380
q6 211 165 124 124
q7 2031 1978 1882 1882
q8 2819 2630 2589 2589
q9 7646 7647 7531 7531
q10 3037 3232 2876 2876
q11 594 540 503 503
q12 687 783 648 648
q13 3772 3943 3334 3334
q14 286 306 278 278
q15 542 508 508 508
q16 1079 1102 1063 1063
q17 1205 1497 1429 1429
q18 7980 7685 7586 7586
q19 792 789 868 789
q20 1980 1966 1838 1838
q21 4769 4399 4370 4370
q22 1063 1041 998 998
Total cold run time: 53594 ms
Total hot run time: 51637 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3585530359
<details>
<summary>ClickBench: <b>Total hot run time: 27.27 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit badb955dee391a9cf9e9111b6ff8f5ebdf73bb27, data reload: false
query1 0.06 0.05 0.05
query2 0.10 0.05 0.05
query3 0.25 0.09 0.09
query4 1.61 0.11 0.10
query5 0.27 0.26 0.26
query6 1.18 0.64 0.63
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.57 0.51 0.51
query10 0.56 0.56 0.55
query11 0.17 0.11 0.11
query12 0.15 0.11 0.12
query13 0.61 0.59 0.62
query14 0.99 0.97 0.98
query15 0.81 0.80 0.80
query16 0.40 0.41 0.42
query17 1.04 1.01 0.98
query18 0.23 0.21 0.21
query19 1.94 1.86 1.85
query20 0.01 0.02 0.02
query21 15.48 0.27 0.13
query22 4.78 0.05 0.05
query23 16.08 0.26 0.11
query24 1.26 0.62 0.22
query25 0.07 0.08 0.05
query26 0.15 0.14 0.13
query27 0.09 0.04 0.06
query28 3.90 1.21 1.03
query29 12.57 3.91 3.21
query30 0.28 0.14 0.12
query31 2.83 0.59 0.39
query32 3.23 0.54 0.46
query33 3.12 3.04 3.07
query34 16.81 5.23 4.58
query35 4.54 4.57 4.52
query36 0.66 0.50 0.49
query37 0.10 0.07 0.07
query38 0.07 0.04 0.04
query39 0.04 0.03 0.04
query40 0.18 0.15 0.13
query41 0.08 0.03 0.03
query42 0.04 0.03 0.02
query43 0.04 0.04 0.03
Total cold run time: 97.42 s
Total hot run time: 27.27 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3584757791
<details>
<summary>TPC-DS: <b>Total hot run time: 185112 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 992bca6e7a2be03c9ca5d5a8c575bce69f0a4c33, data reload: false
query1 1061 412 399 399
query2 6633 1560 1599 1560
query3 6763 226 225 225
query4 25808 23190 22922 22922
query5 4477 665 504 504
query6 346 236 227 227
query7 4642 503 305 305
query8 313 264 237 237
query9 8722 2571 2569 2569
query10 534 359 324 324
query11 15610 14853 14583 14583
query12 190 120 117 117
query13 1697 570 434 434
query14 11318 9235 9268 9235
query15 215 206 194 194
query16 7653 690 529 529
query17 1255 815 617 617
query18 2028 426 331 331
query19 219 205 181 181
query20 134 126 129 126
query21 218 134 117 117
query22 3970 4060 3827 3827
query23 32992 32070 31862 31862
query24 8452 2432 2452 2432
query25 589 524 463 463
query26 1231 272 173 173
query27 2743 498 355 355
query28 4357 2144 2132 2132
query29 811 617 487 487
query30 318 239 217 217
query31 877 735 644 644
query32 88 75 73 73
query33 597 391 325 325
query34 821 876 550 550
query35 823 855 744 744
query36 933 961 868 868
query37 125 117 87 87
query38 3398 3461 3350 3350
query39 1564 1423 1414 1414
query40 222 135 124 124
query41 71 64 64 64
query42 135 115 110 110
query43 467 481 426 426
query44 1271 758 761 758
query45 198 193 191 191
query46 937 1011 652 652
query47 1678 1731 1668 1668
query48 392 438 335 335
query49 787 487 408 408
query50 694 702 409 409
query51 3897 3936 3872 3872
query52 116 115 109 109
query53 260 279 193 193
query54 309 293 279 279
query55 98 93 94 93
query56 355 348 336 336
query57 1154 1166 1114 1114
query58 295 290 280 280
query59 2330 2534 2372 2372
query60 356 363 358 358
query61 164 165 173 165
query62 793 725 644 644
query63 246 201 206 201
query64 4643 1307 1017 1017
query65 4069 3960 3994 3960
query66 1126 469 348 348
query67 15467 15039 14828 14828
query68 8395 925 634 634
query69 507 345 321 321
query70 1289 1142 1237 1142
query71 438 344 317 317
query72 5815 4906 4985 4906
query73 681 613 357 357
query74 8633 8527 8472 8472
query75 3514 3348 2846 2846
query76 3405 1148 739 739
query77 728 417 319 319
query78 9479 9638 8924 8924
query79 2050 826 594 594
query80 688 596 506 506
query81 507 273 241 241
query82 447 159 132 132
query83 274 272 261 261
query84 250 128 99 99
query85 907 494 447 447
query86 355 319 299 299
query87 3510 3587 3432 3432
query88 3670 2255 2263 2255
query89 404 336 293 293
query90 1972 231 229 229
query91 180 177 145 145
query92 89 69 64 64
query93 1706 1017 650 650
query94 788 451 339 339
query95 512 411 409 409
query96 515 591 293 293
query97 2901 2961 2913 2913
query98 248 211 211 211
query99 1402 1415 1270 1270
Total cold run time: 273314 ms
Total hot run time: 185112 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3592484873
<details>
<summary>TPC-DS: <b>Total hot run time: 182146 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 069ef57cb63b9a043fa2a08fcf29006c4e147c38, data reload: false
query1 1044 443 390 390
query2 6567 1197 1192 1192
query3 6759 230 221 221
query4 25072 23500 22907 22907
query5 4441 646 497 497
query6 345 240 224 224
query7 4644 501 307 307
query8 320 265 248 248
query9 8706 2624 2641 2624
query10 518 348 326 326
query11 15261 15135 15112 15112
query12 183 125 121 121
query13 1697 573 448 448
query14 9316 5969 6041 5969
query15 210 200 190 190
query16 7317 705 533 533
query17 1263 789 657 657
query18 2022 442 352 352
query19 214 211 195 195
query20 129 127 125 125
query21 220 138 121 121
query22 3801 3981 3856 3856
query23 32812 32114 31876 31876
query24 8649 2426 2417 2417
query25 662 566 497 497
query26 1248 281 175 175
query27 2735 488 351 351
query28 4340 2174 2161 2161
query29 844 681 545 545
query30 318 242 211 211
query31 869 767 622 622
query32 88 86 78 78
query33 641 398 390 390
query34 815 877 539 539
query35 803 840 729 729
query36 881 927 838 838
query37 119 107 89 89
query38 3849 3843 3762 3762
query39 1462 1403 1402 1402
query40 228 128 119 119
query41 68 65 66 65
query42 131 109 114 109
query43 448 458 432 432
query44 1302 766 751 751
query45 198 193 185 185
query46 879 1003 644 644
query47 1677 1727 1645 1645
query48 395 428 329 329
query49 759 500 426 426
query50 646 685 413 413
query51 4033 3911 4012 3911
query52 118 112 105 105
query53 247 263 203 203
query54 312 302 276 276
query55 104 99 93 93
query56 345 329 314 314
query57 1157 1168 1106 1106
query58 287 284 276 276
query59 2386 2409 2274 2274
query60 356 358 343 343
query61 162 161 160 160
query62 786 698 681 681
query63 232 200 201 200
query64 4559 1229 907 907
query65 4093 3987 3962 3962
query66 1170 445 343 343
query67 15593 15076 14809 14809
query68 8373 966 640 640
query69 524 346 307 307
query70 1100 1035 1022 1022
query71 485 351 319 319
query72 5829 5002 4677 4677
query73 706 572 345 345
query74 8904 8883 8631 8631
query75 3616 3036 2560 2560
query76 3750 1148 744 744
query77 806 416 324 324
query78 9467 9687 8804 8804
query79 1546 841 583 583
query80 639 596 507 507
query81 503 270 243 243
query82 422 142 117 117
query83 270 269 249 249
query84 257 110 101 101
query85 915 484 450 450
query86 341 303 281 281
query87 4166 4080 4041 4041
query88 2902 2329 2323 2323
query89 396 332 294 294
query90 1942 234 230 230
query91 173 169 138 138
query92 85 73 66 66
query93 1157 976 666 666
query94 716 443 338 338
query95 520 426 417 417
query96 517 527 291 291
query97 2633 2711 2566 2566
query98 235 226 209 209
query99 1400 1370 1285 1285
Total cold run time: 269150 ms
Total hot run time: 182146 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3592617895
# FE Regression Coverage Report
Increment line coverage ` 78.17% (111/142)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58166_069ef57cb63b9a043fa2a08fcf29006c4e147c38_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58166_069ef57cb63b9a043fa2a08fcf29006c4e147c38_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3585416053
# FE UT Coverage Report
Increment line coverage ` 36.75% (43/117)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58166_badb955dee391a9cf9e9111b6ff8f5ebdf73bb27/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58166_badb955dee391a9cf9e9111b6ff8f5ebdf73bb27/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3586309173
# FE Regression Coverage Report
Increment line coverage ` 4.27% (5/117)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58166_badb955dee391a9cf9e9111b6ff8f5ebdf73bb27_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58166_badb955dee391a9cf9e9111b6ff8f5ebdf73bb27_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3573638327
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3569304038
<details>
<summary>TPC-DS: <b>Total hot run time: 187095 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c6b4ca7ab1660349b505067613960c32684a28c, data reload: false
query1 1057 440 390 390
query2 6563 1713 1683 1683
query3 6755 221 220 220
query4 25875 23097 22923 22923
query5 4395 614 470 470
query6 330 239 216 216
query7 4662 490 294 294
query8 313 247 244 244
query9 8708 2625 2605 2605
query10 480 336 292 292
query11 15475 14943 14863 14863
query12 177 117 111 111
query13 1677 577 443 443
query14 10828 9159 9125 9125
query15 201 184 172 172
query16 7247 662 487 487
query17 1233 740 657 657
query18 1974 463 318 318
query19 212 204 173 173
query20 136 120 119 119
query21 218 134 115 115
query22 3916 4164 4010 4010
query23 33759 33073 32864 32864
query24 8523 2401 2352 2352
query25 670 521 432 432
query26 1240 270 167 167
query27 2745 490 370 370
query28 4415 2205 2188 2188
query29 844 627 481 481
query30 304 228 205 205
query31 910 799 733 733
query32 86 77 82 77
query33 601 381 319 319
query34 800 856 515 515
query35 798 831 739 739
query36 944 989 880 880
query37 117 112 88 88
query38 3525 3527 3495 3495
query39 1467 1418 1392 1392
query40 225 129 118 118
query41 65 62 63 62
query42 125 119 110 110
query43 486 484 470 470
query44 1263 769 779 769
query45 187 180 180 180
query46 878 1002 646 646
query47 1739 1769 1712 1712
query48 415 438 341 341
query49 809 505 425 425
query50 674 701 426 426
query51 4032 3905 3860 3860
query52 114 114 111 111
query53 260 269 201 201
query54 322 304 295 295
query55 90 89 86 86
query56 334 340 343 340
query57 1176 1189 1127 1127
query58 294 286 279 279
query59 2519 2621 2509 2509
query60 355 373 358 358
query61 200 196 194 194
query62 793 706 669 669
query63 234 199 201 199
query64 4609 1153 851 851
query65 4010 3935 3934 3934
query66 1186 440 327 327
query67 15144 15076 14907 14907
query68 5793 920 629 629
query69 491 320 291 291
query70 1368 1294 1225 1225
query71 429 333 320 320
query72 6004 5051 4944 4944
query73 666 618 373 373
query74 8943 9037 8636 8636
query75 3313 3351 2856 2856
query76 3314 1155 758 758
query77 531 408 322 322
query78 9396 9540 8871 8871
query79 2393 809 621 621
query80 752 621 524 524
query81 519 265 230 230
query82 386 165 135 135
query83 277 271 262 262
query84 260 124 102 102
query85 1017 498 447 447
query86 383 315 310 310
query87 3750 3733 3651 3651
query88 3651 2226 2250 2226
query89 379 333 294 294
query90 1989 244 225 225
query91 178 164 134 134
query92 81 67 63 63
query93 2642 999 677 677
query94 698 446 338 338
query95 414 324 318 318
query96 476 582 280 280
query97 2890 2960 2853 2853
query98 241 209 209 209
query99 1337 1403 1238 1238
Total cold run time: 271679 ms
Total hot run time: 187095 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3569388280
# FE UT Coverage Report
Increment line coverage ` 26.27% (31/118)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58166_6c6b4ca7ab1660349b505067613960c32684a28c/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58166_6c6b4ca7ab1660349b505067613960c32684a28c/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3569132069
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3556390361
# FE Regression Coverage Report
Increment line coverage ` 87.50% (98/112)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58166_f37442a801d36436009ff8fdb6ea6bd9a9f56a09_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58166_f37442a801d36436009ff8fdb6ea6bd9a9f56a09_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3584776554
<details>
<summary>ClickBench: <b>Total hot run time: 27.8 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 992bca6e7a2be03c9ca5d5a8c575bce69f0a4c33, data reload: false
query1 0.06 0.05 0.05
query2 0.10 0.05 0.05
query3 0.25 0.08 0.09
query4 1.61 0.11 0.11
query5 0.29 0.26 0.26
query6 1.21 0.67 0.65
query7 0.02 0.03 0.02
query8 0.06 0.04 0.04
query9 0.59 0.52 0.52
query10 0.58 0.57 0.57
query11 0.16 0.11 0.12
query12 0.16 0.12 0.12
query13 0.64 0.61 0.61
query14 1.00 1.03 1.00
query15 0.87 0.83 0.84
query16 0.39 0.41 0.41
query17 1.06 1.02 1.05
query18 0.21 0.20 0.20
query19 1.92 1.81 1.85
query20 0.02 0.01 0.01
query21 15.45 0.20 0.13
query22 5.04 0.06 0.04
query23 15.67 0.27 0.10
query24 2.73 0.68 0.54
query25 0.06 0.06 0.06
query26 0.14 0.14 0.14
query27 0.07 0.06 0.05
query28 4.27 1.15 0.95
query29 12.59 3.84 3.20
query30 0.28 0.13 0.11
query31 2.82 0.62 0.40
query32 3.24 0.55 0.48
query33 2.99 3.11 3.15
query34 15.88 5.16 4.58
query35 4.56 4.58 4.58
query36 0.69 0.51 0.50
query37 0.10 0.07 0.07
query38 0.06 0.04 0.04
query39 0.03 0.03 0.03
query40 0.18 0.15 0.13
query41 0.09 0.03 0.02
query42 0.04 0.03 0.03
query43 0.05 0.04 0.04
Total cold run time: 98.23 s
Total hot run time: 27.8 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3555781135
<details>
<summary>ClickBench: <b>Total hot run time: 27.48 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f37442a801d36436009ff8fdb6ea6bd9a9f56a09, data reload: false
query1 0.05 0.05 0.05
query2 0.10 0.05 0.05
query3 0.26 0.08 0.08
query4 1.60 0.11 0.11
query5 0.27 0.25 0.25
query6 1.18 0.64 0.65
query7 0.03 0.02 0.03
query8 0.06 0.04 0.04
query9 0.58 0.53 0.52
query10 0.57 0.58 0.58
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.63 0.60 0.59
query14 1.02 0.99 1.01
query15 0.85 0.82 0.82
query16 0.40 0.41 0.39
query17 1.04 1.04 1.02
query18 0.21 0.20 0.20
query19 1.83 1.82 1.84
query20 0.01 0.01 0.01
query21 15.46 0.22 0.14
query22 5.06 0.06 0.04
query23 15.67 0.27 0.10
query24 3.12 0.65 0.41
query25 0.07 0.06 0.06
query26 0.15 0.14 0.13
query27 0.06 0.05 0.05
query28 3.53 1.17 0.95
query29 12.58 3.84 3.21
query30 0.28 0.14 0.11
query31 2.81 0.58 0.40
query32 3.23 0.55 0.47
query33 2.95 3.11 3.06
query34 15.66 5.15 4.55
query35 4.56 4.56 4.57
query36 0.69 0.50 0.49
query37 0.10 0.06 0.07
query38 0.07 0.04 0.04
query39 0.03 0.02 0.02
query40 0.16 0.16 0.13
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 97.4 s
Total hot run time: 27.48 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3592478155
<details>
<summary>TPC-H: <b>Total hot run time: 34232 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 069ef57cb63b9a043fa2a08fcf29006c4e147c38, data reload: false
------ Round 1 ----------------------------------
q1 17643 5031 4928 4928
q2 2053 311 210 210
q3 10266 1317 724 724
q4 10233 944 373 373
q5 7522 2516 2212 2212
q6 187 170 141 141
q7 931 793 645 645
q8 9359 1348 1007 1007
q9 6963 5298 5277 5277
q10 6814 2235 1853 1853
q11 526 311 285 285
q12 349 374 228 228
q13 17764 3704 3040 3040
q14 246 227 216 216
q15 577 523 521 521
q16 906 863 817 817
q17 587 771 501 501
q18 7469 7102 7072 7072
q19 1098 967 578 578
q20 346 336 230 230
q21 3535 3741 2409 2409
q22 1038 1000 965 965
Total cold run time: 106412 ms
Total hot run time: 34232 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4983 4961 4964 4961
q2 322 404 325 325
q3 2119 2699 2289 2289
q4 1322 1755 1355 1355
q5 4193 4421 4666 4421
q6 218 177 128 128
q7 2020 2026 1848 1848
q8 2661 2501 2535 2501
q9 7484 7446 7405 7405
q10 3042 3295 2799 2799
q11 577 532 494 494
q12 712 782 625 625
q13 3579 3967 3336 3336
q14 296 313 295 295
q15 546 517 525 517
q16 910 959 884 884
q17 1190 1381 1394 1381
q18 8338 7583 7623 7583
q19 819 810 798 798
q20 2057 2113 1904 1904
q21 4869 4389 4289 4289
q22 1081 1024 999 999
Total cold run time: 53338 ms
Total hot run time: 51137 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3592516260
# FE UT Coverage Report
Increment line coverage ` 30.28% (43/142)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58166_069ef57cb63b9a043fa2a08fcf29006c4e147c38/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58166_069ef57cb63b9a043fa2a08fcf29006c4e147c38/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3573750782
<details>
<summary>ClickBench: <b>Total hot run time: 27.74 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f988f83353e0a7e0ee3d3447cef9e208469e17e8, data reload: false
query1 0.06 0.05 0.05
query2 0.10 0.05 0.04
query3 0.26 0.09 0.08
query4 1.60 0.12 0.11
query5 0.27 0.26 0.26
query6 1.20 0.64 0.66
query7 0.03 0.03 0.02
query8 0.06 0.04 0.04
query9 0.58 0.52 0.52
query10 0.58 0.56 0.57
query11 0.16 0.11 0.12
query12 0.15 0.12 0.12
query13 0.62 0.60 0.60
query14 1.02 0.99 1.00
query15 0.84 0.86 0.83
query16 0.38 0.38 0.40
query17 1.01 1.00 1.02
query18 0.22 0.20 0.20
query19 1.94 1.82 1.80
query20 0.02 0.01 0.03
query21 15.44 0.20 0.13
query22 5.10 0.07 0.05
query23 15.65 0.26 0.10
query24 2.34 0.91 0.65
query25 0.07 0.06 0.07
query26 0.15 0.14 0.13
query27 0.06 0.06 0.06
query28 4.90 1.13 0.94
query29 12.60 3.97 3.18
query30 0.28 0.13 0.12
query31 2.81 0.57 0.38
query32 3.24 0.55 0.46
query33 3.07 3.10 3.10
query34 15.90 5.14 4.58
query35 4.58 4.57 4.63
query36 0.68 0.50 0.50
query37 0.10 0.07 0.07
query38 0.06 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 98.51 s
Total hot run time: 27.74 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3585460723
<details>
<summary>TPC-H: <b>Total hot run time: 34575 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit badb955dee391a9cf9e9111b6ff8f5ebdf73bb27, data reload: false
------ Round 1 ----------------------------------
q1 17614 4979 4896 4896
q2 2054 328 203 203
q3 10282 1336 718 718
q4 10226 913 372 372
q5 7502 2372 2363 2363
q6 182 175 140 140
q7 942 793 642 642
q8 9346 1349 1063 1063
q9 7106 5285 5266 5266
q10 6903 2224 1833 1833
q11 503 330 303 303
q12 367 372 219 219
q13 17802 3624 3126 3126
q14 237 238 216 216
q15 587 512 508 508
q16 1061 1011 958 958
q17 581 701 527 527
q18 7647 7118 7060 7060
q19 1097 945 574 574
q20 355 351 230 230
q21 3544 3209 2422 2422
q22 1002 1001 936 936
Total cold run time: 106940 ms
Total hot run time: 34575 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4969 4900 4895 4895
q2 349 397 311 311
q3 2148 2710 2292 2292
q4 1323 1747 1282 1282
q5 4224 4368 4519 4368
q6 226 183 136 136
q7 2028 2057 1808 1808
q8 2724 2538 2474 2474
q9 7509 7397 7577 7397
q10 3063 3267 2792 2792
q11 595 541 519 519
q12 703 778 617 617
q13 3562 3873 3551 3551
q14 280 291 269 269
q15 550 507 514 507
q16 1116 1088 1070 1070
q17 1140 1441 1464 1441
q18 7812 7768 7625 7625
q19 854 819 826 819
q20 1968 2087 1905 1905
q21 4663 4181 4126 4126
q22 1064 1027 962 962
Total cold run time: 52870 ms
Total hot run time: 51166 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3555586986
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3555767280
<details>
<summary>TPC-DS: <b>Total hot run time: 188373 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f37442a801d36436009ff8fdb6ea6bd9a9f56a09, data reload: false
query1 1041 413 414 413
query2 6573 1692 1727 1692
query3 6755 230 222 222
query4 26026 22867 22984 22867
query5 4457 669 513 513
query6 337 234 220 220
query7 4648 505 311 311
query8 320 280 243 243
query9 8675 2935 2902 2902
query10 486 358 320 320
query11 15617 15030 14805 14805
query12 174 126 117 117
query13 1668 582 450 450
query14 10201 9197 9032 9032
query15 195 185 184 184
query16 7154 671 491 491
query17 1000 798 663 663
query18 1978 411 321 321
query19 210 199 178 178
query20 131 130 126 126
query21 216 134 116 116
query22 4016 4185 4016 4016
query23 33834 33064 32958 32958
query24 8211 2396 2375 2375
query25 633 533 460 460
query26 1241 270 167 167
query27 2759 496 362 362
query28 4430 2242 2216 2216
query29 858 656 556 556
query30 300 221 197 197
query31 910 820 755 755
query32 96 91 84 84
query33 620 417 364 364
query34 799 872 525 525
query35 829 838 777 777
query36 943 995 894 894
query37 135 128 102 102
query38 3523 3681 3501 3501
query39 1499 1442 1421 1421
query40 234 144 133 133
query41 69 67 64 64
query42 136 122 121 121
query43 461 488 467 467
query44 1257 813 802 802
query45 183 184 174 174
query46 892 992 662 662
query47 1736 1814 1704 1704
query48 411 448 335 335
query49 770 509 435 435
query50 667 690 414 414
query51 3853 4058 4082 4058
query52 119 122 113 113
query53 258 267 216 216
query54 358 356 330 330
query55 98 98 99 98
query56 378 377 363 363
query57 1163 1189 1114 1114
query58 313 295 289 289
query59 2530 2604 2540 2540
query60 384 386 362 362
query61 197 184 196 184
query62 786 716 693 693
query63 233 201 205 201
query64 4675 1260 871 871
query65 3998 3915 3934 3915
query66 1182 454 348 348
query67 15435 15186 14886 14886
query68 8266 956 642 642
query69 491 334 299 299
query70 1314 1312 1304 1304
query71 470 350 349 349
query72 6052 4882 4830 4830
query73 672 578 371 371
query74 8896 8835 8932 8835
query75 3875 3222 2777 2777
query76 3725 1163 768 768
query77 812 432 338 338
query78 9573 9433 8821 8821
query79 2397 904 621 621
query80 644 607 537 537
query81 514 260 226 226
query82 510 168 142 142
query83 262 272 249 249
query84 301 117 91 91
query85 945 493 455 455
query86 396 323 299 299
query87 3696 3686 3602 3602
query88 3845 2244 2237 2237
query89 378 321 304 304
query90 1880 240 230 230
query91 166 174 146 146
query92 93 78 74 74
query93 1862 1011 691 691
query94 728 428 332 332
query95 448 344 347 344
query96 481 570 283 283
query97 2898 2967 2874 2874
query98 262 225 215 215
query99 1620 1404 1279 1279
Total cold run time: 274746 ms
Total hot run time: 188373 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #58166:
URL: https://github.com/apache/doris/pull/58166
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3569264183
<details>
<summary>TPC-H: <b>Total hot run time: 34900 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c6b4ca7ab1660349b505067613960c32684a28c, data reload: false
------ Round 1 ----------------------------------
q1 17612 5129 4900 4900
q2 2014 320 208 208
q3 10247 1277 736 736
q4 10224 905 373 373
q5 7543 2348 2323 2323
q6 182 173 141 141
q7 920 797 628 628
q8 9361 1338 1059 1059
q9 7053 5490 5497 5490
q10 6872 2250 1813 1813
q11 495 319 269 269
q12 338 361 227 227
q13 17802 3649 3013 3013
q14 224 233 209 209
q15 579 499 516 499
q16 1020 982 943 943
q17 594 856 367 367
q18 7338 7370 7707 7370
q19 1539 989 603 603
q20 356 364 237 237
q21 4012 3352 2434 2434
q22 1125 1058 1071 1058
Total cold run time: 107450 ms
Total hot run time: 34900 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5291 5151 5088 5088
q2 328 399 314 314
q3 2390 2938 2483 2483
q4 1426 1882 1368 1368
q5 4527 4466 4413 4413
q6 211 163 130 130
q7 1997 1941 1801 1801
q8 2593 2490 2505 2490
q9 7596 7619 7404 7404
q10 2866 3073 2653 2653
q11 563 518 478 478
q12 616 697 585 585
q13 3275 3686 3024 3024
q14 259 301 258 258
q15 537 508 482 482
q16 1012 1031 999 999
q17 1096 1455 1322 1322
q18 7313 7157 7069 7069
q19 743 762 797 762
q20 1906 1934 1860 1860
q21 4778 4517 4224 4224
q22 1086 1035 980 980
Total cold run time: 52409 ms
Total hot run time: 50187 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3569733044
# FE Regression Coverage Report
Increment line coverage ` 84.75% (100/118)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58166_6c6b4ca7ab1660349b505067613960c32684a28c_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58166_6c6b4ca7ab1660349b505067613960c32684a28c_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3574014754
# FE Regression Coverage Report
Increment line coverage ` 84.75% (100/118)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58166_f988f83353e0a7e0ee3d3447cef9e208469e17e8_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58166_f988f83353e0a7e0ee3d3447cef9e208469e17e8_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3573742527
<details>
<summary>TPC-DS: <b>Total hot run time: 184561 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f988f83353e0a7e0ee3d3447cef9e208469e17e8, data reload: false
query1 1099 408 393 393
query2 6594 1605 1614 1605
query3 6757 222 219 219
query4 25099 22692 22784 22692
query5 4391 605 470 470
query6 331 237 226 226
query7 4668 490 302 302
query8 306 260 257 257
query9 8752 2627 2620 2620
query10 544 352 317 317
query11 15422 14900 14621 14621
query12 226 118 114 114
query13 1699 585 462 462
query14 10640 8968 8825 8825
query15 211 205 186 186
query16 7251 682 524 524
query17 1268 724 613 613
query18 1988 422 322 322
query19 209 201 181 181
query20 125 122 119 119
query21 242 140 114 114
query22 3940 3991 3781 3781
query23 33051 32033 32078 32033
query24 8497 2430 2410 2410
query25 618 526 457 457
query26 1236 273 160 160
query27 2770 508 352 352
query28 4402 2190 2172 2172
query29 825 612 478 478
query30 332 235 206 206
query31 866 742 636 636
query32 87 73 68 68
query33 582 367 344 344
query34 787 864 535 535
query35 815 846 776 776
query36 930 959 866 866
query37 139 106 84 84
query38 3322 3367 3271 3271
query39 1459 1416 1405 1405
query40 233 128 119 119
query41 65 64 63 63
query42 127 115 114 114
query43 467 483 446 446
query44 1242 767 755 755
query45 211 197 188 188
query46 893 988 652 652
query47 1689 1757 1658 1658
query48 398 433 329 329
query49 795 498 434 434
query50 641 682 410 410
query51 3892 3930 3958 3930
query52 111 116 100 100
query53 241 263 187 187
query54 308 293 273 273
query55 86 86 82 82
query56 324 310 329 310
query57 1152 1187 1102 1102
query58 284 277 274 274
query59 2438 2507 2414 2414
query60 349 362 336 336
query61 167 157 162 157
query62 799 718 667 667
query63 226 190 208 190
query64 4591 1234 890 890
query65 4056 3953 3999 3953
query66 1170 435 324 324
query67 15267 15266 14848 14848
query68 4710 914 658 658
query69 525 343 305 305
query70 1300 1240 1157 1157
query71 399 341 316 316
query72 6169 5090 5072 5072
query73 653 581 378 378
query74 8534 8503 8615 8503
query75 3320 3364 2848 2848
query76 3257 1120 725 725
query77 516 410 342 342
query78 9747 9747 8848 8848
query79 2286 833 603 603
query80 1658 579 521 521
query81 567 271 241 241
query82 488 160 131 131
query83 349 264 264 264
query84 253 120 103 103
query85 935 490 439 439
query86 470 301 306 301
query87 3487 3505 3330 3330
query88 2827 2274 2278 2274
query89 388 330 293 293
query90 1838 224 220 220
query91 180 172 145 145
query92 80 65 59 59
query93 2087 975 656 656
query94 791 443 339 339
query95 491 407 401 401
query96 496 576 278 278
query97 2888 2963 2872 2872
query98 231 212 211 211
query99 1276 1397 1307 1307
Total cold run time: 267898 ms
Total hot run time: 184561 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3573721433
<details>
<summary>TPC-H: <b>Total hot run time: 33961 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f988f83353e0a7e0ee3d3447cef9e208469e17e8, data reload: false
------ Round 1 ----------------------------------
q1 17731 5081 4926 4926
q2 2134 309 212 212
q3 10141 1296 711 711
q4 10230 948 367 367
q5 7554 2285 2411 2285
q6 188 166 133 133
q7 930 821 636 636
q8 9587 1374 1061 1061
q9 7161 5357 5274 5274
q10 6866 2242 1795 1795
q11 496 313 289 289
q12 343 354 218 218
q13 17774 3688 3027 3027
q14 232 238 233 233
q15 589 517 511 511
q16 1042 1020 958 958
q17 591 839 358 358
q18 7423 7175 6998 6998
q19 1411 959 530 530
q20 351 343 233 233
q21 3671 3141 2272 2272
q22 1047 1019 934 934
Total cold run time: 107492 ms
Total hot run time: 33961 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5235 4996 4925 4925
q2 336 396 297 297
q3 2168 2751 2271 2271
q4 1363 1756 1334 1334
q5 4230 4551 4519 4519
q6 219 176 137 137
q7 2041 1914 1862 1862
q8 2631 2541 2584 2541
q9 7510 7562 7488 7488
q10 2983 3332 2798 2798
q11 625 540 494 494
q12 703 779 615 615
q13 3470 3925 3514 3514
q14 283 316 278 278
q15 543 499 512 499
q16 1121 1108 1082 1082
q17 1160 1560 1436 1436
q18 7805 7639 7762 7639
q19 807 858 991 858
q20 2013 2112 1835 1835
q21 4739 4286 4281 4281
q22 1079 1090 1016 1016
Total cold run time: 53064 ms
Total hot run time: 51719 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3592479786
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3592459875
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #58166:
URL: https://github.com/apache/doris/pull/58166#discussion_r2544990667
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSTransaction.java:
##########
@@ -268,10 +268,52 @@ public void finishInsertTable(NameMapping nameMapping) {
insertExistsPartitions.add(Pair.of(pu, hivePartitionStatistics));
break;
case NEW:
+ // Check if partition really exists in HMS (may be cache miss in Doris)
+ List<String> partitionValues = HiveUtil.toPartitionValues(pu.getName());
+ boolean existsInHMS = false;
+ try {
+ Partition hmsPartition = hiveOps.getClient().getPartition(
+ nameMapping.getRemoteDbName(),
+ nameMapping.getRemoteTblName(),
+ partitionValues);
+ existsInHMS = (hmsPartition != null);
+ } catch (Exception e) {
+ // Partition not found in HMS, treat as truly new
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Partition {} not found in HMS, will create it", pu.getName());
+ }
+ }
+
+ if (existsInHMS) {
+ // Partition exists in HMS but not in Doris cache
+ // Treat as APPEND instead of NEW to avoid creation error
+ LOG.info("Partition {} already exists in HMS (Doris cache miss), treating as APPEND",
+ pu.getName());
+ insertExistsPartitions.add(Pair.of(pu, hivePartitionStatistics));
+ } else {
Review Comment:
this `else` block is same as `case OVERWRITE`.
extract a method for it.
Take care of `addPartition()` and `dropPartition()`
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -191,26 +198,57 @@ private void setCompressType(THiveTableSink tSink, TFileFormatType formatType) {
}
private void setPartitionValues(THiveTableSink tSink) throws AnalysisException {
+ if (ConnectContext.get().getExecutor() != null) {
+ ConnectContext.get().getExecutor().getSummaryProfile().setSinkGetPartitionsStartTime();
+ }
+
List<THivePartition> partitions = new ArrayList<>();
- List<org.apache.hadoop.hive.metastore.api.Partition> hivePartitions =
- ((HMSExternalCatalog) targetTable.getCatalog())
- .getClient().listPartitions(targetTable.getRemoteDbName(), targetTable.getRemoteName());
- for (org.apache.hadoop.hive.metastore.api.Partition partition : hivePartitions) {
+
+ // Get partitions from cache instead of HMS client (similar to HiveScanNode)
+ HiveMetaStoreCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
+ .getMetaStoreCache((HMSExternalCatalog) targetTable.getCatalog());
+
+ List<HivePartition> hivePartitions;
+ if (targetTable.isPartitionedTable()) {
+ // Get all partition values from cache
+ List<Type> partitionColumnTypes = targetTable.getPartitionColumnTypes(
+ MvccUtil.getSnapshotFromContext(targetTable));
+ HiveMetaStoreCache.HivePartitionValues partitionValues =
Review Comment:
Rebase and use `targetTable.getHivePartitionValues()` directly
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java:
##########
@@ -561,6 +562,93 @@ public void invalidatePartitionCache(ExternalTable dorisTable, String partitionN
}
}
+ /**
+ * Selectively refreshes cache for affected partitions based on update information from BE.
+ * This method optimizes cache refresh by only invalidating specific partitions
+ * rather than the entire table cache.
+ *
+ * @param table The Hive table whose partitions were modified
+ * @param partitionUpdates List of partition updates from BE
+ */
+ public void refreshAffectedPartitions(HMSExternalTable table,
+ List<org.apache.doris.thrift.THivePartitionUpdate> partitionUpdates) {
+ if (partitionUpdates == null || partitionUpdates.isEmpty()) {
+ return;
+ }
+
+ NameMapping nameMapping = table.getOrBuildNameMapping();
+ long tableId = Util.genIdByName(nameMapping.getLocalDbName(), nameMapping.getLocalTblName());
+
+ // Classify partition updates by type
+ Set<List<String>> modifiedPartitions = new HashSet<>();
+ boolean hasNewPartitions = false;
+
+ for (org.apache.doris.thrift.THivePartitionUpdate update : partitionUpdates) {
+ List<String> partitionValues = HiveUtil.toPartitionValues(update.getName());
Review Comment:
Move this line to `case OVERWRITE:`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/HiveInsertExecutor.java:
##########
@@ -65,6 +73,19 @@ protected void doBeforeCommit() throws UserException {
HMSTransaction transaction = (HMSTransaction) transactionManager.getTransaction(txnId);
loadedRows = transaction.getUpdateCnt();
transaction.finishInsertTable(((ExternalTable) table).getOrBuildNameMapping());
+
+ // Save partition updates for cache refresh after commit
+ partitionUpdates = transaction.getHivePartitionUpdates();
+ }
+
+ @Override
+ protected void doAfterCommit() throws DdlException {
+ // For Hive tables, do selective partition refresh instead of full table refresh
+ if (partitionUpdates != null && !partitionUpdates.isEmpty()) {
+ HiveMetaStoreCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
+ .getMetaStoreCache((HMSExternalCatalog) ((HMSExternalTable) table).getCatalog());
+ cache.refreshAffectedPartitions((HMSExternalTable) table, partitionUpdates);
Review Comment:
Before, the `Env.getCurrentEnv().getRefreshManager().handleRefreshTable()` will write edit log, so that non-master FE will get the latest partition info.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java:
##########
@@ -561,6 +562,93 @@ public void invalidatePartitionCache(ExternalTable dorisTable, String partitionN
}
}
+ /**
+ * Selectively refreshes cache for affected partitions based on update information from BE.
+ * This method optimizes cache refresh by only invalidating specific partitions
+ * rather than the entire table cache.
+ *
+ * @param table The Hive table whose partitions were modified
+ * @param partitionUpdates List of partition updates from BE
+ */
+ public void refreshAffectedPartitions(HMSExternalTable table,
+ List<org.apache.doris.thrift.THivePartitionUpdate> partitionUpdates) {
+ if (partitionUpdates == null || partitionUpdates.isEmpty()) {
+ return;
+ }
+
+ NameMapping nameMapping = table.getOrBuildNameMapping();
+ long tableId = Util.genIdByName(nameMapping.getLocalDbName(), nameMapping.getLocalTblName());
+
+ // Classify partition updates by type
+ Set<List<String>> modifiedPartitions = new HashSet<>();
+ boolean hasNewPartitions = false;
+
+ for (org.apache.doris.thrift.THivePartitionUpdate update : partitionUpdates) {
+ List<String> partitionValues = HiveUtil.toPartitionValues(update.getName());
+
+ switch (update.getUpdateMode()) {
+ case APPEND:
+ case OVERWRITE:
+ modifiedPartitions.add(partitionValues);
+ break;
+ case NEW:
+ hasNewPartitions = true;
+ break;
+ default:
+ LOG.warn("Unknown update mode {} for partition {}",
+ update.getUpdateMode(), update.getName());
+ break;
+ }
+ }
+
+ // Refresh file cache for modified partitions
+ if (!modifiedPartitions.isEmpty()) {
+ invalidateFileCache(nameMapping, tableId, modifiedPartitions);
+ }
+
+ // Refresh partition values cache if new partitions were created
+ if (hasNewPartitions) {
+ invalidatePartitionValuesCache(nameMapping);
Review Comment:
Can we just modify the cache value instead of invalidate it all?
Otherwise, for insert with new partition, it always invalidate all cache
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -191,26 +198,57 @@ private void setCompressType(THiveTableSink tSink, TFileFormatType formatType) {
}
private void setPartitionValues(THiveTableSink tSink) throws AnalysisException {
+ if (ConnectContext.get().getExecutor() != null) {
+ ConnectContext.get().getExecutor().getSummaryProfile().setSinkGetPartitionsStartTime();
+ }
+
List<THivePartition> partitions = new ArrayList<>();
- List<org.apache.hadoop.hive.metastore.api.Partition> hivePartitions =
- ((HMSExternalCatalog) targetTable.getCatalog())
- .getClient().listPartitions(targetTable.getRemoteDbName(), targetTable.getRemoteName());
- for (org.apache.hadoop.hive.metastore.api.Partition partition : hivePartitions) {
+
+ // Get partitions from cache instead of HMS client (similar to HiveScanNode)
+ HiveMetaStoreCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
Review Comment:
Move this line to `if (targetTable.isPartitionedTable()) `
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java:
##########
@@ -561,6 +562,93 @@ public void invalidatePartitionCache(ExternalTable dorisTable, String partitionN
}
}
+ /**
+ * Selectively refreshes cache for affected partitions based on update information from BE.
+ * This method optimizes cache refresh by only invalidating specific partitions
+ * rather than the entire table cache.
+ *
+ * @param table The Hive table whose partitions were modified
+ * @param partitionUpdates List of partition updates from BE
+ */
+ public void refreshAffectedPartitions(HMSExternalTable table,
+ List<org.apache.doris.thrift.THivePartitionUpdate> partitionUpdates) {
+ if (partitionUpdates == null || partitionUpdates.isEmpty()) {
+ return;
+ }
+
+ NameMapping nameMapping = table.getOrBuildNameMapping();
Review Comment:
Move these 2 lines right before `if (!modifiedPartitions.isEmpty()) {`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3555587332
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3569322408
<details>
<summary>ClickBench: <b>Total hot run time: 27.39 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c6b4ca7ab1660349b505067613960c32684a28c, data reload: false
query1 0.06 0.05 0.04
query2 0.09 0.05 0.05
query3 0.26 0.09 0.08
query4 1.61 0.12 0.11
query5 0.27 0.25 0.26
query6 1.17 0.64 0.64
query7 0.03 0.02 0.02
query8 0.05 0.04 0.04
query9 0.57 0.53 0.52
query10 0.58 0.57 0.58
query11 0.16 0.11 0.11
query12 0.14 0.11 0.13
query13 0.61 0.61 0.62
query14 1.00 0.99 0.99
query15 0.85 0.83 0.83
query16 0.38 0.38 0.40
query17 0.99 1.02 1.03
query18 0.21 0.20 0.20
query19 1.91 1.78 1.80
query20 0.02 0.01 0.02
query21 15.47 0.19 0.14
query22 5.03 0.08 0.05
query23 15.66 0.26 0.10
query24 2.61 0.89 0.31
query25 0.07 0.07 0.06
query26 0.13 0.14 0.13
query27 0.07 0.06 0.05
query28 4.08 1.15 0.93
query29 12.56 3.93 3.29
query30 0.28 0.14 0.12
query31 2.82 0.58 0.38
query32 3.24 0.56 0.47
query33 2.98 3.04 3.03
query34 15.80 5.18 4.54
query35 4.53 4.57 4.60
query36 0.66 0.49 0.49
query37 0.10 0.06 0.07
query38 0.06 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 97.49 s
Total hot run time: 27.39 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3585130714
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3584715065
<details>
<summary>TPC-H: <b>Total hot run time: 35248 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 992bca6e7a2be03c9ca5d5a8c575bce69f0a4c33, data reload: false
------ Round 1 ----------------------------------
q1 17624 5100 4942 4942
q2 2078 307 207 207
q3 10229 1279 734 734
q4 10221 909 374 374
q5 7524 2272 2395 2272
q6 185 174 144 144
q7 932 806 665 665
q8 9381 1395 1201 1201
q9 7236 5408 5367 5367
q10 6920 2239 1828 1828
q11 508 310 289 289
q12 374 371 232 232
q13 17788 3723 3052 3052
q14 242 235 215 215
q15 577 527 514 514
q16 1045 1058 979 979
q17 610 876 383 383
q18 7341 7222 7001 7001
q19 1210 950 586 586
q20 369 346 235 235
q21 4112 3336 3079 3079
q22 1017 1009 949 949
Total cold run time: 107523 ms
Total hot run time: 35248 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5087 5055 5021 5021
q2 328 387 315 315
q3 2200 2723 2314 2314
q4 1346 1784 1316 1316
q5 4357 4525 4545 4525
q6 202 183 143 143
q7 2083 2032 1883 1883
q8 2666 2689 2578 2578
q9 7555 7426 7586 7426
q10 3061 3262 2862 2862
q11 598 528 494 494
q12 743 807 622 622
q13 3576 4193 3235 3235
q14 282 299 265 265
q15 531 517 523 517
q16 1135 1160 1160 1160
q17 1217 1606 1429 1429
q18 7938 7754 7520 7520
q19 877 942 1079 942
q20 1991 2022 1836 1836
q21 4809 4228 4388 4228
q22 1068 1033 1028 1028
Total cold run time: 53650 ms
Total hot run time: 51659 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3585509045
<details>
<summary>TPC-DS: <b>Total hot run time: 185029 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit badb955dee391a9cf9e9111b6ff8f5ebdf73bb27, data reload: false
query1 1025 418 397 397
query2 6585 1605 1532 1532
query3 6759 236 230 230
query4 25286 23188 23071 23071
query5 4675 622 483 483
query6 332 242 225 225
query7 4668 503 307 307
query8 309 276 254 254
query9 8748 2615 2606 2606
query10 515 353 329 329
query11 15503 15383 14802 14802
query12 180 122 117 117
query13 1694 595 450 450
query14 10952 8885 8921 8885
query15 213 202 183 183
query16 7531 716 530 530
query17 1251 778 649 649
query18 2043 435 344 344
query19 219 214 187 187
query20 135 126 121 121
query21 225 138 117 117
query22 3856 3887 3833 3833
query23 32831 32125 32014 32014
query24 8479 2431 2437 2431
query25 627 552 502 502
query26 1048 300 181 181
query27 2679 492 363 363
query28 4387 2147 2136 2136
query29 765 660 527 527
query30 311 246 217 217
query31 843 703 650 650
query32 92 77 82 77
query33 612 402 361 361
query34 819 862 546 546
query35 821 878 741 741
query36 875 927 826 826
query37 125 110 91 91
query38 3304 3340 3242 3242
query39 1467 1412 1396 1396
query40 237 132 115 115
query41 67 61 66 61
query42 125 113 112 112
query43 452 456 424 424
query44 1303 755 760 755
query45 197 192 184 184
query46 870 999 641 641
query47 1671 1697 1654 1654
query48 399 438 334 334
query49 743 488 422 422
query50 676 688 421 421
query51 3899 4015 3849 3849
query52 116 113 106 106
query53 239 271 204 204
query54 339 298 277 277
query55 102 105 94 94
query56 340 339 323 323
query57 1131 1150 1078 1078
query58 290 274 276 274
query59 2293 2363 2274 2274
query60 355 349 336 336
query61 162 162 165 162
query62 782 713 642 642
query63 233 197 191 191
query64 3908 1238 893 893
query65 4020 4001 3962 3962
query66 1026 434 330 330
query67 15053 15009 14894 14894
query68 8437 958 624 624
query69 551 342 319 319
query70 1317 1272 1165 1165
query71 482 350 307 307
query72 5854 4908 4933 4908
query73 705 587 345 345
query74 8963 8808 8839 8808
query75 3972 3317 2806 2806
query76 3654 1126 722 722
query77 802 410 319 319
query78 9469 9665 8847 8847
query79 2008 846 573 573
query80 628 576 491 491
query81 491 274 242 242
query82 482 166 128 128
query83 281 257 246 246
query84 258 119 97 97
query85 987 505 457 457
query86 340 283 301 283
query87 3497 3481 3372 3372
query88 3753 2251 2245 2245
query89 391 326 304 304
query90 1968 224 215 215
query91 177 171 144 144
query92 83 73 63 63
query93 1320 964 657 657
query94 686 439 345 345
query95 507 416 408 408
query96 521 555 285 285
query97 2904 2929 2858 2858
query98 237 220 214 214
query99 1341 1418 1267 1267
Total cold run time: 270883 ms
Total hot run time: 185029 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3592487748
<details>
<summary>ClickBench: <b>Total hot run time: 27.2 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 069ef57cb63b9a043fa2a08fcf29006c4e147c38, data reload: false
query1 0.06 0.05 0.05
query2 0.10 0.05 0.05
query3 0.25 0.08 0.08
query4 1.61 0.11 0.11
query5 0.27 0.24 0.26
query6 1.16 0.65 0.63
query7 0.03 0.03 0.03
query8 0.06 0.05 0.05
query9 0.58 0.51 0.50
query10 0.56 0.56 0.55
query11 0.16 0.11 0.11
query12 0.15 0.12 0.12
query13 0.61 0.62 0.60
query14 0.99 0.99 0.98
query15 0.80 0.79 0.79
query16 0.42 0.39 0.40
query17 0.96 1.02 1.01
query18 0.25 0.24 0.22
query19 1.83 1.87 1.77
query20 0.01 0.01 0.01
query21 15.47 0.26 0.14
query22 4.92 0.06 0.05
query23 16.01 0.26 0.10
query24 1.72 0.25 0.36
query25 0.10 0.06 0.05
query26 0.15 0.14 0.13
query27 0.06 0.06 0.05
query28 3.92 1.24 1.03
query29 12.65 3.91 3.24
query30 0.28 0.13 0.11
query31 2.81 0.60 0.40
query32 3.23 0.56 0.47
query33 3.05 3.06 3.02
query34 16.60 5.17 4.50
query35 4.61 4.57 4.55
query36 0.65 0.49 0.48
query37 0.10 0.07 0.07
query38 0.07 0.05 0.03
query39 0.05 0.03 0.03
query40 0.18 0.13 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.02
query43 0.05 0.04 0.03
Total cold run time: 97.66 s
Total hot run time: 27.2 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](hive) Speed up Hive insert on partition tables using cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58166:
URL: https://github.com/apache/doris/pull/58166#issuecomment-3592479802
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org