You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "TangSiyang2001 (via GitHub)" <gi...@apache.org> on 2023/08/11 08:59:43 UTC
[GitHub] [doris] TangSiyang2001 opened a new pull request, #22889: [enhancement-wip](label) support label num threshold for bulk load, routine load and sync
TangSiyang2001 opened a new pull request, #22889:
URL: https://github.com/apache/doris/pull/22889
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1887071790
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1879921119
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1879927060
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.49 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.1 seconds inserted 10000000 Rows, about 369K ops/s
storage size: 17184275139 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880081841
<details>
<summary>TPC-DS: <b>Total hot run time: 184642 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46e5ff2b247fcc8f5366baed918a8a8824b004fd, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 908 345 330 330
query2 5692 2100 2036 2036
query3 6230 220 218 218
query4 26200 22551 22416 22416
query5 5154 631 613 613
query6 327 221 224 221
query7 4575 303 307 303
query8 262 224 227 224
query9 8350 2974 2967 2967
query10 568 339 327 327
query11 15781 15614 15745 15614
query12 177 101 98 98
query13 1641 355 335 335
query14 12550 7611 7612 7611
query15 253 219 211 211
query16 6263 337 335 335
query17 1574 520 522 520
query18 1942 310 311 310
query19 275 163 167 163
query20 106 103 103 103
query21 186 95 91 91
query22 5155 4819 4886 4819
query23 32413 31332 31294 31294
query24 11322 2995 2952 2952
query25 608 393 360 360
query26 1813 161 163 161
query27 2751 314 314 314
query28 6293 2109 2128 2109
query29 1686 426 417 417
query30 276 145 150 145
query31 1054 822 816 816
query32 116 86 86 86
query33 788 328 353 328
query34 849 478 484 478
query35 959 881 878 878
query36 1382 1313 1303 1303
query37 183 94 94 94
query38 3559 3413 3334 3334
query39 1379 1293 1295 1293
query40 287 91 89 89
query41 39 39 35 35
query42 116 103 110 103
query43 546 541 540 540
query44 1132 813 800 800
query45 212 201 205 201
query46 1031 683 710 683
query47 1677 1634 1594 1594
query48 380 286 287 286
query49 1033 306 310 306
query50 796 337 326 326
query51 5483 5269 5261 5261
query52 123 94 98 94
query53 229 164 163 163
query54 1131 679 674 674
query55 106 100 98 98
query56 249 246 259 246
query57 1038 958 998 958
query58 297 293 289 289
query59 3156 2840 2884 2840
query60 303 298 295 295
query61 154 153 150 150
query62 596 481 492 481
query63 182 163 165 163
query64 5228 1711 1703 1703
query65 3414 3334 3351 3334
query66 1225 386 384 384
query67 15784 15540 15355 15355
query68 11454 545 561 545
query69 637 343 330 330
query70 1770 1607 1630 1607
query71 623 293 284 284
query72 5336 3453 3478 3453
query73 2011 347 332 332
query74 6984 6487 6477 6477
query75 5582 2327 2274 2274
query76 6705 1219 1226 1219
query77 928 354 333 333
query78 9093 8723 8607 8607
query79 1060 530 531 530
query80 582 354 353 353
query81 482 214 215 214
query82 227 104 106 104
query83 189 145 145 145
query84 258 59 59 59
query85 957 280 274 274
query86 441 419 438 419
query87 3658 3451 3408 3408
query88 3065 2498 2496 2496
query89 367 287 274 274
query90 2038 256 254 254
query91 130 99 99 99
query92 80 72 73 72
query93 1129 436 483 436
query94 936 263 250 250
query95 562 505 505 505
query96 644 330 348 330
query97 4326 4297 4202 4202
query98 252 223 218 218
query99 1143 861 867 861
Total cold run time: 291196 ms
Total hot run time: 184642 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1887145786
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 831c0da36d3e4ce0fb2e06d74ee830af6ba3c261 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.1 seconds inserted 10000000 Rows, about 763K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1891207842
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880101176
<details>
<summary>TPC-DS: <b>Total hot run time: 183780 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f6172ac91ce5a6e8a8392d6f3551ea533e152a60, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 927 338 332 332
query2 6576 2033 1997 1997
query3 6711 223 217 217
query4 26235 22588 22210 22210
query5 5856 609 629 609
query6 317 201 198 198
query7 4644 296 314 296
query8 244 232 217 217
query9 8749 2975 2909 2909
query10 514 351 338 338
query11 16268 15554 15686 15554
query12 157 95 98 95
query13 1677 326 332 326
query14 12771 7577 7578 7577
query15 263 227 218 218
query16 6404 334 333 333
query17 1692 507 512 507
query18 1974 316 305 305
query19 237 171 162 162
query20 118 101 116 101
query21 191 93 98 93
query22 4915 4829 4632 4632
query23 32531 31293 31597 31293
query24 12394 2957 2895 2895
query25 598 361 360 360
query26 1799 160 171 160
query27 2930 314 311 311
query28 6948 2135 2132 2132
query29 2057 423 405 405
query30 283 145 143 143
query31 1022 849 811 811
query32 133 75 72 72
query33 846 333 334 333
query34 892 463 489 463
query35 1001 854 858 854
query36 1357 1245 1231 1231
query37 197 89 98 89
query38 3515 3318 3301 3301
query39 1360 1316 1302 1302
query40 306 91 87 87
query41 39 37 37 37
query42 108 101 105 101
query43 623 533 525 525
query44 1276 797 808 797
query45 217 213 203 203
query46 1076 676 698 676
query47 1693 1609 1673 1609
query48 371 283 278 278
query49 1214 317 305 305
query50 797 360 338 338
query51 5369 5426 5401 5401
query52 107 103 108 103
query53 229 163 157 157
query54 1525 629 653 629
query55 105 97 94 94
query56 291 248 256 248
query57 1085 980 945 945
query58 300 284 250 250
query59 3116 2975 3091 2975
query60 320 287 263 263
query61 147 159 150 150
query62 599 474 474 474
query63 193 180 162 162
query64 6035 1721 1747 1721
query65 3408 3325 3342 3325
query66 1382 372 385 372
query67 15804 15254 15576 15254
query68 13239 553 527 527
query69 592 324 319 319
query70 1776 1628 1604 1604
query71 572 293 292 292
query72 5327 3455 3493 3455
query73 3024 342 355 342
query74 7019 6544 6504 6504
query75 5399 2298 2313 2298
query76 6343 1132 1129 1129
query77 788 333 305 305
query78 9115 8622 8641 8622
query79 1600 525 531 525
query80 543 349 357 349
query81 477 213 212 212
query82 238 105 107 105
query83 180 141 143 141
query84 262 55 58 55
query85 931 273 258 258
query86 441 425 446 425
query87 3611 3516 3461 3461
query88 3044 2480 2529 2480
query89 364 289 288 288
query90 1925 246 251 246
query91 125 96 97 96
query92 93 80 74 74
query93 1263 496 476 476
query94 923 246 250 246
query95 559 507 488 488
query96 632 357 337 337
query97 4372 4198 4222 4198
query98 248 220 219 219
query99 1143 844 861 844
Total cold run time: 301206 ms
Total hot run time: 183780 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1887081028
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880103068
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.34 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 27.7 seconds inserted 10000000 Rows, about 361K ops/s
storage size: 17188493128 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement-wip](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1752332543
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.6 seconds
stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162009688 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #22889:
URL: https://github.com/apache/doris/pull/22889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #22889: [enhancement-wip](label) support label num threshold for bulk load, routine load and sync
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1676193194
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1879726450
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.59 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.0 seconds inserted 10000000 Rows, about 370K ops/s
storage size: 17183865526 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880082098
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.81 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.8 seconds inserted 10000000 Rows, about 359K ops/s
storage size: 17184063866 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1887129985
<details>
<summary>TPC-DS: <b>Total hot run time: 179423 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 831c0da36d3e4ce0fb2e06d74ee830af6ba3c261, data reload: false
query1 915 328 327 327
query2 6739 1900 1901 1900
query3 6719 206 210 206
query4 26132 22119 21994 21994
query5 5447 494 516 494
query6 272 186 197 186
query7 4632 282 284 282
query8 225 204 205 204
query9 9155 2754 2793 2754
query10 550 242 241 241
query11 16234 15414 15243 15243
query12 122 72 72 72
query13 1694 401 408 401
query14 11739 7364 7287 7287
query15 230 187 198 187
query16 6559 272 248 248
query17 1952 494 474 474
query18 1938 276 265 265
query19 285 145 154 145
query20 82 77 72 72
query21 197 129 128 128
query22 5020 4853 4676 4676
query23 31815 30804 30932 30804
query24 12073 2866 2854 2854
query25 552 330 320 320
query26 1748 156 156 156
query27 2828 279 287 279
query28 6930 1853 1822 1822
query29 2077 695 656 656
query30 285 137 142 137
query31 979 776 763 763
query32 96 63 56 56
query33 720 241 264 241
query34 1007 466 477 466
query35 881 733 750 733
query36 1375 1266 1233 1233
query37 108 66 68 66
query38 3392 3271 3251 3251
query39 1335 1264 1275 1264
query40 312 100 95 95
query41 39 34 36 34
query42 101 95 95 95
query43 541 509 521 509
query44 1089 705 722 705
query45 208 180 181 180
query46 1061 654 673 654
query47 1684 1540 1572 1540
query48 418 322 330 322
query49 1283 308 316 308
query50 707 323 330 323
query51 5262 5139 5253 5139
query52 91 98 88 88
query53 380 291 302 291
query54 2067 1604 1622 1604
query55 98 84 94 84
query56 212 201 194 194
query57 1042 989 942 942
query58 219 197 194 194
query59 2730 2608 2546 2546
query60 236 225 233 225
query61 87 80 82 80
query62 644 419 385 385
query63 304 285 299 285
query64 5734 1656 1637 1637
query65 3350 3259 3264 3259
query66 1330 330 317 317
query67 15558 15608 15149 15149
query68 10048 518 524 518
query69 654 394 396 394
query70 1712 1542 1571 1542
query71 487 245 245 245
query72 4671 2797 2826 2797
query73 2002 313 312 312
query74 6977 6410 6450 6410
query75 4710 2357 2328 2328
query76 6350 1083 1049 1049
query77 657 265 292 265
query78 9595 8662 8702 8662
query79 1022 519 511 511
query80 506 338 338 338
query81 444 207 206 206
query82 228 91 88 88
query83 150 124 130 124
query84 280 73 79 73
query85 1006 337 334 334
query86 432 352 397 352
query87 3506 3425 3340 3340
query88 3217 2239 2232 2232
query89 475 384 387 384
query90 1836 201 204 201
query91 154 133 132 132
query92 65 53 55 53
query93 952 449 417 417
query94 1133 180 177 177
query95 543 482 471 471
query96 622 314 326 314
query97 4263 4142 4197 4142
query98 206 193 186 186
query99 1006 756 755 755
Total cold run time: 291735 ms
Total hot run time: 179423 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement-wip](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1752315998
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880091275
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1879718045
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #22889:
URL: https://github.com/apache/doris/pull/22889#discussion_r1439181237
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadManager.java:
##########
@@ -90,7 +92,7 @@
public class LoadManager implements Writable {
private static final Logger LOG = LogManager.getLogger(LoadManager.class);
- private Map<Long, LoadJob> idToLoadJob = Maps.newConcurrentMap();
+ private Map<Long, LoadJob> idToLoadJob = Collections.synchronizedMap(Maps.newLinkedHashMap());
private Map<Long, Map<String, List<LoadJob>>> dbIdToLabelToLoadJobs = Maps.newConcurrentMap();
Review Comment:
Why replace concurrentMap with synchronizedMap ? what is the advantages and disadvantages?
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadManager.java:
##########
@@ -399,13 +401,26 @@ public long getLoadJobNum(JobState jobState, EtlJobType jobType) {
**/
public void removeOldLoadJob() {
long currentTimeMs = System.currentTimeMillis();
+ removeLoadJobIf(job -> job.isExpired(currentTimeMs), -1);
+ }
+
+ /**
+ * Remove completed jobs if total job num exceed Config.label_num_threshold
+ */
+ public void removeOverLimitLoadJob() {
+ if (idToLoadJob.size() <= Config.label_num_threshold) {
+ return;
+ }
+ removeLoadJobIf(LoadJob::isCompleted, Config.label_num_threshold);
+ }
+ private void removeLoadJobIf(Predicate<LoadJob> pred, int numLimit) {
writeLock();
try {
Iterator<Map.Entry<Long, LoadJob>> iter = idToLoadJob.entrySet().iterator();
- while (iter.hasNext()) {
+ while (iter.hasNext() && idToLoadJob.size() > numLimit) {
Review Comment:
do we need think about sorting by finishTime ? I think we keep the last recently label_num_threshold final job maybe better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880474847
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.22 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.6 seconds inserted 10000000 Rows, about 362K ops/s
storage size: 17183082520 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1887221291
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.62 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.6 seconds inserted 10000000 Rows, about 462K ops/s
storage size: 17183831020 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1837001690
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #22889:
URL: https://github.com/apache/doris/pull/22889#discussion_r1434811587
##########
fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncJobManager.java:
##########
@@ -283,13 +286,28 @@ public void updateNeedSchedule() throws UserException {
// Stopped jobs will be removed after Config.label_keep_max_second.
public void cleanOldSyncJobs() {
LOG.debug("begin to clean old sync jobs ");
+ cleanFinishedSyncJobsIf(job -> job.isExpired(System.currentTimeMillis()), -1);
+ }
+
+ /**
+ * Remove completed jobs if total job num exceed Config.label_num_threshold
+ */
+ public void cleanOverLimitSyncJobs() {
+ if (idToSyncJob.size() <= Config.label_num_threshold) {
+ return;
+ }
+ LOG.debug("begin to clean finished sync jobs ");
+ cleanFinishedSyncJobsIf(SyncJob::isCompleted, Config.label_num_threshold);
Review Comment:
Is it better to remove oldest jobs when exceed limit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880074165
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880493552
<details>
<summary>TPC-H: <b>Total hot run time: 38276 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1ddfd7865cbd4ddbcc907c950a41b994e697318f, data reload: false
------ Round 1 ----------------------------------
q1 17635 4928 4995 4928
q2 2016 150 144 144
q3 10585 1100 1151 1100
q4 10382 783 812 783
q5 8219 2886 2921 2886
q6 209 120 122 120
q7 908 504 482 482
q8 9272 1959 1985 1959
q9 6420 6336 6330 6330
q10 8204 3066 2985 2985
q11 424 218 217 217
q12 387 217 218 217
q13 17986 3368 3356 3356
q14 249 209 215 209
q15 554 508 493 493
q16 453 392 423 392
q17 939 719 627 627
q18 7212 6611 6628 6611
q19 1554 1504 1516 1504
q20 749 304 313 304
q21 2715 2386 2316 2316
q22 357 313 314 313
Total cold run time: 107429 ms
Total hot run time: 38276 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4916 4941 4903 4903
q2 341 227 242 227
q3 3271 3287 3286 3286
q4 2089 2010 1989 1989
q5 5765 5752 5767 5752
q6 203 113 116 113
q7 2306 1919 1865 1865
q8 3349 3446 3450 3446
q9 8751 8673 8672 8672
q10 3775 3842 3863 3842
q11 547 441 451 441
q12 790 634 624 624
q13 6628 3158 3195 3158
q14 285 269 267 267
q15 546 500 504 500
q16 521 486 491 486
q17 1882 1850 1830 1830
q18 8565 8346 8162 8162
q19 1594 1634 1627 1627
q20 2193 1969 1959 1959
q21 5509 5300 5193 5193
q22 497 471 434 434
Total cold run time: 64323 ms
Total hot run time: 58776 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1887111259
<details>
<summary>TPC-H: <b>Total hot run time: 38454 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 831c0da36d3e4ce0fb2e06d74ee830af6ba3c261, data reload: false
------ Round 1 ----------------------------------
q1 18005 5026 4953 4953
q2 2039 146 127 127
q3 10678 1156 1221 1156
q4 10521 794 831 794
q5 7801 3162 3152 3152
q6 195 121 122 121
q7 871 499 488 488
q8 9259 1948 2001 1948
q9 7288 6447 6399 6399
q10 8226 3074 3052 3052
q11 412 207 210 207
q12 363 188 190 188
q13 18114 3436 3383 3383
q14 242 208 211 208
q15 553 518 509 509
q16 429 370 380 370
q17 936 511 505 505
q18 7284 6677 6638 6638
q19 1588 1345 1359 1345
q20 582 293 283 283
q21 2734 2408 2325 2325
q22 354 303 303 303
Total cold run time: 108474 ms
Total hot run time: 38454 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4954 4960 4964 4960
q2 311 200 193 193
q3 3319 3279 3291 3279
q4 2221 2225 2233 2225
q5 5783 5769 5786 5769
q6 191 115 115 115
q7 2314 1825 1837 1825
q8 3424 3510 3552 3510
q9 8851 8779 8829 8779
q10 3742 3817 3775 3775
q11 536 426 418 418
q12 783 599 610 599
q13 5594 3225 3186 3186
q14 285 256 254 254
q15 585 512 515 512
q16 494 444 459 444
q17 2030 2019 2003 2003
q18 8726 8343 8278 8278
q19 1633 1593 1593 1593
q20 2154 1908 1910 1908
q21 5967 5737 5740 5737
q22 575 466 481 466
Total cold run time: 64472 ms
Total hot run time: 59828 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22889: [enhancement-wip](label) support label num threshold for bulk load, routine load and sync
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1676201788
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.15 seconds
stream load tsv: 512 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162069355 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1879723227
<details>
<summary>TPC-H: <b>Total hot run time: 38708 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c922696130949e4707d8045b6de3ceeda1b86ea1, data reload: false
------ Round 1 ----------------------------------
q1 17708 4946 4988 4946
q2 2031 155 144 144
q3 10808 1123 1161 1123
q4 10516 821 785 785
q5 7796 2968 2945 2945
q6 204 121 123 121
q7 905 511 497 497
q8 9285 2006 1979 1979
q9 6437 6453 6406 6406
q10 8293 3075 3002 3002
q11 445 218 207 207
q12 388 233 232 232
q13 18005 3414 3398 3398
q14 234 215 213 213
q15 548 501 509 501
q16 503 404 386 386
q17 969 713 605 605
q18 7409 6747 6670 6670
q19 1564 1520 1519 1519
q20 738 345 320 320
q21 2733 2463 2399 2399
q22 363 320 310 310
Total cold run time: 107882 ms
Total hot run time: 38708 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4924 4924 4925 4924
q2 346 232 238 232
q3 3293 3307 3288 3288
q4 2120 2064 2060 2060
q5 5831 5788 5790 5788
q6 206 115 117 115
q7 2325 1900 1903 1900
q8 3421 3462 3461 3461
q9 8844 8771 8795 8771
q10 3785 3848 3877 3848
q11 561 443 444 443
q12 800 626 636 626
q13 4637 3194 3179 3179
q14 294 246 261 246
q15 543 502 499 499
q16 544 480 485 480
q17 1865 1871 1855 1855
q18 8663 8347 8322 8322
q19 1596 1648 1632 1632
q20 2194 1974 1946 1946
q21 5589 5213 5328 5213
q22 502 461 431 431
Total cold run time: 62883 ms
Total hot run time: 59259 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #22889:
URL: https://github.com/apache/doris/pull/22889#discussion_r1434810351
##########
fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java:
##########
@@ -679,19 +681,35 @@ public void processTimeoutTasks() {
// Remove old routine load jobs from idToRoutineLoadJob
// This function is called periodically.
- // Cancelled and stopped job will be remove after Configure.label_keep_max_second seconds
+ // Cancelled and stopped job will be removed after Configure.label_keep_max_second seconds
public void cleanOldRoutineLoadJobs() {
LOG.debug("begin to clean old routine load jobs ");
+ clearRoutineLoadJobIf(RoutineLoadJob::isExpired, -1);
+ }
+
+ /**
+ * Remove finished routine load jobs from idToRoutineLoadJob
+ * This function is called periodically if Config.label_num_threshold is set.
+ * Cancelled and stopped job will be removed.
+ */
+ public void cleanOverLimitRoutineLoadJobs() {
+ if (idToRoutineLoadJob.size() <= Config.label_num_threshold) {
+ return;
+ }
+ LOG.debug("begin to clean routine load jobs");
+ clearRoutineLoadJobIf(RoutineLoadJob::isFinal, Config.label_num_threshold);
Review Comment:
Is it better to remove oldest jobs when exceed limit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1872540790
I'd like to introduce a better policy for num based restriction. Eviction is better to happend after insertion, rather than in a fix interval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1887139833
<details>
<summary>ClickBench: <b>Total hot run time: 30.38 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 831c0da36d3e4ce0fb2e06d74ee830af6ba3c261, data reload: false
query1 0.06 0.05 0.05
query2 0.06 0.03 0.02
query3 0.26 0.11 0.11
query4 1.77 0.11 0.11
query5 0.53 0.52 0.53
query6 1.33 0.64 0.63
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.56 0.49 0.49
query10 0.53 0.54 0.54
query11 0.14 0.09 0.08
query12 0.11 0.09 0.09
query13 0.60 0.60 0.60
query14 0.77 0.81 0.80
query15 0.80 0.80 0.78
query16 0.36 0.35 0.35
query17 0.98 0.99 0.93
query18 0.25 0.26 0.25
query19 1.89 1.72 1.74
query20 0.01 0.01 0.01
query21 15.40 0.57 0.56
query22 2.27 2.18 2.02
query23 17.45 0.72 0.78
query24 16.05 0.57 0.59
query25 2.17 0.16 0.15
query26 0.13 0.13 0.13
query27 0.14 0.15 0.15
query28 5.69 0.84 0.79
query29 12.58 3.19 3.11
query30 0.60 0.47 0.48
query31 2.78 0.36 0.36
query32 3.36 0.49 0.48
query33 3.19 3.24 3.21
query34 15.78 4.18 4.12
query35 4.18 4.16 4.11
query36 1.11 1.04 1.05
query37 0.07 0.05 0.05
query38 0.04 0.03 0.02
query39 0.02 0.02 0.01
query40 0.16 0.14 0.14
query41 0.07 0.02 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 114.35 s
Total hot run time: 30.38 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #22889:
URL: https://github.com/apache/doris/pull/22889#discussion_r1439183568
##########
fe/fe-core/src/test/java/org/apache/doris/load/routineload/RoutineLoadManagerTest.java:
##########
@@ -801,7 +801,7 @@ public void testCleanOldRoutineLoadJobs(@Injectable RoutineLoadJob routineLoadJo
new Expectations() {
{
- routineLoadJob.needRemove();
+ routineLoadJob.isExpired();
Review Comment:
lack of lots of jobs ut test and regression test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1879926238
<details>
<summary>TPC-DS: <b>Total hot run time: 183980 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fc41b8a0b61c1bc9c00f07369017df4578fc17ad, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 944 338 335 335
query2 6578 1939 2011 1939
query3 6714 257 228 228
query4 26754 22547 22574 22547
query5 5966 598 594 594
query6 319 201 199 199
query7 4625 301 309 301
query8 268 232 212 212
query9 8829 2999 3018 2999
query10 565 319 307 307
query11 16262 15471 15612 15471
query12 175 101 104 101
query13 1653 330 329 329
query14 11991 7603 7619 7603
query15 327 226 220 220
query16 6391 343 331 331
query17 1733 533 520 520
query18 1959 312 295 295
query19 325 163 158 158
query20 99 102 100 100
query21 188 96 94 94
query22 4689 4801 4753 4753
query23 31893 31242 31650 31242
query24 12616 3015 2928 2928
query25 599 369 362 362
query26 1738 163 175 163
query27 3159 304 308 304
query28 6941 2143 2130 2130
query29 2060 429 415 415
query30 289 146 147 146
query31 1003 825 822 822
query32 139 80 83 80
query33 847 327 340 327
query34 884 482 472 472
query35 990 892 901 892
query36 1373 1239 1296 1239
query37 207 82 83 82
query38 3462 3389 3395 3389
query39 1378 1309 1318 1309
query40 304 90 90 90
query41 39 37 35 35
query42 112 101 109 101
query43 562 526 500 500
query44 1206 817 816 816
query45 214 201 211 201
query46 1089 692 717 692
query47 1771 1641 1642 1641
query48 380 289 297 289
query49 1220 311 314 311
query50 781 334 368 334
query51 5457 5249 5251 5249
query52 111 95 101 95
query53 240 166 162 162
query54 1470 675 621 621
query55 106 95 97 95
query56 254 236 271 236
query57 1040 1001 992 992
query58 311 277 268 268
query59 3055 2805 2944 2805
query60 315 293 291 291
query61 155 164 150 150
query62 565 492 475 475
query63 196 173 178 173
query64 6035 1789 1767 1767
query65 3429 3355 3357 3355
query66 1353 384 373 373
query67 15564 15205 15254 15205
query68 13384 531 551 531
query69 625 322 303 303
query70 1870 1587 1641 1587
query71 608 287 300 287
query72 5391 3482 3475 3475
query73 3145 356 335 335
query74 7103 6589 6426 6426
query75 5334 2263 2301 2263
query76 6294 1192 1259 1192
query77 820 344 337 337
query78 9056 8577 8644 8577
query79 1420 514 518 514
query80 563 353 336 336
query81 457 213 211 211
query82 238 106 108 106
query83 187 142 143 142
query84 250 60 57 57
query85 934 282 261 261
query86 430 441 438 438
query87 3581 3437 3450 3437
query88 2972 2513 2454 2454
query89 373 288 281 281
query90 1950 257 231 231
query91 127 100 104 100
query92 95 73 81 73
query93 1149 428 477 428
query94 922 231 243 231
query95 548 490 483 483
query96 649 332 327 327
query97 4316 4353 4235 4235
query98 244 245 212 212
query99 1150 850 870 850
Total cold run time: 300451 ms
Total hot run time: 183980 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880078885
<details>
<summary>TPC-H: <b>Total hot run time: 38839 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46e5ff2b247fcc8f5366baed918a8a8824b004fd, data reload: false
------ Round 1 ----------------------------------
q1 17706 4975 5001 4975
q2 2021 155 146 146
q3 10811 1098 1132 1098
q4 10495 840 781 781
q5 7784 2926 2929 2926
q6 208 122 123 122
q7 928 508 515 508
q8 9319 1994 1993 1993
q9 6431 6409 6343 6343
q10 8242 3098 3002 3002
q11 436 210 209 209
q12 385 233 227 227
q13 18095 3502 3555 3502
q14 255 220 218 218
q15 544 520 520 520
q16 511 418 412 412
q17 986 743 634 634
q18 7444 6745 6681 6681
q19 1567 1534 1516 1516
q20 721 346 341 341
q21 2726 2415 2377 2377
q22 354 308 311 308
Total cold run time: 107969 ms
Total hot run time: 38839 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4956 4929 4910 4910
q2 338 234 257 234
q3 3282 3266 3277 3266
q4 2121 1994 2016 1994
q5 5804 5824 5797 5797
q6 205 117 115 115
q7 2339 1903 1952 1903
q8 3382 3468 3467 3467
q9 8852 8797 8740 8740
q10 3813 3836 3861 3836
q11 550 441 442 441
q12 791 654 642 642
q13 5739 3193 3200 3193
q14 284 260 254 254
q15 554 510 509 509
q16 539 505 480 480
q17 1851 1866 1853 1853
q18 8621 8328 8235 8235
q19 1592 1642 1822 1642
q20 2219 1961 1941 1941
q21 5503 5299 5315 5299
q22 501 461 452 452
Total cold run time: 63836 ms
Total hot run time: 59203 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880097879
<details>
<summary>TPC-H: <b>Total hot run time: 38388 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6172ac91ce5a6e8a8392d6f3551ea533e152a60, data reload: false
------ Round 1 ----------------------------------
q1 17774 5041 4931 4931
q2 2040 158 146 146
q3 10563 1137 1116 1116
q4 10200 795 805 795
q5 7765 2959 2952 2952
q6 199 125 125 125
q7 905 516 489 489
q8 9254 1966 1965 1965
q9 6478 6355 6301 6301
q10 8253 3018 2973 2973
q11 435 210 219 210
q12 379 227 232 227
q13 17997 3399 3382 3382
q14 241 218 204 204
q15 538 495 505 495
q16 452 416 415 415
q17 931 718 603 603
q18 7204 6647 6620 6620
q19 1564 1507 1502 1502
q20 699 308 316 308
q21 2741 2334 2344 2334
q22 346 295 303 295
Total cold run time: 106958 ms
Total hot run time: 38388 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4881 4897 4872 4872
q2 337 235 233 233
q3 3289 3272 3258 3258
q4 2112 1997 2009 1997
q5 5785 5777 5762 5762
q6 202 118 116 116
q7 2272 1881 1901 1881
q8 3354 3442 3470 3442
q9 8732 8766 8716 8716
q10 3781 3834 3828 3828
q11 547 444 442 442
q12 783 646 621 621
q13 6514 3173 3149 3149
q14 284 255 252 252
q15 546 498 501 498
q16 537 500 500 500
q17 1868 1841 1848 1841
q18 8543 8324 8243 8243
q19 1607 1622 1631 1622
q20 2190 2002 1976 1976
q21 5536 5225 5217 5217
q22 489 427 454 427
Total cold run time: 64189 ms
Total hot run time: 58893 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880504227
<details>
<summary>TPC-DS: <b>Total hot run time: 183811 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1ddfd7865cbd4ddbcc907c950a41b994e697318f, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 940 341 327 327
query2 6574 2060 1936 1936
query3 6716 229 226 226
query4 27140 22417 22401 22401
query5 4795 593 602 593
query6 341 209 224 209
query7 4632 315 304 304
query8 269 242 222 222
query9 8653 2991 3009 2991
query10 620 326 310 310
query11 16344 15638 15789 15638
query12 176 94 101 94
query13 1689 352 344 344
query14 13574 7648 7574 7574
query15 271 211 222 211
query16 6524 340 334 334
query17 1692 525 512 512
query18 1967 300 291 291
query19 235 167 154 154
query20 111 106 98 98
query21 185 102 101 101
query22 4883 4798 4768 4768
query23 32265 31200 31497 31200
query24 11841 2922 2912 2912
query25 593 367 349 349
query26 1855 167 172 167
query27 2788 306 306 306
query28 7025 2176 2145 2145
query29 2075 406 403 403
query30 281 144 144 144
query31 1031 813 807 807
query32 131 85 72 72
query33 848 341 354 341
query34 864 484 473 473
query35 1008 915 894 894
query36 1363 1216 1287 1216
query37 205 84 97 84
query38 3470 3364 3374 3364
query39 1349 1307 1294 1294
query40 297 87 87 87
query41 39 39 35 35
query42 108 103 111 103
query43 542 539 515 515
query44 1225 805 810 805
query45 227 199 204 199
query46 1048 676 712 676
query47 1666 1642 1673 1642
query48 360 288 312 288
query49 1225 308 307 307
query50 741 328 375 328
query51 5407 5355 5287 5287
query52 117 91 104 91
query53 231 160 162 160
query54 1395 678 641 641
query55 107 95 106 95
query56 304 252 240 240
query57 1054 978 992 978
query58 297 286 291 286
query59 3070 2790 2721 2721
query60 332 297 299 297
query61 133 163 127 127
query62 580 495 500 495
query63 198 184 182 182
query64 5988 1733 1791 1733
query65 3423 3362 3345 3345
query66 1454 394 375 375
query67 15553 15075 15242 15075
query68 13188 546 561 546
query69 600 320 339 320
query70 1820 1628 1577 1577
query71 622 271 300 271
query72 5320 3465 3492 3465
query73 3059 347 347 347
query74 7143 6568 6486 6486
query75 5273 2301 2287 2287
query76 6448 1181 1235 1181
query77 754 327 337 327
query78 9132 8599 8544 8544
query79 1493 533 540 533
query80 577 345 341 341
query81 465 215 213 213
query82 231 103 101 101
query83 191 144 142 142
query84 256 60 58 58
query85 958 276 277 276
query86 432 430 445 430
query87 3648 3507 3437 3437
query88 3082 2504 2529 2504
query89 370 287 288 287
query90 2028 253 256 253
query91 125 96 94 94
query92 83 73 83 73
query93 1233 488 497 488
query94 943 249 237 237
query95 547 506 495 495
query96 642 334 331 331
query97 4388 4198 4237 4198
query98 235 215 214 214
query99 1141 875 833 833
Total cold run time: 300871 ms
Total hot run time: 183811 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1826361824
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1826361830
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement-wip](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1752564013
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1837037224
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.83 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17163863598 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1880419074
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1879726253
<details>
<summary>TPC-DS: <b>Total hot run time: 184561 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c922696130949e4707d8045b6de3ceeda1b86ea1, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 932 343 334 334
query2 6536 2071 1949 1949
query3 6529 229 235 229
query4 25305 22465 22424 22424
query5 5823 616 611 611
query6 338 214 214 214
query7 4643 311 302 302
query8 249 233 223 223
query9 8717 2986 2977 2977
query10 640 374 320 320
query11 16230 15682 15566 15566
query12 147 109 100 100
query13 1655 340 343 340
query14 12858 7572 7617 7572
query15 283 239 214 214
query16 6308 317 320 317
query17 1669 534 517 517
query18 1942 313 307 307
query19 244 157 157 157
query20 111 106 101 101
query21 193 96 95 95
query22 5252 4946 5001 4946
query23 32128 31091 31467 31091
query24 11695 2977 3000 2977
query25 563 364 371 364
query26 1695 166 172 166
query27 2928 307 330 307
query28 6402 2138 2124 2124
query29 1723 429 413 413
query30 300 145 149 145
query31 1008 856 821 821
query32 119 90 88 88
query33 825 351 330 330
query34 850 469 471 469
query35 1004 864 901 864
query36 1397 1237 1245 1237
query37 183 99 95 95
query38 3505 3388 3299 3299
query39 1369 1287 1292 1287
query40 297 88 85 85
query41 39 36 36 36
query42 110 108 105 105
query43 543 505 538 505
query44 1163 805 806 805
query45 213 209 203 203
query46 1055 721 725 721
query47 1756 1669 1640 1640
query48 365 290 293 290
query49 1090 321 303 303
query50 732 320 358 320
query51 5470 5413 5301 5301
query52 112 97 100 97
query53 235 171 166 166
query54 1107 684 677 677
query55 102 100 98 98
query56 256 255 282 255
query57 1064 1042 1012 1012
query58 303 276 259 259
query59 3024 2826 2902 2826
query60 316 297 292 292
query61 151 150 161 150
query62 568 483 474 474
query63 195 182 187 182
query64 5354 1701 1699 1699
query65 3414 3351 3311 3311
query66 1249 389 424 389
query67 15483 15461 15923 15461
query68 13296 525 538 525
query69 633 319 312 312
query70 1737 1639 1586 1586
query71 610 291 286 286
query72 5624 3460 3460 3460
query73 3319 376 354 354
query74 7398 6487 6539 6487
query75 5559 2311 2294 2294
query76 6699 1236 1213 1213
query77 770 349 309 309
query78 9107 8657 8651 8651
query79 1117 532 509 509
query80 566 342 337 337
query81 459 215 212 212
query82 223 107 109 107
query83 200 142 141 141
query84 253 55 62 55
query85 918 275 269 269
query86 412 433 383 383
query87 3639 3479 3431 3431
query88 3141 2505 2535 2505
query89 385 287 273 273
query90 1996 259 260 259
query91 124 100 98 98
query92 88 72 80 72
query93 1348 484 473 473
query94 905 247 246 246
query95 599 511 505 505
query96 634 347 341 341
query97 4413 4268 4222 4222
query98 235 218 222 218
query99 1142 874 892 874
Total cold run time: 297613 ms
Total hot run time: 184561 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1879924789
<details>
<summary>TPC-H: <b>Total hot run time: 38550 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fc41b8a0b61c1bc9c00f07369017df4578fc17ad, data reload: false
------ Round 1 ----------------------------------
q1 17677 4947 4949 4947
q2 2019 155 142 142
q3 10588 1144 1137 1137
q4 10193 795 778 778
q5 7764 2970 2964 2964
q6 206 122 125 122
q7 908 518 505 505
q8 9275 1991 1986 1986
q9 6421 6366 6317 6317
q10 8232 3030 3033 3030
q11 427 220 221 220
q12 395 226 228 226
q13 18020 3419 3395 3395
q14 235 210 201 201
q15 549 511 505 505
q16 452 389 416 389
q17 938 680 527 527
q18 7204 6675 6645 6645
q19 1552 1526 1536 1526
q20 706 336 310 310
q21 2766 2380 2392 2380
q22 349 298 315 298
Total cold run time: 106876 ms
Total hot run time: 38550 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4919 4947 4961 4947
q2 337 230 236 230
q3 3287 3307 3253 3253
q4 2118 2023 2033 2023
q5 5800 5790 5770 5770
q6 205 116 115 115
q7 2292 1885 1883 1883
q8 3373 3435 3425 3425
q9 8766 8765 8784 8765
q10 3766 3819 3834 3819
q11 550 442 449 442
q12 798 630 624 624
q13 7072 3197 3183 3183
q14 287 261 271 261
q15 546 499 502 499
q16 528 468 499 468
q17 1875 1838 1853 1838
q18 8586 8334 8342 8334
q19 1587 1638 1624 1624
q20 2183 1986 1984 1984
q21 5603 5396 5229 5229
q22 514 466 464 464
Total cold run time: 64992 ms
Total hot run time: 59180 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](fe-memory) support label num threshold to reduce fe memory consumption [doris]
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on PR #22889:
URL: https://github.com/apache/doris/pull/22889#issuecomment-1879949574
ut is ongoing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org