You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wangshuo128 (via GitHub)" <gi...@apache.org> on 2024/03/22 08:48:11 UTC
[PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
wangshuo128 opened a new pull request, #32681:
URL: https://github.com/apache/doris/pull/32681
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017457019
<details>
<summary>TPC-H: <b>Total hot run time: 38335 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 543505ddda22b95ad389d4ddce20ef423133deb8, data reload: false
------ Round 1 ----------------------------------
q1 16333 4764 4103 4103
q2 2122 163 157 157
q3 10036 1179 1231 1179
q4 9723 778 787 778
q5 7151 3033 2989 2989
q6 209 126 124 124
q7 1040 593 584 584
q8 9045 2004 1979 1979
q9 7243 6608 6566 6566
q10 8450 3454 3491 3454
q11 432 234 229 229
q12 375 197 201 197
q13 17807 2846 2862 2846
q14 243 204 205 204
q15 508 458 465 458
q16 523 389 373 373
q17 951 611 509 509
q18 7381 6796 6684 6684
q19 3274 1446 1447 1446
q20 543 269 251 251
q21 3697 2942 2950 2942
q22 356 293 283 283
Total cold run time: 107442 ms
Total hot run time: 38335 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4138 4077 4072 4072
q2 328 233 231 231
q3 3064 2890 2911 2890
q4 1899 1568 1580 1568
q5 5444 5448 5478 5448
q6 204 116 119 116
q7 2289 1927 1877 1877
q8 3187 3305 3309 3305
q9 8758 8705 8692 8692
q10 3771 3756 3712 3712
q11 545 443 441 441
q12 733 549 550 549
q13 17143 2870 2879 2870
q14 272 261 250 250
q15 475 455 460 455
q16 468 425 430 425
q17 1722 1487 1456 1456
q18 7688 7501 7536 7501
q19 1624 1509 1490 1490
q20 1918 1732 1695 1695
q21 4812 4813 4659 4659
q22 533 456 451 451
Total cold run time: 71015 ms
Total hot run time: 54153 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017118179
TeamCity be ut coverage result:
Function Coverage: 35.26% (8733/24770)
Line Coverage: 27.06% (71523/264342)
Region Coverage: 26.29% (37101/141125)
Branch Coverage: 23.19% (18977/81836)
Coverage Report: http://coverage.selectdb-in.cc/coverage/99a508d3140668dbab5ee8b33cffcac0873bc215_99a508d3140668dbab5ee8b33cffcac0873bc215/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "wangshuo128 (via GitHub)" <gi...@apache.org>.
wangshuo128 commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017423656
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017135175
<details>
<summary>TPC-DS: <b>Total hot run time: 186334 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 99a508d3140668dbab5ee8b33cffcac0873bc215, data reload: false
query1 1051 367 353 353
query2 7468 2099 2044 2044
query3 6709 216 217 216
query4 31785 21444 21300 21300
query5 4401 425 408 408
query6 269 175 180 175
query7 4657 287 288 287
query8 226 178 165 165
query9 9390 2316 2298 2298
query10 568 248 252 248
query11 17369 14296 14328 14296
query12 134 98 86 86
query13 1632 431 424 424
query14 12505 10484 10754 10484
query15 326 193 200 193
query16 8240 258 265 258
query17 2155 574 532 532
query18 2094 275 271 271
query19 334 149 153 149
query20 92 84 87 84
query21 202 128 129 128
query22 4995 4781 4803 4781
query23 33865 32756 33006 32756
query24 10579 2874 2839 2839
query25 622 388 392 388
query26 1188 154 156 154
query27 2420 349 357 349
query28 6937 1884 1902 1884
query29 868 648 636 636
query30 301 148 154 148
query31 985 737 733 733
query32 97 58 59 58
query33 770 269 249 249
query34 1041 484 489 484
query35 840 616 601 601
query36 996 909 884 884
query37 133 77 78 77
query38 3549 3495 3483 3483
query39 1462 1437 1433 1433
query40 217 121 112 112
query41 51 47 47 47
query42 109 98 102 98
query43 479 445 462 445
query44 1186 726 736 726
query45 273 265 264 264
query46 1118 701 704 701
query47 1910 1873 1830 1830
query48 438 359 357 357
query49 1104 343 340 340
query50 764 372 365 365
query51 6767 6641 6651 6641
query52 121 91 91 91
query53 344 282 279 279
query54 311 254 247 247
query55 93 86 81 81
query56 254 244 239 239
query57 1218 1117 1138 1117
query58 236 202 212 202
query59 2819 2631 2675 2631
query60 275 249 257 249
query61 129 112 114 112
query62 659 456 444 444
query63 310 282 277 277
query64 5749 4086 3979 3979
query65 3065 3026 3022 3022
query66 886 379 377 377
query67 15228 15058 14686 14686
query68 8958 552 545 545
query69 638 414 405 405
query70 1206 1192 1114 1114
query71 553 285 288 285
query72 6904 2739 2547 2547
query73 768 314 321 314
query74 7132 6682 6663 6663
query75 4856 2991 3041 2991
query76 5574 860 895 860
query77 872 264 262 262
query78 10896 10096 10195 10096
query79 11812 518 529 518
query80 1420 392 391 391
query81 513 218 215 215
query82 700 212 201 201
query83 195 148 151 148
query84 286 80 78 78
query85 1406 348 309 309
query86 401 312 316 312
query87 3760 3530 3526 3526
query88 4708 2317 2286 2286
query89 508 369 377 369
query90 2068 177 175 175
query91 174 139 142 139
query92 59 49 47 47
query93 6993 514 488 488
query94 1292 173 171 171
query95 428 329 329 329
query96 616 265 269 265
query97 3085 2897 2890 2890
query98 237 214 220 214
query99 1158 896 886 886
Total cold run time: 319544 ms
Total hot run time: 186334 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2022026510
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017465206
TeamCity be ut coverage result:
Function Coverage: 35.26% (8733/24770)
Line Coverage: 27.06% (71523/264355)
Region Coverage: 26.29% (37110/141132)
Branch Coverage: 23.19% (18978/81842)
Coverage Report: http://coverage.selectdb-in.cc/coverage/543505ddda22b95ad389d4ddce20ef423133deb8_543505ddda22b95ad389d4ddce20ef423133deb8/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017479742
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 543505ddda22b95ad389d4ddce20ef423133deb8 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 19.9 seconds inserted 10000000 Rows, about 502K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "platoneko (via GitHub)" <gi...@apache.org>.
platoneko commented on code in PR #32681:
URL: https://github.com/apache/doris/pull/32681#discussion_r1535283875
##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -826,18 +826,7 @@ Status CloudMetaMgr::get_storage_vault_info(
});
}
for (const auto& vault : resp.storage_vault()) {
- THdfsParams params;
- params.fs_name = vault.hdfs_info().build_conf().fs_name();
- params.user = vault.hdfs_info().build_conf().user();
- params.hdfs_kerberos_keytab = vault.hdfs_info().build_conf().hdfs_kerberos_keytab();
- params.hdfs_kerberos_principal = vault.hdfs_info().build_conf().hdfs_kerberos_principal();
- for (const auto& confs : vault.hdfs_info().build_conf().hdfs_confs()) {
- THdfsConf conf;
- conf.key = confs.key();
- conf.value = confs.value();
- params.hdfs_conf.emplace_back(std::move(conf));
- }
- vault_infos->emplace_back(vault.id(), std::move(params));
+ vault_infos->emplace_back(vault.id(), std::move(vault.hdfs_info()));
Review Comment:
```suggestion
if (vault.has_hdfs_info()) {
vault_infos->emplace_back(vault.id(), vault.hdfs_info());
}
```
std::move has no effect on const
##########
be/src/util/vault_info_util.cpp:
##########
Review Comment:
Consider to move these code to `io/hdfs_util.cpp`
##########
be/src/cloud/cloud_meta_mgr.h:
##########
@@ -71,7 +71,7 @@ class CloudMetaMgr {
Status precommit_txn(const StreamLoadContext& ctx);
Status get_storage_vault_info(
Review Comment:
```suggestion
using StorageVaultInfos = std::vector<...>;
Result<StorageVaultInfos> get_storage_vault_info()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32681:
URL: https://github.com/apache/doris/pull/32681#discussion_r1535480884
##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -801,8 +801,7 @@ Status CloudMetaMgr::precommit_txn(const StreamLoadContext& ctx) {
return retry_rpc("precommit txn", req, &res, &MetaService_Stub::precommit_txn);
}
-Status CloudMetaMgr::get_storage_vault_info(
- std::vector<std::tuple<std::string, std::variant<S3Conf, THdfsParams>>>* vault_infos) {
+Status CloudMetaMgr::get_storage_vault_info(StorageVaultInfos* vault_infos) {
Review Comment:
warning: method 'get_storage_vault_info' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status CloudMetaMgr::get_storage_vault_info(StorageVaultInfos* vault_infos) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017361494
<details>
<summary>TPC-DS: <b>Total hot run time: 186654 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 99a508d3140668dbab5ee8b33cffcac0873bc215, data reload: false
query1 934 367 358 358
query2 7487 1988 2017 1988
query3 6714 208 209 208
query4 31842 21473 21328 21328
query5 4344 497 414 414
query6 271 182 173 173
query7 4626 296 294 294
query8 229 170 183 170
query9 9082 2337 2312 2312
query10 566 254 262 254
query11 15367 14367 14402 14367
query12 133 83 86 83
query13 1623 415 435 415
query14 12322 10757 10843 10757
query15 263 215 203 203
query16 8077 266 265 265
query17 1935 584 551 551
query18 2036 282 289 282
query19 241 154 156 154
query20 95 84 90 84
query21 193 125 121 121
query22 5018 4857 4806 4806
query23 33513 32760 32807 32760
query24 10734 2927 2857 2857
query25 625 388 389 388
query26 1187 155 158 155
query27 2625 353 359 353
query28 7259 1901 1900 1900
query29 893 648 612 612
query30 302 148 158 148
query31 993 725 727 725
query32 95 60 60 60
query33 780 258 272 258
query34 1059 483 494 483
query35 828 606 613 606
query36 1018 887 869 869
query37 129 80 77 77
query38 3529 3440 3415 3415
query39 1471 1474 1455 1455
query40 210 119 111 111
query41 52 49 48 48
query42 104 96 100 96
query43 494 466 473 466
query44 1200 730 737 730
query45 281 273 249 249
query46 1120 718 702 702
query47 1900 1867 1839 1839
query48 473 351 356 351
query49 1127 349 348 348
query50 760 369 361 361
query51 6814 6578 6653 6578
query52 105 98 90 90
query53 341 274 275 274
query54 324 250 248 248
query55 90 79 79 79
query56 255 237 250 237
query57 1216 1146 1159 1146
query58 239 204 209 204
query59 2784 2680 2573 2573
query60 284 253 333 253
query61 115 112 112 112
query62 666 461 463 461
query63 298 273 272 272
query64 5914 4165 4053 4053
query65 3132 3106 3088 3088
query66 861 365 355 355
query67 15549 14813 15078 14813
query68 7086 517 518 517
query69 620 383 370 370
query70 1266 1213 1154 1154
query71 525 284 277 277
query72 6409 2746 2554 2554
query73 743 316 321 316
query74 7358 6666 6679 6666
query75 4081 2849 2881 2849
query76 5027 898 809 809
query77 666 259 254 254
query78 10976 10135 10077 10077
query79 10395 536 524 524
query80 1825 386 387 386
query81 516 220 208 208
query82 610 208 196 196
query83 235 153 149 149
query84 284 81 77 77
query85 1244 314 304 304
query86 380 329 315 315
query87 3769 3539 3519 3519
query88 4700 2317 2300 2300
query89 502 387 362 362
query90 2099 170 172 170
query91 169 137 141 137
query92 59 47 47 47
query93 7023 502 489 489
query94 1287 190 177 177
query95 440 329 316 316
query96 601 272 269 269
query97 3058 2900 2896 2896
query98 253 220 209 209
query99 1163 928 916 916
Total cold run time: 312515 ms
Total hot run time: 186654 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017347444
<details>
<summary>TPC-H: <b>Total hot run time: 37737 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 99a508d3140668dbab5ee8b33cffcac0873bc215, data reload: false
------ Round 1 ----------------------------------
q1 17705 4188 4069 4069
q2 2138 171 153 153
q3 10559 1177 1168 1168
q4 10234 752 750 750
q5 7456 3004 2944 2944
q6 205 126 121 121
q7 1052 589 554 554
q8 9344 1964 1965 1964
q9 7171 6599 6544 6544
q10 8431 3483 3553 3483
q11 438 219 218 218
q12 418 194 189 189
q13 17802 2875 2866 2866
q14 225 207 205 205
q15 503 475 468 468
q16 496 371 380 371
q17 926 606 532 532
q18 7113 6407 6404 6404
q19 1543 1420 1457 1420
q20 552 272 240 240
q21 3514 2914 2800 2800
q22 339 289 274 274
Total cold run time: 108164 ms
Total hot run time: 37737 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4127 4063 4044 4044
q2 334 227 232 227
q3 2941 2861 2829 2829
q4 1833 1523 1536 1523
q5 5293 5315 5338 5315
q6 192 113 117 113
q7 2256 1871 1847 1847
q8 3130 3264 3247 3247
q9 8645 8678 8636 8636
q10 3805 3784 3771 3771
q11 547 441 440 440
q12 716 530 539 530
q13 16929 2837 2850 2837
q14 275 249 252 249
q15 498 459 458 458
q16 472 409 419 409
q17 1718 1496 1450 1450
q18 7519 7252 7208 7208
q19 1613 1536 1544 1536
q20 1909 1738 1721 1721
q21 4775 4704 4732 4704
q22 521 457 449 449
Total cold run time: 70048 ms
Total hot run time: 53543 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2020130729
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 1b4dce8152048de6680841c0e09352b836aa76fa with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 22.1 seconds inserted 10000000 Rows, about 452K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #32681:
URL: https://github.com/apache/doris/pull/32681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017365383
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 99a508d3140668dbab5ee8b33cffcac0873bc215 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 22.0 seconds inserted 10000000 Rows, about 454K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2019358832
TeamCity be ut coverage result:
Function Coverage: 35.27% (8738/24778)
Line Coverage: 27.05% (71559/264535)
Region Coverage: 26.28% (37114/141202)
Branch Coverage: 23.18% (18979/81882)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6b1d0401f6c85f64f67b84703e33c43f483c4396_6b1d0401f6c85f64f67b84703e33c43f483c4396/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "wangshuo128 (via GitHub)" <gi...@apache.org>.
wangshuo128 commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2020060272
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #32681:
URL: https://github.com/apache/doris/pull/32681#discussion_r1535269407
##########
be/src/util/vault_info_util.cpp:
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vault_info_util.h"
+
+namespace doris {
+
+THdfsParams to_hdfs_params(const cloud::HdfsVaultInfo& vault) {
Review Comment:
also consider renaming HdfsVaultInfo to HdfsVaultInfoPB to keep the name consistent.
maybe in a separate PR
##########
be/src/util/vault_info_util.cpp:
##########
Review Comment:
seems no need to add a new file.
this helper function can be inlined in cloud_storage_engin.cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017126533
<details>
<summary>TPC-H: <b>Total hot run time: 37834 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 99a508d3140668dbab5ee8b33cffcac0873bc215, data reload: false
------ Round 1 ----------------------------------
q1 17654 4338 4095 4095
q2 2118 155 153 153
q3 10596 1188 1207 1188
q4 10231 748 751 748
q5 7463 3033 2994 2994
q6 206 124 122 122
q7 1034 582 580 580
q8 9346 1943 1985 1943
q9 7160 6592 6532 6532
q10 8441 3473 3631 3473
q11 428 223 221 221
q12 408 203 193 193
q13 17778 2892 2874 2874
q14 246 211 207 207
q15 513 470 455 455
q16 486 374 369 369
q17 943 605 530 530
q18 7081 6471 6344 6344
q19 4210 1438 1374 1374
q20 541 251 251 251
q21 3674 2921 2898 2898
q22 334 294 290 290
Total cold run time: 110891 ms
Total hot run time: 37834 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4096 4069 4061 4061
q2 323 228 227 227
q3 2957 2817 2863 2817
q4 1808 1542 1567 1542
q5 5320 5338 5393 5338
q6 198 117 117 117
q7 2266 1901 1870 1870
q8 3170 3323 3304 3304
q9 8738 8779 8763 8763
q10 3832 3807 3804 3804
q11 553 447 454 447
q12 723 600 535 535
q13 16909 2894 2907 2894
q14 295 250 261 250
q15 512 456 462 456
q16 503 441 417 417
q17 1747 1502 1472 1472
q18 7587 7771 7990 7771
q19 5858 1527 1531 1527
q20 2035 1848 1804 1804
q21 5022 4862 4664 4664
q22 571 455 454 454
Total cold run time: 75023 ms
Total hot run time: 54534 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32681:
URL: https://github.com/apache/doris/pull/32681#discussion_r1535248847
##########
be/src/util/vault_info_util.h:
##########
@@ -0,0 +1,27 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "gen_cpp/cloud.pb.h"
Review Comment:
warning: 'gen_cpp/cloud.pb.h' file not found [clang-diagnostic-error]
```cpp
#include "gen_cpp/cloud.pb.h"
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "wangshuo128 (via GitHub)" <gi...@apache.org>.
wangshuo128 commented on code in PR #32681:
URL: https://github.com/apache/doris/pull/32681#discussion_r1535276713
##########
be/src/util/vault_info_util.cpp:
##########
Review Comment:
I think this converting logic should also be used by `recycler`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2014629614
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017137301
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 99a508d3140668dbab5ee8b33cffcac0873bc215 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 20.6 seconds inserted 10000000 Rows, about 485K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "wangshuo128 (via GitHub)" <gi...@apache.org>.
wangshuo128 commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2019295129
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "wangshuo128 (via GitHub)" <gi...@apache.org>.
wangshuo128 commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017095422
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2020138182
TeamCity be ut coverage result:
Function Coverage: 35.27% (8738/24777)
Line Coverage: 27.04% (71535/264516)
Region Coverage: 26.28% (37119/141219)
Branch Coverage: 23.18% (18978/81882)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1b4dce8152048de6680841c0e09352b836aa76fa_1b4dce8152048de6680841c0e09352b836aa76fa/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "wangshuo128 (via GitHub)" <gi...@apache.org>.
wangshuo128 commented on code in PR #32681:
URL: https://github.com/apache/doris/pull/32681#discussion_r1535280564
##########
be/src/util/vault_info_util.cpp:
##########
Review Comment:
After checking, `recycler` and `BE` are different projects...
moving this to cloud_storage_engin.cpp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017501714
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "wangshuo128 (via GitHub)" <gi...@apache.org>.
wangshuo128 commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017310658
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017337730
TeamCity be ut coverage result:
Function Coverage: 35.25% (8732/24770)
Line Coverage: 27.06% (71522/264346)
Region Coverage: 26.29% (37109/141128)
Branch Coverage: 23.19% (18977/81838)
Coverage Report: http://coverage.selectdb-in.cc/coverage/99a508d3140668dbab5ee8b33cffcac0873bc215_99a508d3140668dbab5ee8b33cffcac0873bc215/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2017474964
<details>
<summary>TPC-DS: <b>Total hot run time: 184393 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 543505ddda22b95ad389d4ddce20ef423133deb8, data reload: false
query1 954 369 353 353
query2 7492 1981 2042 1981
query3 5174 216 208 208
query4 31527 21260 21375 21260
query5 3813 392 390 390
query6 264 179 176 176
query7 4257 293 288 288
query8 222 177 173 173
query9 8461 2281 2273 2273
query10 589 251 247 247
query11 17498 14526 14556 14526
query12 136 91 86 86
query13 1935 416 422 416
query14 12724 11226 10494 10494
query15 260 215 197 197
query16 7876 268 264 264
query17 1847 591 550 550
query18 1961 295 295 295
query19 237 156 151 151
query20 99 88 98 88
query21 203 130 129 129
query22 4962 4783 4819 4783
query23 32649 31743 32036 31743
query24 10766 2890 2851 2851
query25 799 379 380 379
query26 1242 163 163 163
query27 2871 349 347 347
query28 6749 1874 1867 1867
query29 886 636 629 629
query30 306 153 149 149
query31 969 729 733 729
query32 104 59 62 59
query33 768 346 275 275
query34 938 489 494 489
query35 829 633 630 630
query36 1002 866 873 866
query37 113 77 76 76
query38 3545 3402 3432 3402
query39 1485 1459 1419 1419
query40 221 119 116 116
query41 55 48 48 48
query42 102 96 90 90
query43 504 461 462 461
query44 1313 747 722 722
query45 289 266 237 237
query46 1126 700 694 694
query47 1814 1781 1731 1731
query48 444 348 359 348
query49 1112 351 344 344
query50 752 357 350 350
query51 6583 6570 6548 6548
query52 105 91 91 91
query53 351 277 277 277
query54 308 247 253 247
query55 80 79 77 77
query56 253 230 230 230
query57 1213 1132 1136 1132
query58 237 209 210 209
query59 2890 2629 2592 2592
query60 313 247 252 247
query61 117 116 115 115
query62 643 445 431 431
query63 300 274 276 274
query64 5417 4180 4062 4062
query65 3102 3041 3022 3022
query66 985 370 359 359
query67 15030 14217 14118 14118
query68 8599 535 535 535
query69 632 397 395 395
query70 1275 1181 1186 1181
query71 543 274 282 274
query72 6837 2728 2563 2563
query73 1440 327 314 314
query74 7143 6690 6610 6610
query75 4246 2849 2883 2849
query76 5173 879 961 879
query77 757 278 271 271
query78 11026 10150 10137 10137
query79 9649 533 535 533
query80 1737 417 370 370
query81 486 218 213 213
query82 651 199 195 195
query83 231 155 149 149
query84 284 77 79 77
query85 1321 325 330 325
query86 374 292 252 252
query87 3782 3498 3557 3498
query88 4489 2269 2266 2266
query89 492 369 366 366
query90 1974 173 173 173
query91 172 136 136 136
query92 79 47 46 46
query93 7045 499 480 480
query94 886 172 170 170
query95 423 321 327 321
query96 602 271 266 266
query97 3028 2871 2859 2859
query98 236 217 210 210
query99 1236 930 907 907
Total cold run time: 311019 ms
Total hot run time: 184393 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2020125259
<details>
<summary>TPC-DS: <b>Total hot run time: 181277 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b4dce8152048de6680841c0e09352b836aa76fa, data reload: false
query1 931 370 349 349
query2 6539 1968 1994 1968
query3 6705 218 212 212
query4 31790 21408 21276 21276
query5 4308 403 392 392
query6 269 180 171 171
query7 4635 305 295 295
query8 226 168 178 168
query9 9190 2346 2349 2346
query10 570 234 252 234
query11 17176 14232 14283 14232
query12 138 91 81 81
query13 1609 419 424 419
query14 10120 7593 8017 7593
query15 257 208 205 205
query16 8130 275 263 263
query17 1938 584 561 561
query18 2095 292 282 282
query19 361 159 159 159
query20 93 88 90 88
query21 202 129 127 127
query22 4992 4799 4780 4780
query23 33663 32648 32436 32436
query24 11821 2902 2891 2891
query25 654 386 393 386
query26 1792 158 171 158
query27 3102 348 360 348
query28 7888 1925 1921 1921
query29 1044 647 639 639
query30 302 148 155 148
query31 999 723 737 723
query32 90 59 59 59
query33 768 256 260 256
query34 1127 495 486 486
query35 838 616 617 616
query36 1025 907 886 886
query37 271 66 67 66
query38 3581 3436 3434 3434
query39 1475 1456 1432 1432
query40 300 120 117 117
query41 51 48 48 48
query42 109 95 101 95
query43 489 437 441 437
query44 1228 745 735 735
query45 286 251 255 251
query46 1108 716 715 715
query47 1920 1847 1840 1840
query48 443 377 363 363
query49 1237 348 341 341
query50 763 373 384 373
query51 6620 6628 6580 6580
query52 109 85 96 85
query53 348 277 286 277
query54 318 232 236 232
query55 87 75 78 75
query56 242 220 219 219
query57 1213 1120 1149 1120
query58 230 207 207 207
query59 2905 2562 2478 2478
query60 267 244 243 243
query61 97 92 90 90
query62 688 436 438 436
query63 304 277 278 277
query64 6364 4070 3953 3953
query65 3636 3039 3032 3032
query66 1413 367 353 353
query67 15054 14797 14505 14505
query68 6689 537 533 533
query69 615 386 390 386
query70 1280 1119 1112 1112
query71 483 268 266 266
query72 6304 2709 2640 2640
query73 736 325 324 324
query74 7248 6376 6420 6376
query75 3528 2194 2220 2194
query76 4779 852 947 852
query77 632 278 265 265
query78 10752 10303 10183 10183
query79 8553 525 526 525
query80 1453 393 400 393
query81 516 213 218 213
query82 760 97 86 86
query83 214 157 159 157
query84 298 82 83 82
query85 1510 376 369 369
query86 420 314 331 314
query87 3624 3591 3535 3535
query88 4889 2388 2368 2368
query89 507 364 368 364
query90 2008 176 175 175
query91 166 141 133 133
query92 63 45 47 45
query93 6399 517 487 487
query94 1111 178 180 178
query95 433 331 326 326
query96 614 270 269 269
query97 2676 2451 2482 2451
query98 236 211 208 208
query99 1253 907 905 905
Total cold run time: 311292 ms
Total hot run time: 181277 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](cloud) Support path prefix for HDFS storage vault. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32681:
URL: https://github.com/apache/doris/pull/32681#issuecomment-2020103121
<details>
<summary>TPC-H: <b>Total hot run time: 38132 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b4dce8152048de6680841c0e09352b836aa76fa, data reload: false
------ Round 1 ----------------------------------
q1 17625 4456 4126 4126
q2 2112 155 153 153
q3 10597 1101 1219 1101
q4 10390 772 749 749
q5 7482 3046 3050 3046
q6 202 124 123 123
q7 1035 604 575 575
q8 9335 1985 1991 1985
q9 7203 6605 6613 6605
q10 8469 3471 3539 3471
q11 436 230 218 218
q12 367 203 198 198
q13 18188 2907 2873 2873
q14 230 215 203 203
q15 512 475 455 455
q16 502 374 374 374
q17 962 576 532 532
q18 7522 6863 6453 6453
q19 3699 1495 1404 1404
q20 550 258 255 255
q21 3563 2990 2933 2933
q22 356 300 316 300
Total cold run time: 111337 ms
Total hot run time: 38132 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4123 4077 4068 4068
q2 326 228 227 227
q3 2995 2882 2837 2837
q4 1809 1567 1564 1564
q5 5333 5369 5365 5365
q6 193 114 114 114
q7 2256 1865 1847 1847
q8 3163 3310 3326 3310
q9 8730 8720 8720 8720
q10 3854 3814 3790 3790
q11 533 442 458 442
q12 719 571 547 547
q13 16900 2839 2844 2839
q14 286 276 248 248
q15 489 460 453 453
q16 477 417 419 417
q17 1742 1504 1474 1474
q18 7444 7382 7181 7181
q19 1641 1535 1497 1497
q20 1896 1760 1707 1707
q21 4863 4622 4596 4596
q22 526 456 446 446
Total cold run time: 70298 ms
Total hot run time: 53689 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org