You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ByteYue (via GitHub)" <gi...@apache.org> on 2024/02/26 09:22:45 UTC
[PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
ByteYue opened a new pull request, #31415:
URL: https://github.com/apache/doris/pull/31415
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1965993083
<details>
<summary>TPC-DS: <b>Total hot run time: 169055 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1cc1d73fd8dea6a92a432e3d2025c19064636813, data reload: false
query1 939 344 339 339
query2 6521 1749 1842 1749
query3 6722 201 200 200
query4 23573 20594 20603 20594
query5 4348 361 349 349
query6 256 174 176 174
query7 4627 306 304 304
query8 270 199 197 197
query9 8338 2294 2272 2272
query10 421 211 219 211
query11 14562 14269 14077 14077
query12 137 83 84 83
query13 1651 421 442 421
query14 8727 6135 6313 6135
query15 216 178 179 178
query16 7087 256 250 250
query17 1056 522 508 508
query18 1909 269 263 263
query19 195 151 156 151
query20 87 83 85 83
query21 198 116 126 116
query22 4820 4594 4598 4594
query23 30447 29923 29996 29923
query24 11206 3147 3097 3097
query25 622 352 363 352
query26 1685 156 165 156
query27 3044 336 317 317
query28 7014 1830 1816 1816
query29 1108 563 557 557
query30 280 136 140 136
query31 870 689 693 689
query32 93 60 57 57
query33 725 228 225 225
query34 1030 487 499 487
query35 865 750 744 744
query36 941 890 848 848
query37 136 63 63 63
query38 3105 2978 2998 2978
query39 1333 1282 1255 1255
query40 287 106 103 103
query41 41 36 36 36
query42 100 103 102 102
query43 442 427 434 427
query44 1078 704 721 704
query45 197 183 171 171
query46 1038 795 761 761
query47 1441 1493 1444 1444
query48 413 347 372 347
query49 1171 293 298 293
query50 770 378 388 378
query51 4327 4330 4284 4284
query52 97 96 91 91
query53 334 271 268 268
query54 309 237 232 232
query55 83 80 78 78
query56 232 213 201 201
query57 951 879 883 879
query58 219 196 205 196
query59 2199 2135 2125 2125
query60 253 219 219 219
query61 89 83 84 83
query62 615 378 363 363
query63 288 258 251 251
query64 6602 3012 3148 3012
query65 3238 3231 3241 3231
query66 1430 323 320 320
query67 14378 13876 14056 13876
query68 4992 555 556 555
query69 552 372 379 372
query70 1188 1222 1202 1202
query71 400 247 276 247
query72 6215 2802 2663 2663
query73 703 322 326 322
query74 6334 6003 5929 5929
query75 3070 2408 2411 2408
query76 2778 1017 1164 1017
query77 354 237 255 237
query78 9130 8512 8408 8408
query79 1303 541 552 541
query80 970 366 356 356
query81 472 196 198 196
query82 955 90 87 87
query83 223 122 123 122
query84 283 82 76 76
query85 1321 354 354 354
query86 373 294 298 294
query87 3303 3101 3049 3049
query88 2782 2353 2335 2335
query89 383 328 340 328
query90 1856 159 170 159
query91 150 119 124 119
query92 58 49 49 49
query93 1237 521 500 500
query94 1276 177 176 176
query95 442 333 342 333
query96 580 267 271 267
query97 4254 4096 4153 4096
query98 231 204 203 203
query99 1059 635 632 632
Total cold run time: 263013 ms
Total hot run time: 169055 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1987689228
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1503735213
##########
cloud/src/meta-service/keys.cpp:
##########
@@ -115,11 +115,12 @@ static void encode_prefix(const T& t, std::string* key) {
RecycleIndexKeyInfo, RecyclePartKeyInfo, RecycleRowsetKeyInfo, RecycleTxnKeyInfo, RecycleStageKeyInfo,
StatsTabletKeyInfo,
JobTabletKeyInfo, JobRecycleKeyInfo, RLJobProgressKeyInfo,
- CopyJobKeyInfo, CopyFileKeyInfo>);
+ CopyJobKeyInfo, CopyFileKeyInfo, StorageVaultKeyInfo>);
key->push_back(CLOUD_USER_KEY_SPACE01);
// Prefixes for key families
- if constexpr (std::is_same_v<T, InstanceKeyInfo>) {
+ if constexpr (std::is_same_v<T, InstanceKeyInfo>
+ || std::is_same_v<T, StorageVaultKeyInfo>) {
Review Comment:
StorageVaultKeyInfo is not int the family of InstanceKeyInfo.
it should be
0x01 "storage_vault" $instance_id $resource_id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1971495383
<details>
<summary>TPC-H: <b>Total hot run time: 38111 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee0c4da2e607810f875dc1c11ae4aa9434cc8f0f, data reload: false
------ Round 1 ----------------------------------
q1 17727 4143 4024 4024
q2 2060 149 147 147
q3 10623 1002 943 943
q4 4670 939 963 939
q5 7639 2960 3064 2960
q6 196 129 122 122
q7 1320 828 836 828
q8 9593 2039 2069 2039
q9 7374 6499 6499 6499
q10 8211 2584 2540 2540
q11 429 222 217 217
q12 756 317 310 310
q13 18010 2945 2964 2945
q14 290 253 247 247
q15 481 457 465 457
q16 465 396 404 396
q17 935 860 886 860
q18 6762 5921 5875 5875
q19 1563 1522 1524 1522
q20 544 273 283 273
q21 7559 3675 3703 3675
q22 800 302 293 293
Total cold run time: 108007 ms
Total hot run time: 38111 ms
----- Round 2, with runtime_filter_mode=off -----
q1 3990 4010 4006 4006
q2 317 220 222 220
q3 2963 2952 2935 2935
q4 1853 1858 1822 1822
q5 5247 5257 5266 5257
q6 200 120 118 118
q7 2259 1846 1822 1822
q8 3194 3300 3271 3271
q9 8539 8587 8541 8541
q10 6157 3735 3748 3735
q11 536 441 440 440
q12 685 526 550 526
q13 10805 2785 2835 2785
q14 281 267 248 248
q15 476 432 429 429
q16 463 419 410 410
q17 1703 1674 1658 1658
q18 8013 7475 7399 7399
q19 1630 1625 1624 1624
q20 1936 1748 1758 1748
q21 4951 4808 4778 4778
q22 461 495 472 472
Total cold run time: 66659 ms
Total hot run time: 54244 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1964449096
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1973312353
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1972679534
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1985378059
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1972712910
<details>
<summary>TPC-H: <b>Total hot run time: 32855 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 222d139e2a39b9332643858654815aeaf742cc53, data reload: false
------ Round 1 ----------------------------------
q1 18321 4494 4533 4494
q2 2584 227 225 225
q3 11380 1183 1222 1183
q4 4927 1255 1280 1255
q5 7913 2144 2145 2144
q6 156 122 115 115
q7 1246 1128 1123 1123
q8 10043 1092 1082 1082
q9 6075 5069 4967 4967
q10 7916 2025 2001 2001
q11 529 314 316 314
q12 847 424 422 422
q13 18080 3045 2968 2968
q14 346 315 299 299
q15 385 365 359 359
q16 543 513 509 509
q17 751 318 320 318
q18 5701 5170 4834 4834
q19 1165 403 410 403
q20 613 436 461 436
q21 5953 2960 2945 2945
q22 527 459 478 459
Total cold run time: 106001 ms
Total hot run time: 32855 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4605 4683 4646 4646
q2 278 215 221 215
q3 2225 1936 1931 1931
q4 1687 1393 1357 1357
q5 3375 3089 3068 3068
q6 142 104 103 103
q7 1915 1890 1957 1890
q8 1762 1655 1646 1646
q9 5518 5445 5476 5445
q10 2700 2359 2356 2356
q11 478 434 408 408
q12 754 659 672 659
q13 3116 2645 2853 2645
q14 303 295 292 292
q15 366 358 355 355
q16 496 494 489 489
q17 1137 757 783 757
q18 6084 5558 5262 5262
q19 1098 656 651 651
q20 1603 1386 1435 1386
q21 4913 4483 4308 4308
q22 659 618 562 562
Total cold run time: 45214 ms
Total hot run time: 40431 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1508612036
##########
cloud/test/meta_service_test.cpp:
##########
@@ -4924,6 +4978,109 @@ TEST(MetaServiceTest, LegacyUpdateAkSkTest) {
SyncPoint::get_instance()->clear_all_call_backs();
}
+TEST(MetaServiceTest, AddHdfsInfoTest) {
Review Comment:
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST(MetaServiceTest, AddHdfsInfoTest) {
^
```
<details>
<summary>Additional context</summary>
**cloud/test/meta_service_test.cpp:4980:** 101 lines including whitespace and comments (threshold 80)
```cpp
TEST(MetaServiceTest, AddHdfsInfoTest) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1503775189
##########
cloud/test/meta_service_test.cpp:
##########
@@ -4924,6 +4978,108 @@ TEST(MetaServiceTest, LegacyUpdateAkSkTest) {
SyncPoint::get_instance()->clear_all_call_backs();
}
+TEST(MetaServiceTest, AddHdfsInfoTest) {
Review Comment:
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST(MetaServiceTest, AddHdfsInfoTest) {
^
```
<details>
<summary>Additional context</summary>
**cloud/test/meta_service_test.cpp:4980:** 100 lines including whitespace and comments (threshold 80)
```cpp
TEST(MetaServiceTest, AddHdfsInfoTest) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1503775189
##########
cloud/test/meta_service_test.cpp:
##########
@@ -4924,6 +4978,108 @@ TEST(MetaServiceTest, LegacyUpdateAkSkTest) {
SyncPoint::get_instance()->clear_all_call_backs();
}
+TEST(MetaServiceTest, AddHdfsInfoTest) {
Review Comment:
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST(MetaServiceTest, AddHdfsInfoTest) {
^
```
<details>
<summary>Additional context</summary>
**cloud/test/meta_service_test.cpp:4980:** 100 lines including whitespace and comments (threshold 80)
```cpp
TEST(MetaServiceTest, AddHdfsInfoTest) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1963742026
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1965928009
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1503730789
##########
cloud/test/meta_service_test.cpp:
##########
@@ -302,6 +302,60 @@
sp->disable_processing();
}
+ // case: normal create instance without obj info and hdfs info
+ {
+ brpc::Controller cntl;
+ CreateInstanceRequest req;
+ req.set_instance_id("test_instance");
+ req.set_user_id("test_user");
+ req.set_name("test_name");
+
+ auto sp = SyncPoint::get_instance();
+ sp->set_call_back("encrypt_ak_sk:get_encryption_key_ret",
+ [](void* p) { *reinterpret_cast<int*>(p) = 0; });
+ sp->set_call_back("encrypt_ak_sk:get_encryption_key",
+ [](void* p) { *reinterpret_cast<std::string*>(p) = "test"; });
+ sp->set_call_back("encrypt_ak_sk:get_encryption_key_id",
+ [](void* p) { *reinterpret_cast<int*>(p) = 1; });
+ sp->enable_processing();
+ CreateInstanceResponse res;
+ meta_service->create_instance(reinterpret_cast<::google::protobuf::RpcController*>(&cntl),
+ &req, &res, nullptr);
+ ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
+ sp->clear_all_call_backs();
+ sp->clear_trace();
+ sp->disable_processing();
+ }
+
+ // case: normal create instance with hdfs info
+ {
+ brpc::Controller cntl;
+ CreateInstanceRequest req;
+ req.set_instance_id("test_instance");
+ req.set_user_id("test_user");
+ req.set_name("test_name");
+ HdfsParams hdfs;
+ hdfs.set_fs_name("test_name_node");
+ hdfs.set_user("test_user");
+ req.mutable_hdfs_info()->CopyFrom(hdfs);
+
+ auto sp = SyncPoint::get_instance();
Review Comment:
warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
```suggestion
auto *sp = SyncPoint::get_instance();
```
##########
cloud/test/meta_service_test.cpp:
##########
@@ -302,6 +302,60 @@ TEST(MetaServiceTest, CreateInstanceTest) {
sp->disable_processing();
}
+ // case: normal create instance without obj info and hdfs info
+ {
+ brpc::Controller cntl;
+ CreateInstanceRequest req;
+ req.set_instance_id("test_instance");
+ req.set_user_id("test_user");
+ req.set_name("test_name");
+
+ auto sp = SyncPoint::get_instance();
Review Comment:
warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
```suggestion
auto *sp = SyncPoint::get_instance();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1966001490
<details>
<summary>ClickBench: <b>Total hot run time: 32.12 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1cc1d73fd8dea6a92a432e3d2025c19064636813, data reload: false
query1 0.03 0.04 0.03
query2 0.06 0.02 0.02
query3 0.23 0.06 0.06
query4 1.68 0.10 0.10
query5 0.51 0.50 0.52
query6 1.35 0.67 0.68
query7 0.02 0.02 0.02
query8 0.03 0.02 0.03
query9 0.58 0.53 0.51
query10 0.57 0.56 0.57
query11 0.13 0.09 0.10
query12 0.12 0.10 0.09
query13 0.61 0.60 0.61
query14 0.79 0.79 0.80
query15 0.84 0.85 0.81
query16 0.38 0.37 0.37
query17 0.93 0.98 0.99
query18 0.27 0.24 0.25
query19 1.90 1.74 1.73
query20 0.01 0.01 0.01
query21 15.42 0.63 0.64
query22 3.10 4.51 2.36
query23 17.65 1.08 0.95
query24 2.08 0.49 0.38
query25 0.47 0.14 0.06
query26 0.19 0.16 0.17
query27 0.05 0.06 0.04
query28 12.07 0.85 0.82
query29 12.49 3.51 3.17
query30 0.61 0.55 0.53
query31 2.78 0.36 0.36
query32 3.35 0.47 0.48
query33 3.20 3.23 3.23
query34 15.35 4.72 4.75
query35 4.77 4.72 4.74
query36 1.14 1.07 1.06
query37 0.06 0.04 0.04
query38 0.04 0.03 0.03
query39 0.02 0.02 0.02
query40 0.16 0.13 0.13
query41 0.07 0.02 0.01
query42 0.02 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 106.16 s
Total hot run time: 32.12 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1972729405
<details>
<summary>TPC-DS: <b>Total hot run time: 189590 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 222d139e2a39b9332643858654815aeaf742cc53, data reload: false
query1 1057 395 381 381
query2 7855 2336 2334 2334
query3 6724 303 292 292
query4 26622 23943 24269 23943
query5 4449 861 680 680
query6 362 330 312 312
query7 4669 444 446 444
query8 296 258 247 247
query9 8409 3394 3356 3356
query10 549 343 333 333
query11 19694 20426 19793 19793
query12 315 232 254 232
query13 1716 650 656 650
query14 11492 8229 7604 7604
query15 349 246 258 246
query16 8309 395 391 391
query17 2079 698 664 664
query18 2204 411 386 386
query19 332 243 238 238
query20 145 140 151 140
query21 236 166 159 159
query22 5318 5121 5035 5035
query23 25083 24770 24355 24355
query24 6671 2068 2058 2058
query25 614 497 514 497
query26 785 233 223 223
query27 2272 493 507 493
query28 3265 2794 2816 2794
query29 931 753 782 753
query30 319 191 205 191
query31 1411 856 862 856
query32 138 98 95 95
query33 669 393 395 393
query34 890 653 660 653
query35 970 892 865 865
query36 1263 1175 1181 1175
query37 130 115 106 106
query38 3250 3171 3134 3134
query39 1170 1133 1133 1133
query40 251 172 170 170
query41 90 86 84 84
query42 149 151 154 151
query43 598 593 578 578
query44 1271 1061 1049 1049
query45 228 223 218 218
query46 886 790 795 790
query47 1710 1702 1701 1701
query48 596 516 520 516
query49 739 482 491 482
query50 654 542 553 542
query51 8888 8980 9048 8980
query52 156 145 144 144
query53 334 339 331 331
query54 372 352 349 349
query55 176 134 134 134
query56 356 357 346 346
query57 1097 1061 1073 1061
query58 352 329 330 329
query59 3467 3443 3402 3402
query60 394 400 378 378
query61 169 163 172 163
query62 503 440 405 405
query63 321 329 318 318
query64 4266 2181 1842 1842
query65 2388 2364 2358 2358
query66 936 392 397 392
query67 15495 15661 15135 15135
query68 2925 697 712 697
query69 650 604 552 552
query70 1629 1586 1604 1586
query71 441 423 426 423
query72 2767 2349 2344 2344
query73 680 472 481 472
query74 6772 7085 7204 7085
query75 3065 2807 2786 2786
query76 2185 658 639 639
query77 465 399 396 396
query78 6809 6293 6460 6293
query79 1065 742 757 742
query80 663 594 621 594
query81 471 259 279 259
query82 196 135 136 135
query83 238 244 224 224
query84 270 138 127 127
query85 860 521 547 521
query86 422 424 432 424
query87 3477 3284 3308 3284
query88 3759 3430 3445 3430
query89 442 425 414 414
query90 1046 219 217 217
query91 218 200 210 200
query92 83 80 81 80
query93 1029 724 696 696
query94 495 270 275 270
query95 583 521 501 501
query96 631 380 384 380
query97 2917 2820 2793 2793
query98 307 321 298 298
query99 849 754 746 746
Total cold run time: 258263 ms
Total hot run time: 189590 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1515676940
##########
cloud/test/meta_service_test.cpp:
##########
@@ -4924,6 +4980,110 @@ TEST(MetaServiceTest, LegacyUpdateAkSkTest) {
SyncPoint::get_instance()->clear_all_call_backs();
}
+TEST(MetaServiceTest, AddHdfsInfoTest) {
Review Comment:
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST(MetaServiceTest, AddHdfsInfoTest) {
^
```
<details>
<summary>Additional context</summary>
**cloud/test/meta_service_test.cpp:4982:** 102 lines including whitespace and comments (threshold 80)
```cpp
TEST(MetaServiceTest, AddHdfsInfoTest) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1963657628
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1972736191
<details>
<summary>ClickBench: <b>Total hot run time: 26.85 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 222d139e2a39b9332643858654815aeaf742cc53, data reload: false
query1 0.06 0.04 0.04
query2 0.09 0.04 0.04
query3 0.25 0.06 0.06
query4 1.60 0.08 0.08
query5 0.50 0.51 0.51
query6 1.46 0.97 0.96
query7 0.03 0.02 0.02
query8 0.07 0.04 0.04
query9 0.68 0.60 0.66
query10 0.68 0.69 0.68
query11 0.20 0.16 0.14
query12 0.18 0.15 0.15
query13 0.48 0.47 0.48
query14 0.74 0.71 0.72
query15 0.91 0.86 0.86
query16 0.38 0.40 0.40
query17 1.17 1.17 1.18
query18 0.31 0.34 0.36
query19 2.20 2.10 2.02
query20 0.02 0.02 0.02
query21 15.36 1.01 0.91
query22 0.58 0.58 0.57
query23 13.72 0.64 0.61
query24 0.60 0.20 0.18
query25 0.04 0.04 0.05
query26 0.21 0.19 0.19
query27 0.05 0.05 0.05
query28 1.32 0.69 0.69
query29 12.64 4.96 4.72
query30 0.82 0.75 0.75
query31 0.87 0.40 0.40
query32 3.22 0.39 0.40
query33 2.26 2.25 2.32
query34 2.26 2.21 2.21
query35 2.22 2.21 2.23
query36 0.93 0.99 1.00
query37 0.08 0.06 0.07
query38 0.05 0.05 0.04
query39 0.04 0.02 0.03
query40 0.18 0.21 0.17
query41 0.08 0.04 0.02
query42 0.03 0.04 0.03
query43 0.05 0.04 0.04
Total cold run time: 69.62 s
Total hot run time: 26.85 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1503746238
##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,158 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
return;
}
for (auto& obj_info : *instance.mutable_obj_info()) {
- if (obj_info.has_encryption_info()) {
- AkSkPair plain_ak_sk_pair;
- int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
- &plain_ak_sk_pair, code, msg);
- if (ret != 0) return;
- obj_info.set_ak(std::move(plain_ak_sk_pair.first));
- obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+ if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+ return;
+ }
+ }
+ // Iterate all the resources to return to the rpc caller
+ for (const auto& resource_id : instance.resource_ids()) {
+ std::string storage_vault_k;
+ storage_vault_key({instance_id, resource_id}, &storage_vault_k);
+ if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+ txn_kv_.get(), code, msg);
+ ret != 0) {
+ return;
}
}
response->mutable_obj_info()->CopyFrom(instance.obj_info());
}
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {
+ std::string vault_id = "1";
+ auto cmp = [](const std::string& prev, const auto& last) {
+ size_t prev_value = std::stoi(prev);
+ size_t last_id = 0;
+ if constexpr (std::is_same_v<decltype(last), ObjectStoreInfoPB>) {
+ last_id = std::stoi(last.id());
+ } else if constexpr (std::is_same_v<decltype(last), std::string>) {
+ last_id = std::stoi(last);
+ } else {
+ static_assert("Should never come here");
+ }
+
+ return prev_value > last_id ? prev : std::to_string(last_id);
+ };
+ ;
+ return std::accumulate(
+ instance.resource_ids().begin(), instance.resource_ids().end(),
+ std::accumulate(instance.obj_info().begin(), instance.obj_info().end(), vault_id, cmp),
+ cmp);
+}
+
+static int add_hdfs_storage_valut(InstanceInfoPB& instance, TxnKv* txn_kv,
+ const AlterHdfsParams& hdfs_param, MetaServiceCode& code,
+ std::string& msg) {
+ std::string key;
+ std::string vault_id = next_avaiable_vault_id(instance);
+ storage_vault_key({instance.instance_id(), vault_id}, &key);
+ StorageVaultPB vault;
+ vault.set_id(vault_id);
+ vault.set_name(hdfs_param.vault_name());
+ *vault.mutable_hdfs_infos() = hdfs_param.hdfs();
+ std::string val = vault.SerializeAsString();
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ msg = "failed to create txn";
+ LOG(WARNING) << msg << " err=" << err;
+ return -1;
+ }
+ err = txn->get(key, &val);
+ LOG(INFO) << "get instance_key=" << hex(key);
+
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::READ>(err);
+ msg = fmt::format("failed to get instance, instance_id={}, err={}", instance.instance_id(),
+ err);
+ return -1;
+ }
+ txn->put(key, val);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ msg = fmt::format("failed to commit for putting storage vault_id={}, vault_name={}, err={}",
+ vault_id, hdfs_param.vault_name(), err);
+ return -1;
+ }
+ instance.mutable_resource_ids()->Add(std::move(vault_id));
+ *instance.mutable_storage_vault_names()->Add() = hdfs_param.vault_name();
+ return 0;
+}
+
+[[maybe_unused]] static int remove_hdfs_storage_valut(std::string_view vault_key, TxnKv* txn_kv,
+ MetaServiceCode& code, std::string& msg) {
+ std::unique_ptr<Transaction> txn;
+ TxnErrorCode err = txn_kv->create_txn(&txn);
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::CREATE>(err);
+ msg = "failed to create txn";
+ LOG(WARNING) << msg << " err=" << err;
+ return -1;
+ }
+ txn->remove(vault_key);
+ LOG(INFO) << "remove storage_vault_key=" << hex(vault_key);
+
+ if (err != TxnErrorCode::TXN_OK) {
+ code = cast_as<ErrCategory::COMMIT>(err);
+ msg = fmt::format("failed to commit for removing storage vault_key={}, err={}", vault_key,
+ err);
+ return -1;
+ }
+ return 0;
+}
+
void MetaServiceImpl::alter_obj_store_info(google::protobuf::RpcController* controller,
const AlterObjStoreInfoRequest* request,
AlterObjStoreInfoResponse* response,
::google::protobuf::Closure* done) {
+ std::string ak, sk, bucket, prefix, endpoint, external_endpoint, region;
+ EncryptionInfoPB encryption_info;
+ AkSkPair cipher_ak_sk_pair;
RPC_PREPROCESS(alter_obj_store_info);
- // Prepare data
- if (!request->has_obj() || !request->obj().has_ak() || !request->obj().has_sk()) {
- code = MetaServiceCode::INVALID_ARGUMENT;
- msg = "s3 obj info err " + proto_to_json(*request);
- return;
- }
+ switch (request->op()) {
+ case AlterObjStoreInfoRequest::ADD_OBJ_INFO:
+ case AlterObjStoreInfoRequest::LEGACY_UPDATE_AK_SK:
+ case AlterObjStoreInfoRequest::UPDATE_AK_SK: {
+ // Prepare data
+ if (!request->has_obj() || !request->obj().has_ak() || !request->obj().has_sk()) {
+ code = MetaServiceCode::INVALID_ARGUMENT;
+ msg = "s3 obj info err " + proto_to_json(*request);
+ return;
+ }
- auto& obj = request->obj();
- std::string plain_ak = obj.has_ak() ? obj.ak() : "";
- std::string plain_sk = obj.has_sk() ? obj.sk() : "";
+ auto& obj = request->obj();
+ std::string plain_ak = obj.has_ak() ? obj.ak() : "";
+ std::string plain_sk = obj.has_sk() ? obj.sk() : "";
- EncryptionInfoPB encryption_info;
- AkSkPair cipher_ak_sk_pair;
- if (encrypt_ak_sk_helper(plain_ak, plain_sk, &encryption_info, &cipher_ak_sk_pair, code, msg) !=
- 0) {
- return;
- }
- const auto& [ak, sk] = cipher_ak_sk_pair;
- std::string bucket = obj.has_bucket() ? obj.bucket() : "";
- std::string prefix = obj.has_prefix() ? obj.prefix() : "";
- std::string endpoint = obj.has_endpoint() ? obj.endpoint() : "";
- std::string external_endpoint = obj.has_external_endpoint() ? obj.external_endpoint() : "";
- std::string region = obj.has_region() ? obj.region() : "";
+ if (encrypt_ak_sk_helper(plain_ak, plain_sk, &encryption_info, &cipher_ak_sk_pair, code,
+ msg) != 0) {
+ return;
+ }
+ ak = cipher_ak_sk_pair.first;
+ sk = cipher_ak_sk_pair.second;
+ bucket = obj.has_bucket() ? obj.bucket() : "";
+ prefix = obj.has_prefix() ? obj.prefix() : "";
+ endpoint = obj.has_endpoint() ? obj.endpoint() : "";
+ external_endpoint = obj.has_external_endpoint() ? obj.external_endpoint() : "";
+ region = obj.has_region() ? obj.region() : "";
- // obj size > 1k, refuse
- if (obj.ByteSizeLong() > 1024) {
- code = MetaServiceCode::INVALID_ARGUMENT;
- msg = "s3 obj info greater than 1k " + proto_to_json(*request);
- return;
+ // obj size > 1k, refuse
+ if (obj.ByteSizeLong() > 1024) {
+ code = MetaServiceCode::INVALID_ARGUMENT;
+ msg = "s3 obj info greater than 1k " + proto_to_json(*request);
+ return;
+ };
+ } break;
+ case AlterObjStoreInfoRequest::ADD_HDFS_INFO: {
+ if (!request->has_hdfs()) {
+ code = MetaServiceCode::INVALID_ARGUMENT;
+ msg = "hdfs info is not found " + proto_to_json(*request);
+ return;
+ }
+ } break;
+ case AlterObjStoreInfoRequest::UNKNOWN:
+ break;
Review Comment:
any error/warning msgs if we reach this branch?
it seems unexpected
##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,158 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
return;
}
for (auto& obj_info : *instance.mutable_obj_info()) {
- if (obj_info.has_encryption_info()) {
- AkSkPair plain_ak_sk_pair;
- int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
- &plain_ak_sk_pair, code, msg);
- if (ret != 0) return;
- obj_info.set_ak(std::move(plain_ak_sk_pair.first));
- obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+ if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+ return;
+ }
+ }
+ // Iterate all the resources to return to the rpc caller
+ for (const auto& resource_id : instance.resource_ids()) {
+ std::string storage_vault_k;
+ storage_vault_key({instance_id, resource_id}, &storage_vault_k);
+ if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+ txn_kv_.get(), code, msg);
+ ret != 0) {
+ return;
}
}
response->mutable_obj_info()->CopyFrom(instance.obj_info());
}
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {
Review Comment:
lack of comment and UT, this function should be carefully tested.
##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,158 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
return;
}
for (auto& obj_info : *instance.mutable_obj_info()) {
- if (obj_info.has_encryption_info()) {
- AkSkPair plain_ak_sk_pair;
- int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
- &plain_ak_sk_pair, code, msg);
- if (ret != 0) return;
- obj_info.set_ak(std::move(plain_ak_sk_pair.first));
- obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+ if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+ return;
+ }
+ }
+ // Iterate all the resources to return to the rpc caller
+ for (const auto& resource_id : instance.resource_ids()) {
+ std::string storage_vault_k;
+ storage_vault_key({instance_id, resource_id}, &storage_vault_k);
Review Comment:
use the form `key = storage_vault_key()`;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1965976476
<details>
<summary>TPC-H: <b>Total hot run time: 37577 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1cc1d73fd8dea6a92a432e3d2025c19064636813, data reload: false
------ Round 1 ----------------------------------
q1 17627 4074 4033 4033
q2 2043 140 132 132
q3 10581 944 944 944
q4 4846 971 969 969
q5 7957 2830 2978 2830
q6 179 122 120 120
q7 1227 757 734 734
q8 9253 2018 1996 1996
q9 7134 6259 6262 6259
q10 8154 2445 2531 2445
q11 421 207 211 207
q12 752 321 303 303
q13 17946 3187 3187 3187
q14 292 250 238 238
q15 536 501 493 493
q16 475 398 401 398
q17 926 908 842 842
q18 6566 5952 5866 5866
q19 1575 1506 1508 1506
q20 572 275 274 274
q21 6629 3511 3541 3511
q22 783 294 290 290
Total cold run time: 106474 ms
Total hot run time: 37577 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4007 3977 4007 3977
q2 314 212 220 212
q3 2967 2922 2917 2917
q4 1838 1808 1823 1808
q5 5144 5106 5141 5106
q6 197 115 115 115
q7 2142 1726 1709 1709
q8 3125 3176 3182 3176
q9 8229 8223 8190 8190
q10 6213 3646 3636 3636
q11 511 415 438 415
q12 679 554 529 529
q13 4071 3067 3098 3067
q14 286 263 246 246
q15 535 489 489 489
q16 515 447 455 447
q17 1699 1676 1679 1676
q18 7861 7558 7436 7436
q19 1635 1628 1619 1619
q20 2087 1911 1890 1890
q21 4846 4632 4708 4632
q22 520 435 418 418
Total cold run time: 59421 ms
Total hot run time: 53710 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1987688526
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #31415:
URL: https://github.com/apache/doris/pull/31415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1965953170
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1966006471
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 1cc1d73fd8dea6a92a432e3d2025c19064636813 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 16.7 seconds inserted 10000000 Rows, about 598K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1971411301
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1509105037
##########
cloud/test/meta_service_test.cpp:
##########
@@ -4924,6 +4978,110 @@ TEST(MetaServiceTest, LegacyUpdateAkSkTest) {
SyncPoint::get_instance()->clear_all_call_backs();
}
+TEST(MetaServiceTest, AddHdfsInfoTest) {
Review Comment:
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST(MetaServiceTest, AddHdfsInfoTest) {
^
```
<details>
<summary>Additional context</summary>
**cloud/test/meta_service_test.cpp:4980:** 102 lines including whitespace and comments (threshold 80)
```cpp
TEST(MetaServiceTest, AddHdfsInfoTest) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1987689268
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]
Posted by "platoneko (via GitHub)" <gi...@apache.org>.
platoneko commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1517356605
##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,156 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
return;
}
for (auto& obj_info : *instance.mutable_obj_info()) {
- if (obj_info.has_encryption_info()) {
- AkSkPair plain_ak_sk_pair;
- int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
- &plain_ak_sk_pair, code, msg);
- if (ret != 0) return;
- obj_info.set_ak(std::move(plain_ak_sk_pair.first));
- obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+ if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+ return;
+ }
+ }
+ // Iterate all the resources to return to the rpc caller
+ for (const auto& resource_id : instance.resource_ids()) {
+ std::string storage_vault_k = storage_vault_key({instance_id, resource_id});
+ if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+ txn_kv_.get(), code, msg);
+ ret != 0) {
+ return;
}
}
response->mutable_obj_info()->CopyFrom(instance.obj_info());
}
+// The next avaiable vault id would be max(max(obj info id), max(vault id)) + 1.
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {
+ size_t vault_id = 0;
+ auto cmp = [](size_t prev, const auto& last) {
+ size_t last_id = 0;
+ if constexpr (std::is_same_v<std::decay_t<decltype(last)>, ObjectStoreInfoPB>) {
+ last_id = std::stoi(last.id());
Review Comment:
防一手不是 int 的情况
##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,156 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
return;
}
for (auto& obj_info : *instance.mutable_obj_info()) {
- if (obj_info.has_encryption_info()) {
- AkSkPair plain_ak_sk_pair;
- int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
- &plain_ak_sk_pair, code, msg);
- if (ret != 0) return;
- obj_info.set_ak(std::move(plain_ak_sk_pair.first));
- obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+ if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+ return;
+ }
+ }
+ // Iterate all the resources to return to the rpc caller
+ for (const auto& resource_id : instance.resource_ids()) {
+ std::string storage_vault_k = storage_vault_key({instance_id, resource_id});
+ if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
Review Comment:
Use range get
##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,156 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
return;
}
for (auto& obj_info : *instance.mutable_obj_info()) {
- if (obj_info.has_encryption_info()) {
- AkSkPair plain_ak_sk_pair;
- int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
- &plain_ak_sk_pair, code, msg);
- if (ret != 0) return;
- obj_info.set_ak(std::move(plain_ak_sk_pair.first));
- obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+ if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+ return;
+ }
+ }
+ // Iterate all the resources to return to the rpc caller
+ for (const auto& resource_id : instance.resource_ids()) {
+ std::string storage_vault_k = storage_vault_key({instance_id, resource_id});
+ if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+ txn_kv_.get(), code, msg);
+ ret != 0) {
+ return;
}
}
response->mutable_obj_info()->CopyFrom(instance.obj_info());
}
+// The next avaiable vault id would be max(max(obj info id), max(vault id)) + 1.
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {
+ size_t vault_id = 0;
+ auto cmp = [](size_t prev, const auto& last) {
+ size_t last_id = 0;
+ if constexpr (std::is_same_v<std::decay_t<decltype(last)>, ObjectStoreInfoPB>) {
+ last_id = std::stoi(last.id());
+ } else if constexpr (std::is_same_v<std::decay_t<decltype(last)>, std::string>) {
+ last_id = std::stoi(last);
+ }
+ return std::max(prev, last_id);
+ };
+ auto prev = std::accumulate(
+ instance.resource_ids().begin(), instance.resource_ids().end(),
+ std::accumulate(instance.obj_info().begin(), instance.obj_info().end(), vault_id, cmp),
+ cmp);
+ return std::to_string(prev + 1);
+}
+
+static int add_hdfs_storage_valut(InstanceInfoPB& instance, TxnKv* txn_kv,
Review Comment:
```suggestion
static int add_hdfs_storage_valut(InstanceInfoPB& instance, Transaction* txn,
```
##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,156 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
return;
}
for (auto& obj_info : *instance.mutable_obj_info()) {
- if (obj_info.has_encryption_info()) {
- AkSkPair plain_ak_sk_pair;
- int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
- &plain_ak_sk_pair, code, msg);
- if (ret != 0) return;
- obj_info.set_ak(std::move(plain_ak_sk_pair.first));
- obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+ if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+ return;
+ }
+ }
+ // Iterate all the resources to return to the rpc caller
+ for (const auto& resource_id : instance.resource_ids()) {
+ std::string storage_vault_k = storage_vault_key({instance_id, resource_id});
+ if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+ txn_kv_.get(), code, msg);
+ ret != 0) {
+ return;
}
}
response->mutable_obj_info()->CopyFrom(instance.obj_info());
}
+// The next avaiable vault id would be max(max(obj info id), max(vault id)) + 1.
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {
Review Comment:
typo `avaiable`
##########
gensrc/proto/cloud.proto:
##########
@@ -691,16 +717,24 @@ message GetObjStoreInfoRequest {
optional string cloud_unique_id = 1; // For auth
};
+message AlterHdfsVaultInfo {
+ optional string vault_name = 1;
+ optional HdfsVaultInfo hdfs = 2;
+}
+
message AlterObjStoreInfoRequest {
enum Operation {
UNKNOWN = 0;
UPDATE_AK_SK = 1;
ADD_OBJ_INFO = 2;
LEGACY_UPDATE_AK_SK = 3;
+
+ ADD_HDFS_INFO = 100;
}
optional string cloud_unique_id = 1; // For auth
optional ObjectStoreInfoPB obj = 2;
optional Operation op = 3;
+ optional AlterHdfsVaultInfo hdfs = 4;
Review Comment:
直接用 `StorageVaultPB` 吧
##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,156 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
return;
}
for (auto& obj_info : *instance.mutable_obj_info()) {
- if (obj_info.has_encryption_info()) {
- AkSkPair plain_ak_sk_pair;
- int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
- &plain_ak_sk_pair, code, msg);
- if (ret != 0) return;
- obj_info.set_ak(std::move(plain_ak_sk_pair.first));
- obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+ if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+ return;
+ }
+ }
+ // Iterate all the resources to return to the rpc caller
+ for (const auto& resource_id : instance.resource_ids()) {
+ std::string storage_vault_k = storage_vault_key({instance_id, resource_id});
+ if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+ txn_kv_.get(), code, msg);
+ ret != 0) {
+ return;
}
}
response->mutable_obj_info()->CopyFrom(instance.obj_info());
}
+// The next avaiable vault id would be max(max(obj info id), max(vault id)) + 1.
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {
+ size_t vault_id = 0;
+ auto cmp = [](size_t prev, const auto& last) {
Review Comment:
```suggestion
auto max = [](size_t prev, const auto& last) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org