You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ByteYue (via GitHub)" <gi...@apache.org> on 2024/02/26 09:22:45 UTC

[PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

ByteYue opened a new pull request, #31415:
URL: https://github.com/apache/doris/pull/31415

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1965993083

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 169055 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 1cc1d73fd8dea6a92a432e3d2025c19064636813, data reload: false
   
   query1	939	344	339	339
   query2	6521	1749	1842	1749
   query3	6722	201	200	200
   query4	23573	20594	20603	20594
   query5	4348	361	349	349
   query6	256	174	176	174
   query7	4627	306	304	304
   query8	270	199	197	197
   query9	8338	2294	2272	2272
   query10	421	211	219	211
   query11	14562	14269	14077	14077
   query12	137	83	84	83
   query13	1651	421	442	421
   query14	8727	6135	6313	6135
   query15	216	178	179	178
   query16	7087	256	250	250
   query17	1056	522	508	508
   query18	1909	269	263	263
   query19	195	151	156	151
   query20	87	83	85	83
   query21	198	116	126	116
   query22	4820	4594	4598	4594
   query23	30447	29923	29996	29923
   query24	11206	3147	3097	3097
   query25	622	352	363	352
   query26	1685	156	165	156
   query27	3044	336	317	317
   query28	7014	1830	1816	1816
   query29	1108	563	557	557
   query30	280	136	140	136
   query31	870	689	693	689
   query32	93	60	57	57
   query33	725	228	225	225
   query34	1030	487	499	487
   query35	865	750	744	744
   query36	941	890	848	848
   query37	136	63	63	63
   query38	3105	2978	2998	2978
   query39	1333	1282	1255	1255
   query40	287	106	103	103
   query41	41	36	36	36
   query42	100	103	102	102
   query43	442	427	434	427
   query44	1078	704	721	704
   query45	197	183	171	171
   query46	1038	795	761	761
   query47	1441	1493	1444	1444
   query48	413	347	372	347
   query49	1171	293	298	293
   query50	770	378	388	378
   query51	4327	4330	4284	4284
   query52	97	96	91	91
   query53	334	271	268	268
   query54	309	237	232	232
   query55	83	80	78	78
   query56	232	213	201	201
   query57	951	879	883	879
   query58	219	196	205	196
   query59	2199	2135	2125	2125
   query60	253	219	219	219
   query61	89	83	84	83
   query62	615	378	363	363
   query63	288	258	251	251
   query64	6602	3012	3148	3012
   query65	3238	3231	3241	3231
   query66	1430	323	320	320
   query67	14378	13876	14056	13876
   query68	4992	555	556	555
   query69	552	372	379	372
   query70	1188	1222	1202	1202
   query71	400	247	276	247
   query72	6215	2802	2663	2663
   query73	703	322	326	322
   query74	6334	6003	5929	5929
   query75	3070	2408	2411	2408
   query76	2778	1017	1164	1017
   query77	354	237	255	237
   query78	9130	8512	8408	8408
   query79	1303	541	552	541
   query80	970	366	356	356
   query81	472	196	198	196
   query82	955	90	87	87
   query83	223	122	123	122
   query84	283	82	76	76
   query85	1321	354	354	354
   query86	373	294	298	294
   query87	3303	3101	3049	3049
   query88	2782	2353	2335	2335
   query89	383	328	340	328
   query90	1856	159	170	159
   query91	150	119	124	119
   query92	58	49	49	49
   query93	1237	521	500	500
   query94	1276	177	176	176
   query95	442	333	342	333
   query96	580	267	271	267
   query97	4254	4096	4153	4096
   query98	231	204	203	203
   query99	1059	635	632	632
   Total cold run time: 263013 ms
   Total hot run time: 169055 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1987689228

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1503735213


##########
cloud/src/meta-service/keys.cpp:
##########
@@ -115,11 +115,12 @@ static void encode_prefix(const T& t, std::string* key) {
         RecycleIndexKeyInfo, RecyclePartKeyInfo, RecycleRowsetKeyInfo, RecycleTxnKeyInfo, RecycleStageKeyInfo,
         StatsTabletKeyInfo,
         JobTabletKeyInfo, JobRecycleKeyInfo, RLJobProgressKeyInfo,
-        CopyJobKeyInfo, CopyFileKeyInfo>);
+        CopyJobKeyInfo, CopyFileKeyInfo, StorageVaultKeyInfo>);
 
     key->push_back(CLOUD_USER_KEY_SPACE01);
     // Prefixes for key families
-    if        constexpr (std::is_same_v<T, InstanceKeyInfo>) {
+    if        constexpr (std::is_same_v<T, InstanceKeyInfo>
+                      || std::is_same_v<T, StorageVaultKeyInfo>) {

Review Comment:
   StorageVaultKeyInfo is not int the family of  InstanceKeyInfo.
   it should be 
   0x01 "storage_vault" $instance_id $resource_id



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1971495383

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38111 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit ee0c4da2e607810f875dc1c11ae4aa9434cc8f0f, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17727	4143	4024	4024
   q2	2060	149	147	147
   q3	10623	1002	943	943
   q4	4670	939	963	939
   q5	7639	2960	3064	2960
   q6	196	129	122	122
   q7	1320	828	836	828
   q8	9593	2039	2069	2039
   q9	7374	6499	6499	6499
   q10	8211	2584	2540	2540
   q11	429	222	217	217
   q12	756	317	310	310
   q13	18010	2945	2964	2945
   q14	290	253	247	247
   q15	481	457	465	457
   q16	465	396	404	396
   q17	935	860	886	860
   q18	6762	5921	5875	5875
   q19	1563	1522	1524	1522
   q20	544	273	283	273
   q21	7559	3675	3703	3675
   q22	800	302	293	293
   Total cold run time: 108007 ms
   Total hot run time: 38111 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	3990	4010	4006	4006
   q2	317	220	222	220
   q3	2963	2952	2935	2935
   q4	1853	1858	1822	1822
   q5	5247	5257	5266	5257
   q6	200	120	118	118
   q7	2259	1846	1822	1822
   q8	3194	3300	3271	3271
   q9	8539	8587	8541	8541
   q10	6157	3735	3748	3735
   q11	536	441	440	440
   q12	685	526	550	526
   q13	10805	2785	2835	2785
   q14	281	267	248	248
   q15	476	432	429	429
   q16	463	419	410	410
   q17	1703	1674	1658	1658
   q18	8013	7475	7399	7399
   q19	1630	1625	1624	1624
   q20	1936	1748	1758	1748
   q21	4951	4808	4778	4778
   q22	461	495	472	472
   Total cold run time: 66659 ms
   Total hot run time: 54244 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1964449096

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1973312353

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1972679534

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1985378059

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1972712910

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 32855 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 222d139e2a39b9332643858654815aeaf742cc53, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18321	4494	4533	4494
   q2	2584	227	225	225
   q3	11380	1183	1222	1183
   q4	4927	1255	1280	1255
   q5	7913	2144	2145	2144
   q6	156	122	115	115
   q7	1246	1128	1123	1123
   q8	10043	1092	1082	1082
   q9	6075	5069	4967	4967
   q10	7916	2025	2001	2001
   q11	529	314	316	314
   q12	847	424	422	422
   q13	18080	3045	2968	2968
   q14	346	315	299	299
   q15	385	365	359	359
   q16	543	513	509	509
   q17	751	318	320	318
   q18	5701	5170	4834	4834
   q19	1165	403	410	403
   q20	613	436	461	436
   q21	5953	2960	2945	2945
   q22	527	459	478	459
   Total cold run time: 106001 ms
   Total hot run time: 32855 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4605	4683	4646	4646
   q2	278	215	221	215
   q3	2225	1936	1931	1931
   q4	1687	1393	1357	1357
   q5	3375	3089	3068	3068
   q6	142	104	103	103
   q7	1915	1890	1957	1890
   q8	1762	1655	1646	1646
   q9	5518	5445	5476	5445
   q10	2700	2359	2356	2356
   q11	478	434	408	408
   q12	754	659	672	659
   q13	3116	2645	2853	2645
   q14	303	295	292	292
   q15	366	358	355	355
   q16	496	494	489	489
   q17	1137	757	783	757
   q18	6084	5558	5262	5262
   q19	1098	656	651	651
   q20	1603	1386	1435	1386
   q21	4913	4483	4308	4308
   q22	659	618	562	562
   Total cold run time: 45214 ms
   Total hot run time: 40431 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1508612036


##########
cloud/test/meta_service_test.cpp:
##########
@@ -4924,6 +4978,109 @@ TEST(MetaServiceTest, LegacyUpdateAkSkTest) {
     SyncPoint::get_instance()->clear_all_call_backs();
 }
 
+TEST(MetaServiceTest, AddHdfsInfoTest) {

Review Comment:
   warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
   TEST(MetaServiceTest, AddHdfsInfoTest) {
   ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **cloud/test/meta_service_test.cpp:4980:** 101 lines including whitespace and comments (threshold 80)
   ```cpp
   TEST(MetaServiceTest, AddHdfsInfoTest) {
   ^
   ```
   
   </details>
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1503775189


##########
cloud/test/meta_service_test.cpp:
##########
@@ -4924,6 +4978,108 @@ TEST(MetaServiceTest, LegacyUpdateAkSkTest) {
     SyncPoint::get_instance()->clear_all_call_backs();
 }
 
+TEST(MetaServiceTest, AddHdfsInfoTest) {

Review Comment:
   warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
   TEST(MetaServiceTest, AddHdfsInfoTest) {
   ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **cloud/test/meta_service_test.cpp:4980:** 100 lines including whitespace and comments (threshold 80)
   ```cpp
   TEST(MetaServiceTest, AddHdfsInfoTest) {
   ^
   ```
   
   </details>
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1503775189


##########
cloud/test/meta_service_test.cpp:
##########
@@ -4924,6 +4978,108 @@ TEST(MetaServiceTest, LegacyUpdateAkSkTest) {
     SyncPoint::get_instance()->clear_all_call_backs();
 }
 
+TEST(MetaServiceTest, AddHdfsInfoTest) {

Review Comment:
   warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
   TEST(MetaServiceTest, AddHdfsInfoTest) {
   ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **cloud/test/meta_service_test.cpp:4980:** 100 lines including whitespace and comments (threshold 80)
   ```cpp
   TEST(MetaServiceTest, AddHdfsInfoTest) {
   ^
   ```
   
   </details>
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1963742026

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1965928009

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1503730789


##########
cloud/test/meta_service_test.cpp:
##########
@@ -302,6 +302,60 @@
         sp->disable_processing();
     }
 
+    // case: normal create instance without obj info and hdfs info
+    {
+        brpc::Controller cntl;
+        CreateInstanceRequest req;
+        req.set_instance_id("test_instance");
+        req.set_user_id("test_user");
+        req.set_name("test_name");
+
+        auto sp = SyncPoint::get_instance();
+        sp->set_call_back("encrypt_ak_sk:get_encryption_key_ret",
+                          [](void* p) { *reinterpret_cast<int*>(p) = 0; });
+        sp->set_call_back("encrypt_ak_sk:get_encryption_key",
+                          [](void* p) { *reinterpret_cast<std::string*>(p) = "test"; });
+        sp->set_call_back("encrypt_ak_sk:get_encryption_key_id",
+                          [](void* p) { *reinterpret_cast<int*>(p) = 1; });
+        sp->enable_processing();
+        CreateInstanceResponse res;
+        meta_service->create_instance(reinterpret_cast<::google::protobuf::RpcController*>(&cntl),
+                                      &req, &res, nullptr);
+        ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
+        sp->clear_all_call_backs();
+        sp->clear_trace();
+        sp->disable_processing();
+    }
+
+    // case: normal create instance with hdfs info
+    {
+        brpc::Controller cntl;
+        CreateInstanceRequest req;
+        req.set_instance_id("test_instance");
+        req.set_user_id("test_user");
+        req.set_name("test_name");
+        HdfsParams hdfs;
+        hdfs.set_fs_name("test_name_node");
+        hdfs.set_user("test_user");
+        req.mutable_hdfs_info()->CopyFrom(hdfs);
+
+        auto sp = SyncPoint::get_instance();

Review Comment:
   warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
   
   ```suggestion
           auto *sp = SyncPoint::get_instance();
   ```
   



##########
cloud/test/meta_service_test.cpp:
##########
@@ -302,6 +302,60 @@ TEST(MetaServiceTest, CreateInstanceTest) {
         sp->disable_processing();
     }
 
+    // case: normal create instance without obj info and hdfs info
+    {
+        brpc::Controller cntl;
+        CreateInstanceRequest req;
+        req.set_instance_id("test_instance");
+        req.set_user_id("test_user");
+        req.set_name("test_name");
+
+        auto sp = SyncPoint::get_instance();

Review Comment:
   warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
   
   ```suggestion
           auto *sp = SyncPoint::get_instance();
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1966001490

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 32.12 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 1cc1d73fd8dea6a92a432e3d2025c19064636813, data reload: false
   
   query1	0.03	0.04	0.03
   query2	0.06	0.02	0.02
   query3	0.23	0.06	0.06
   query4	1.68	0.10	0.10
   query5	0.51	0.50	0.52
   query6	1.35	0.67	0.68
   query7	0.02	0.02	0.02
   query8	0.03	0.02	0.03
   query9	0.58	0.53	0.51
   query10	0.57	0.56	0.57
   query11	0.13	0.09	0.10
   query12	0.12	0.10	0.09
   query13	0.61	0.60	0.61
   query14	0.79	0.79	0.80
   query15	0.84	0.85	0.81
   query16	0.38	0.37	0.37
   query17	0.93	0.98	0.99
   query18	0.27	0.24	0.25
   query19	1.90	1.74	1.73
   query20	0.01	0.01	0.01
   query21	15.42	0.63	0.64
   query22	3.10	4.51	2.36
   query23	17.65	1.08	0.95
   query24	2.08	0.49	0.38
   query25	0.47	0.14	0.06
   query26	0.19	0.16	0.17
   query27	0.05	0.06	0.04
   query28	12.07	0.85	0.82
   query29	12.49	3.51	3.17
   query30	0.61	0.55	0.53
   query31	2.78	0.36	0.36
   query32	3.35	0.47	0.48
   query33	3.20	3.23	3.23
   query34	15.35	4.72	4.75
   query35	4.77	4.72	4.74
   query36	1.14	1.07	1.06
   query37	0.06	0.04	0.04
   query38	0.04	0.03	0.03
   query39	0.02	0.02	0.02
   query40	0.16	0.13	0.13
   query41	0.07	0.02	0.01
   query42	0.02	0.02	0.01
   query43	0.03	0.02	0.02
   Total cold run time: 106.16 s
   Total hot run time: 32.12 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1972729405

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 189590 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 222d139e2a39b9332643858654815aeaf742cc53, data reload: false
   
   query1	1057	395	381	381
   query2	7855	2336	2334	2334
   query3	6724	303	292	292
   query4	26622	23943	24269	23943
   query5	4449	861	680	680
   query6	362	330	312	312
   query7	4669	444	446	444
   query8	296	258	247	247
   query9	8409	3394	3356	3356
   query10	549	343	333	333
   query11	19694	20426	19793	19793
   query12	315	232	254	232
   query13	1716	650	656	650
   query14	11492	8229	7604	7604
   query15	349	246	258	246
   query16	8309	395	391	391
   query17	2079	698	664	664
   query18	2204	411	386	386
   query19	332	243	238	238
   query20	145	140	151	140
   query21	236	166	159	159
   query22	5318	5121	5035	5035
   query23	25083	24770	24355	24355
   query24	6671	2068	2058	2058
   query25	614	497	514	497
   query26	785	233	223	223
   query27	2272	493	507	493
   query28	3265	2794	2816	2794
   query29	931	753	782	753
   query30	319	191	205	191
   query31	1411	856	862	856
   query32	138	98	95	95
   query33	669	393	395	393
   query34	890	653	660	653
   query35	970	892	865	865
   query36	1263	1175	1181	1175
   query37	130	115	106	106
   query38	3250	3171	3134	3134
   query39	1170	1133	1133	1133
   query40	251	172	170	170
   query41	90	86	84	84
   query42	149	151	154	151
   query43	598	593	578	578
   query44	1271	1061	1049	1049
   query45	228	223	218	218
   query46	886	790	795	790
   query47	1710	1702	1701	1701
   query48	596	516	520	516
   query49	739	482	491	482
   query50	654	542	553	542
   query51	8888	8980	9048	8980
   query52	156	145	144	144
   query53	334	339	331	331
   query54	372	352	349	349
   query55	176	134	134	134
   query56	356	357	346	346
   query57	1097	1061	1073	1061
   query58	352	329	330	329
   query59	3467	3443	3402	3402
   query60	394	400	378	378
   query61	169	163	172	163
   query62	503	440	405	405
   query63	321	329	318	318
   query64	4266	2181	1842	1842
   query65	2388	2364	2358	2358
   query66	936	392	397	392
   query67	15495	15661	15135	15135
   query68	2925	697	712	697
   query69	650	604	552	552
   query70	1629	1586	1604	1586
   query71	441	423	426	423
   query72	2767	2349	2344	2344
   query73	680	472	481	472
   query74	6772	7085	7204	7085
   query75	3065	2807	2786	2786
   query76	2185	658	639	639
   query77	465	399	396	396
   query78	6809	6293	6460	6293
   query79	1065	742	757	742
   query80	663	594	621	594
   query81	471	259	279	259
   query82	196	135	136	135
   query83	238	244	224	224
   query84	270	138	127	127
   query85	860	521	547	521
   query86	422	424	432	424
   query87	3477	3284	3308	3284
   query88	3759	3430	3445	3430
   query89	442	425	414	414
   query90	1046	219	217	217
   query91	218	200	210	200
   query92	83	80	81	80
   query93	1029	724	696	696
   query94	495	270	275	270
   query95	583	521	501	501
   query96	631	380	384	380
   query97	2917	2820	2793	2793
   query98	307	321	298	298
   query99	849	754	746	746
   Total cold run time: 258263 ms
   Total hot run time: 189590 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1515676940


##########
cloud/test/meta_service_test.cpp:
##########
@@ -4924,6 +4980,110 @@ TEST(MetaServiceTest, LegacyUpdateAkSkTest) {
     SyncPoint::get_instance()->clear_all_call_backs();
 }
 
+TEST(MetaServiceTest, AddHdfsInfoTest) {

Review Comment:
   warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
   TEST(MetaServiceTest, AddHdfsInfoTest) {
   ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **cloud/test/meta_service_test.cpp:4982:** 102 lines including whitespace and comments (threshold 80)
   ```cpp
   TEST(MetaServiceTest, AddHdfsInfoTest) {
   ^
   ```
   
   </details>
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1963657628

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1972736191

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 26.85 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 222d139e2a39b9332643858654815aeaf742cc53, data reload: false
   
   query1	0.06	0.04	0.04
   query2	0.09	0.04	0.04
   query3	0.25	0.06	0.06
   query4	1.60	0.08	0.08
   query5	0.50	0.51	0.51
   query6	1.46	0.97	0.96
   query7	0.03	0.02	0.02
   query8	0.07	0.04	0.04
   query9	0.68	0.60	0.66
   query10	0.68	0.69	0.68
   query11	0.20	0.16	0.14
   query12	0.18	0.15	0.15
   query13	0.48	0.47	0.48
   query14	0.74	0.71	0.72
   query15	0.91	0.86	0.86
   query16	0.38	0.40	0.40
   query17	1.17	1.17	1.18
   query18	0.31	0.34	0.36
   query19	2.20	2.10	2.02
   query20	0.02	0.02	0.02
   query21	15.36	1.01	0.91
   query22	0.58	0.58	0.57
   query23	13.72	0.64	0.61
   query24	0.60	0.20	0.18
   query25	0.04	0.04	0.05
   query26	0.21	0.19	0.19
   query27	0.05	0.05	0.05
   query28	1.32	0.69	0.69
   query29	12.64	4.96	4.72
   query30	0.82	0.75	0.75
   query31	0.87	0.40	0.40
   query32	3.22	0.39	0.40
   query33	2.26	2.25	2.32
   query34	2.26	2.21	2.21
   query35	2.22	2.21	2.23
   query36	0.93	0.99	1.00
   query37	0.08	0.06	0.07
   query38	0.05	0.05	0.04
   query39	0.04	0.02	0.03
   query40	0.18	0.21	0.17
   query41	0.08	0.04	0.02
   query42	0.03	0.04	0.03
   query43	0.05	0.04	0.04
   Total cold run time: 69.62 s
   Total hot run time: 26.85 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1503746238


##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,158 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
         return;
     }
     for (auto& obj_info : *instance.mutable_obj_info()) {
-        if (obj_info.has_encryption_info()) {
-            AkSkPair plain_ak_sk_pair;
-            int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
-                                           &plain_ak_sk_pair, code, msg);
-            if (ret != 0) return;
-            obj_info.set_ak(std::move(plain_ak_sk_pair.first));
-            obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+        if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+            return;
+        }
+    }
+    // Iterate all the resources to return to the rpc caller
+    for (const auto& resource_id : instance.resource_ids()) {
+        std::string storage_vault_k;
+        storage_vault_key({instance_id, resource_id}, &storage_vault_k);
+        if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+                                               txn_kv_.get(), code, msg);
+            ret != 0) {
+            return;
         }
     }
     response->mutable_obj_info()->CopyFrom(instance.obj_info());
 }
 
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {
+    std::string vault_id = "1";
+    auto cmp = [](const std::string& prev, const auto& last) {
+        size_t prev_value = std::stoi(prev);
+        size_t last_id = 0;
+        if constexpr (std::is_same_v<decltype(last), ObjectStoreInfoPB>) {
+            last_id = std::stoi(last.id());
+        } else if constexpr (std::is_same_v<decltype(last), std::string>) {
+            last_id = std::stoi(last);
+        } else {
+            static_assert("Should never come here");
+        }
+
+        return prev_value > last_id ? prev : std::to_string(last_id);
+    };
+    ;
+    return std::accumulate(
+            instance.resource_ids().begin(), instance.resource_ids().end(),
+            std::accumulate(instance.obj_info().begin(), instance.obj_info().end(), vault_id, cmp),
+            cmp);
+}
+
+static int add_hdfs_storage_valut(InstanceInfoPB& instance, TxnKv* txn_kv,
+                                  const AlterHdfsParams& hdfs_param, MetaServiceCode& code,
+                                  std::string& msg) {
+    std::string key;
+    std::string vault_id = next_avaiable_vault_id(instance);
+    storage_vault_key({instance.instance_id(), vault_id}, &key);
+    StorageVaultPB vault;
+    vault.set_id(vault_id);
+    vault.set_name(hdfs_param.vault_name());
+    *vault.mutable_hdfs_infos() = hdfs_param.hdfs();
+    std::string val = vault.SerializeAsString();
+    std::unique_ptr<Transaction> txn;
+    TxnErrorCode err = txn_kv->create_txn(&txn);
+    if (err != TxnErrorCode::TXN_OK) {
+        code = cast_as<ErrCategory::CREATE>(err);
+        msg = "failed to create txn";
+        LOG(WARNING) << msg << " err=" << err;
+        return -1;
+    }
+    err = txn->get(key, &val);
+    LOG(INFO) << "get instance_key=" << hex(key);
+
+    if (err != TxnErrorCode::TXN_OK) {
+        code = cast_as<ErrCategory::READ>(err);
+        msg = fmt::format("failed to get instance, instance_id={}, err={}", instance.instance_id(),
+                          err);
+        return -1;
+    }
+    txn->put(key, val);
+    if (err != TxnErrorCode::TXN_OK) {
+        code = cast_as<ErrCategory::COMMIT>(err);
+        msg = fmt::format("failed to commit for putting storage vault_id={}, vault_name={}, err={}",
+                          vault_id, hdfs_param.vault_name(), err);
+        return -1;
+    }
+    instance.mutable_resource_ids()->Add(std::move(vault_id));
+    *instance.mutable_storage_vault_names()->Add() = hdfs_param.vault_name();
+    return 0;
+}
+
+[[maybe_unused]] static int remove_hdfs_storage_valut(std::string_view vault_key, TxnKv* txn_kv,
+                                                      MetaServiceCode& code, std::string& msg) {
+    std::unique_ptr<Transaction> txn;
+    TxnErrorCode err = txn_kv->create_txn(&txn);
+    if (err != TxnErrorCode::TXN_OK) {
+        code = cast_as<ErrCategory::CREATE>(err);
+        msg = "failed to create txn";
+        LOG(WARNING) << msg << " err=" << err;
+        return -1;
+    }
+    txn->remove(vault_key);
+    LOG(INFO) << "remove storage_vault_key=" << hex(vault_key);
+
+    if (err != TxnErrorCode::TXN_OK) {
+        code = cast_as<ErrCategory::COMMIT>(err);
+        msg = fmt::format("failed to commit for removing storage vault_key={}, err={}", vault_key,
+                          err);
+        return -1;
+    }
+    return 0;
+}
+
 void MetaServiceImpl::alter_obj_store_info(google::protobuf::RpcController* controller,
                                            const AlterObjStoreInfoRequest* request,
                                            AlterObjStoreInfoResponse* response,
                                            ::google::protobuf::Closure* done) {
+    std::string ak, sk, bucket, prefix, endpoint, external_endpoint, region;
+    EncryptionInfoPB encryption_info;
+    AkSkPair cipher_ak_sk_pair;
     RPC_PREPROCESS(alter_obj_store_info);
-    // Prepare data
-    if (!request->has_obj() || !request->obj().has_ak() || !request->obj().has_sk()) {
-        code = MetaServiceCode::INVALID_ARGUMENT;
-        msg = "s3 obj info err " + proto_to_json(*request);
-        return;
-    }
+    switch (request->op()) {
+    case AlterObjStoreInfoRequest::ADD_OBJ_INFO:
+    case AlterObjStoreInfoRequest::LEGACY_UPDATE_AK_SK:
+    case AlterObjStoreInfoRequest::UPDATE_AK_SK: {
+        // Prepare data
+        if (!request->has_obj() || !request->obj().has_ak() || !request->obj().has_sk()) {
+            code = MetaServiceCode::INVALID_ARGUMENT;
+            msg = "s3 obj info err " + proto_to_json(*request);
+            return;
+        }
 
-    auto& obj = request->obj();
-    std::string plain_ak = obj.has_ak() ? obj.ak() : "";
-    std::string plain_sk = obj.has_sk() ? obj.sk() : "";
+        auto& obj = request->obj();
+        std::string plain_ak = obj.has_ak() ? obj.ak() : "";
+        std::string plain_sk = obj.has_sk() ? obj.sk() : "";
 
-    EncryptionInfoPB encryption_info;
-    AkSkPair cipher_ak_sk_pair;
-    if (encrypt_ak_sk_helper(plain_ak, plain_sk, &encryption_info, &cipher_ak_sk_pair, code, msg) !=
-        0) {
-        return;
-    }
-    const auto& [ak, sk] = cipher_ak_sk_pair;
-    std::string bucket = obj.has_bucket() ? obj.bucket() : "";
-    std::string prefix = obj.has_prefix() ? obj.prefix() : "";
-    std::string endpoint = obj.has_endpoint() ? obj.endpoint() : "";
-    std::string external_endpoint = obj.has_external_endpoint() ? obj.external_endpoint() : "";
-    std::string region = obj.has_region() ? obj.region() : "";
+        if (encrypt_ak_sk_helper(plain_ak, plain_sk, &encryption_info, &cipher_ak_sk_pair, code,
+                                 msg) != 0) {
+            return;
+        }
+        ak = cipher_ak_sk_pair.first;
+        sk = cipher_ak_sk_pair.second;
+        bucket = obj.has_bucket() ? obj.bucket() : "";
+        prefix = obj.has_prefix() ? obj.prefix() : "";
+        endpoint = obj.has_endpoint() ? obj.endpoint() : "";
+        external_endpoint = obj.has_external_endpoint() ? obj.external_endpoint() : "";
+        region = obj.has_region() ? obj.region() : "";
 
-    //  obj size > 1k, refuse
-    if (obj.ByteSizeLong() > 1024) {
-        code = MetaServiceCode::INVALID_ARGUMENT;
-        msg = "s3 obj info greater than 1k " + proto_to_json(*request);
-        return;
+        //  obj size > 1k, refuse
+        if (obj.ByteSizeLong() > 1024) {
+            code = MetaServiceCode::INVALID_ARGUMENT;
+            msg = "s3 obj info greater than 1k " + proto_to_json(*request);
+            return;
+        };
+    } break;
+    case AlterObjStoreInfoRequest::ADD_HDFS_INFO: {
+        if (!request->has_hdfs()) {
+            code = MetaServiceCode::INVALID_ARGUMENT;
+            msg = "hdfs info is not found " + proto_to_json(*request);
+            return;
+        }
+    } break;
+    case AlterObjStoreInfoRequest::UNKNOWN:
+        break;

Review Comment:
   any error/warning msgs if we reach this branch?
   it seems unexpected



##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,158 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
         return;
     }
     for (auto& obj_info : *instance.mutable_obj_info()) {
-        if (obj_info.has_encryption_info()) {
-            AkSkPair plain_ak_sk_pair;
-            int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
-                                           &plain_ak_sk_pair, code, msg);
-            if (ret != 0) return;
-            obj_info.set_ak(std::move(plain_ak_sk_pair.first));
-            obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+        if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+            return;
+        }
+    }
+    // Iterate all the resources to return to the rpc caller
+    for (const auto& resource_id : instance.resource_ids()) {
+        std::string storage_vault_k;
+        storage_vault_key({instance_id, resource_id}, &storage_vault_k);
+        if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+                                               txn_kv_.get(), code, msg);
+            ret != 0) {
+            return;
         }
     }
     response->mutable_obj_info()->CopyFrom(instance.obj_info());
 }
 
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {

Review Comment:
   lack of comment and UT, this function should be carefully tested.



##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,158 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
         return;
     }
     for (auto& obj_info : *instance.mutable_obj_info()) {
-        if (obj_info.has_encryption_info()) {
-            AkSkPair plain_ak_sk_pair;
-            int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
-                                           &plain_ak_sk_pair, code, msg);
-            if (ret != 0) return;
-            obj_info.set_ak(std::move(plain_ak_sk_pair.first));
-            obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+        if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+            return;
+        }
+    }
+    // Iterate all the resources to return to the rpc caller
+    for (const auto& resource_id : instance.resource_ids()) {
+        std::string storage_vault_k;
+        storage_vault_key({instance_id, resource_id}, &storage_vault_k);

Review Comment:
   use the form  `key = storage_vault_key()`;



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1965976476

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37577 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 1cc1d73fd8dea6a92a432e3d2025c19064636813, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17627	4074	4033	4033
   q2	2043	140	132	132
   q3	10581	944	944	944
   q4	4846	971	969	969
   q5	7957	2830	2978	2830
   q6	179	122	120	120
   q7	1227	757	734	734
   q8	9253	2018	1996	1996
   q9	7134	6259	6262	6259
   q10	8154	2445	2531	2445
   q11	421	207	211	207
   q12	752	321	303	303
   q13	17946	3187	3187	3187
   q14	292	250	238	238
   q15	536	501	493	493
   q16	475	398	401	398
   q17	926	908	842	842
   q18	6566	5952	5866	5866
   q19	1575	1506	1508	1506
   q20	572	275	274	274
   q21	6629	3511	3541	3511
   q22	783	294	290	290
   Total cold run time: 106474 ms
   Total hot run time: 37577 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4007	3977	4007	3977
   q2	314	212	220	212
   q3	2967	2922	2917	2917
   q4	1838	1808	1823	1808
   q5	5144	5106	5141	5106
   q6	197	115	115	115
   q7	2142	1726	1709	1709
   q8	3125	3176	3182	3176
   q9	8229	8223	8190	8190
   q10	6213	3646	3636	3636
   q11	511	415	438	415
   q12	679	554	529	529
   q13	4071	3067	3098	3067
   q14	286	263	246	246
   q15	535	489	489	489
   q16	515	447	455	447
   q17	1699	1676	1679	1676
   q18	7861	7558	7436	7436
   q19	1635	1628	1619	1619
   q20	2087	1911	1890	1890
   q21	4846	4632	4708	4632
   q22	520	435	418	418
   Total cold run time: 59421 ms
   Total hot run time: 53710 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1987688526

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #31415:
URL: https://github.com/apache/doris/pull/31415


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1965953170

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1966006471

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 1cc1d73fd8dea6a92a432e3d2025c19064636813 with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
   Insert into select:       16.7 seconds inserted 10000000 Rows, about 598K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1971411301

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1509105037


##########
cloud/test/meta_service_test.cpp:
##########
@@ -4924,6 +4978,110 @@ TEST(MetaServiceTest, LegacyUpdateAkSkTest) {
     SyncPoint::get_instance()->clear_all_call_backs();
 }
 
+TEST(MetaServiceTest, AddHdfsInfoTest) {

Review Comment:
   warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
   TEST(MetaServiceTest, AddHdfsInfoTest) {
   ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **cloud/test/meta_service_test.cpp:4980:** 102 lines including whitespace and comments (threshold 80)
   ```cpp
   TEST(MetaServiceTest, AddHdfsInfoTest) {
   ^
   ```
   
   </details>
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31415:
URL: https://github.com/apache/doris/pull/31415#issuecomment-1987689268

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature-wip](Cloud) Support restore HDFS meta data in Meta Service && Add storage vault property [doris]

Posted by "platoneko (via GitHub)" <gi...@apache.org>.
platoneko commented on code in PR #31415:
URL: https://github.com/apache/doris/pull/31415#discussion_r1517356605


##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,156 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
         return;
     }
     for (auto& obj_info : *instance.mutable_obj_info()) {
-        if (obj_info.has_encryption_info()) {
-            AkSkPair plain_ak_sk_pair;
-            int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
-                                           &plain_ak_sk_pair, code, msg);
-            if (ret != 0) return;
-            obj_info.set_ak(std::move(plain_ak_sk_pair.first));
-            obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+        if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+            return;
+        }
+    }
+    // Iterate all the resources to return to the rpc caller
+    for (const auto& resource_id : instance.resource_ids()) {
+        std::string storage_vault_k = storage_vault_key({instance_id, resource_id});
+        if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+                                               txn_kv_.get(), code, msg);
+            ret != 0) {
+            return;
         }
     }
     response->mutable_obj_info()->CopyFrom(instance.obj_info());
 }
 
+// The next avaiable vault id would be max(max(obj info id), max(vault id)) + 1.
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {
+    size_t vault_id = 0;
+    auto cmp = [](size_t prev, const auto& last) {
+        size_t last_id = 0;
+        if constexpr (std::is_same_v<std::decay_t<decltype(last)>, ObjectStoreInfoPB>) {
+            last_id = std::stoi(last.id());

Review Comment:
   防一手不是 int 的情况



##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,156 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
         return;
     }
     for (auto& obj_info : *instance.mutable_obj_info()) {
-        if (obj_info.has_encryption_info()) {
-            AkSkPair plain_ak_sk_pair;
-            int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
-                                           &plain_ak_sk_pair, code, msg);
-            if (ret != 0) return;
-            obj_info.set_ak(std::move(plain_ak_sk_pair.first));
-            obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+        if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+            return;
+        }
+    }
+    // Iterate all the resources to return to the rpc caller
+    for (const auto& resource_id : instance.resource_ids()) {
+        std::string storage_vault_k = storage_vault_key({instance_id, resource_id});
+        if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,

Review Comment:
   Use range get



##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,156 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
         return;
     }
     for (auto& obj_info : *instance.mutable_obj_info()) {
-        if (obj_info.has_encryption_info()) {
-            AkSkPair plain_ak_sk_pair;
-            int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
-                                           &plain_ak_sk_pair, code, msg);
-            if (ret != 0) return;
-            obj_info.set_ak(std::move(plain_ak_sk_pair.first));
-            obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+        if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+            return;
+        }
+    }
+    // Iterate all the resources to return to the rpc caller
+    for (const auto& resource_id : instance.resource_ids()) {
+        std::string storage_vault_k = storage_vault_key({instance_id, resource_id});
+        if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+                                               txn_kv_.get(), code, msg);
+            ret != 0) {
+            return;
         }
     }
     response->mutable_obj_info()->CopyFrom(instance.obj_info());
 }
 
+// The next avaiable vault id would be max(max(obj info id), max(vault id)) + 1.
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {
+    size_t vault_id = 0;
+    auto cmp = [](size_t prev, const auto& last) {
+        size_t last_id = 0;
+        if constexpr (std::is_same_v<std::decay_t<decltype(last)>, ObjectStoreInfoPB>) {
+            last_id = std::stoi(last.id());
+        } else if constexpr (std::is_same_v<std::decay_t<decltype(last)>, std::string>) {
+            last_id = std::stoi(last);
+        }
+        return std::max(prev, last_id);
+    };
+    auto prev = std::accumulate(
+            instance.resource_ids().begin(), instance.resource_ids().end(),
+            std::accumulate(instance.obj_info().begin(), instance.obj_info().end(), vault_id, cmp),
+            cmp);
+    return std::to_string(prev + 1);
+}
+
+static int add_hdfs_storage_valut(InstanceInfoPB& instance, TxnKv* txn_kv,

Review Comment:
   
   ```suggestion
   static int add_hdfs_storage_valut(InstanceInfoPB& instance, Transaction* txn,
   ```



##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,156 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
         return;
     }
     for (auto& obj_info : *instance.mutable_obj_info()) {
-        if (obj_info.has_encryption_info()) {
-            AkSkPair plain_ak_sk_pair;
-            int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
-                                           &plain_ak_sk_pair, code, msg);
-            if (ret != 0) return;
-            obj_info.set_ak(std::move(plain_ak_sk_pair.first));
-            obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+        if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+            return;
+        }
+    }
+    // Iterate all the resources to return to the rpc caller
+    for (const auto& resource_id : instance.resource_ids()) {
+        std::string storage_vault_k = storage_vault_key({instance_id, resource_id});
+        if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+                                               txn_kv_.get(), code, msg);
+            ret != 0) {
+            return;
         }
     }
     response->mutable_obj_info()->CopyFrom(instance.obj_info());
 }
 
+// The next avaiable vault id would be max(max(obj info id), max(vault id)) + 1.
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {

Review Comment:
   typo `avaiable`



##########
gensrc/proto/cloud.proto:
##########
@@ -691,16 +717,24 @@ message GetObjStoreInfoRequest {
     optional string cloud_unique_id = 1; // For auth
 };
 
+message AlterHdfsVaultInfo {
+    optional string vault_name = 1;
+    optional HdfsVaultInfo hdfs = 2;
+}
+
 message AlterObjStoreInfoRequest {
     enum Operation {
         UNKNOWN         = 0;
         UPDATE_AK_SK    = 1;
         ADD_OBJ_INFO    = 2;
         LEGACY_UPDATE_AK_SK = 3;
+
+        ADD_HDFS_INFO = 100;
     }
     optional string cloud_unique_id = 1; // For auth
     optional ObjectStoreInfoPB obj = 2;
     optional Operation op = 3;
+    optional AlterHdfsVaultInfo hdfs = 4;

Review Comment:
   直接用 `StorageVaultPB` 吧



##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -218,52 +271,156 @@ void MetaServiceImpl::get_obj_store_info(google::protobuf::RpcController* contro
         return;
     }
     for (auto& obj_info : *instance.mutable_obj_info()) {
-        if (obj_info.has_encryption_info()) {
-            AkSkPair plain_ak_sk_pair;
-            int ret = decrypt_ak_sk_helper(obj_info.ak(), obj_info.sk(), obj_info.encryption_info(),
-                                           &plain_ak_sk_pair, code, msg);
-            if (ret != 0) return;
-            obj_info.set_ak(std::move(plain_ak_sk_pair.first));
-            obj_info.set_sk(std::move(plain_ak_sk_pair.second));
+        if (auto ret = decrypt_and_update_ak_sk(obj_info, code, msg); ret != 0) {
+            return;
+        }
+    }
+    // Iterate all the resources to return to the rpc caller
+    for (const auto& resource_id : instance.resource_ids()) {
+        std::string storage_vault_k = storage_vault_key({instance_id, resource_id});
+        if (auto ret = fetch_all_storage_valut(std::move(storage_vault_k), instance_id, response,
+                                               txn_kv_.get(), code, msg);
+            ret != 0) {
+            return;
         }
     }
     response->mutable_obj_info()->CopyFrom(instance.obj_info());
 }
 
+// The next avaiable vault id would be max(max(obj info id), max(vault id)) + 1.
+static std::string next_avaiable_vault_id(const InstanceInfoPB& instance) {
+    size_t vault_id = 0;
+    auto cmp = [](size_t prev, const auto& last) {

Review Comment:
   
   ```suggestion
       auto max = [](size_t prev, const auto& last) {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org