You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wsjz (via GitHub)" <gi...@apache.org> on 2024/02/26 03:09:23 UTC

[PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

wsjz opened a new pull request, #31390:
URL: https://github.com/apache/doris/pull/31390

   ## Proposed changes
   
   add max compute custom odps and tunnel url
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970360998

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970416763

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 169514 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7, data reload: false
   
   query1	943	347	337	337
   query2	6517	1769	1845	1769
   query3	6699	217	204	204
   query4	23076	20573	20594	20573
   query5	4322	368	353	353
   query6	262	178	182	178
   query7	4598	309	289	289
   query8	264	193	204	193
   query9	8449	2239	2218	2218
   query10	423	224	229	224
   query11	14573	14201	14108	14108
   query12	138	80	82	80
   query13	1640	429	426	426
   query14	8414	6251	6662	6251
   query15	217	181	184	181
   query16	7096	265	259	259
   query17	1027	522	496	496
   query18	1898	303	275	275
   query19	196	148	144	144
   query20	85	83	84	83
   query21	200	133	123	123
   query22	4877	4637	4569	4569
   query23	30571	29993	30017	29993
   query24	11813	3157	3141	3141
   query25	670	352	354	352
   query26	1888	154	166	154
   query27	3066	319	316	316
   query28	7043	1823	1804	1804
   query29	1264	564	563	563
   query30	287	131	152	131
   query31	860	689	708	689
   query32	95	62	54	54
   query33	721	223	217	217
   query34	1051	484	481	481
   query35	849	751	743	743
   query36	976	816	813	813
   query37	179	62	61	61
   query38	3107	2994	2988	2988
   query39	1321	1255	1254	1254
   query40	276	104	102	102
   query41	39	38	37	37
   query42	101	95	104	95
   query43	497	435	446	435
   query44	1066	703	715	703
   query45	193	175	178	175
   query46	1031	785	795	785
   query47	1534	1478	1437	1437
   query48	414	362	339	339
   query49	1165	291	279	279
   query50	765	368	373	368
   query51	4325	4351	4315	4315
   query52	101	84	94	84
   query53	328	263	263	263
   query54	293	224	222	222
   query55	83	86	87	86
   query56	220	202	194	194
   query57	942	872	908	872
   query58	217	194	201	194
   query59	2238	2160	2215	2160
   query60	237	218	205	205
   query61	92	86	86	86
   query62	587	368	380	368
   query63	277	258	257	257
   query64	6592	3023	3137	3023
   query65	3289	3236	3231	3231
   query66	1431	323	315	315
   query67	14485	14279	14239	14239
   query68	4992	572	558	558
   query69	527	374	368	368
   query70	1221	1219	1189	1189
   query71	400	242	275	242
   query72	6229	2829	2673	2673
   query73	708	314	321	314
   query74	6288	6042	5949	5949
   query75	3021	2450	2395	2395
   query76	2846	1033	1146	1033
   query77	324	243	234	234
   query78	9089	8647	8379	8379
   query79	1624	531	511	511
   query80	1098	357	344	344
   query81	504	195	196	195
   query82	949	86	87	86
   query83	229	123	122	122
   query84	276	78	82	78
   query85	1368	376	354	354
   query86	424	275	283	275
   query87	3332	3086	3124	3086
   query88	2692	2244	2233	2233
   query89	384	318	322	318
   query90	1964	163	163	163
   query91	158	123	126	123
   query92	59	49	46	46
   query93	1827	532	512	512
   query94	1314	187	180	180
   query95	439	334	344	334
   query96	575	260	258	258
   query97	4303	4145	4147	4145
   query98	219	204	192	192
   query99	1104	669	703	669
   Total cold run time: 264980 ms
   Total hot run time: 169514 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970356799

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970468480

   TeamCity be ut coverage result:
    Function Coverage: 35.68% (8548/23958) 
    Line Coverage: 27.49% (69387/252421)
    Region Coverage: 26.64% (35980/135070)
    Branch Coverage: 23.45% (18393/78430)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7_a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1963241231

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970357240

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #31390:
URL: https://github.com/apache/doris/pull/31390#discussion_r1505222600


##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -1974,6 +1974,18 @@ public class Config extends ConfigBase {
     @ConfField(mutable = true)
     public static boolean enable_query_hive_views = true;
 
+    /**
+     * ODPS server url of max compute, set it to replace default url.
+     */
+    @ConfField(mutable = true)
+    public static String max_compute_odps_url = "";

Review Comment:
   No need, remove these 2 config.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970557839

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #31390:
URL: https://github.com/apache/doris/pull/31390


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970402907

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38056 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17685	4115	4086	4086
   q2	2050	144	134	134
   q3	10639	943	943	943
   q4	4659	951	960	951
   q5	7755	2866	3012	2866
   q6	182	120	121	120
   q7	1247	773	770	770
   q8	9316	2054	2036	2036
   q9	7265	6289	6289	6289
   q10	8227	2518	2518	2518
   q11	422	198	196	196
   q12	722	318	306	306
   q13	17929	3217	3235	3217
   q14	278	264	269	264
   q15	529	505	499	499
   q16	498	397	399	397
   q17	953	853	826	826
   q18	6780	6078	6074	6074
   q19	1546	1505	1517	1505
   q20	553	283	282	282
   q21	6358	3503	3580	3503
   q22	805	292	274	274
   Total cold run time: 106398 ms
   Total hot run time: 38056 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4006	4005	4041	4005
   q2	309	214	214	214
   q3	2983	2943	2954	2943
   q4	1849	1842	1826	1826
   q5	5154	5167	5208	5167
   q6	202	115	115	115
   q7	2201	1718	1723	1718
   q8	3140	3208	3227	3208
   q9	8239	8181	8192	8181
   q10	6099	3661	3670	3661
   q11	510	444	427	427
   q12	696	541	524	524
   q13	8676	3050	3104	3050
   q14	268	269	258	258
   q15	539	504	490	490
   q16	522	462	470	462
   q17	1712	1687	1700	1687
   q18	7936	7511	7474	7474
   q19	1674	1642	1639	1639
   q20	2104	1900	1907	1900
   q21	4787	4630	4772	4630
   q22	523	411	448	411
   Total cold run time: 64129 ms
   Total hot run time: 53990 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970423685

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 32.18 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.06	0.02	0.03
   query3	0.23	0.06	0.06
   query4	1.67	0.09	0.10
   query5	0.52	0.52	0.52
   query6	1.29	0.67	0.71
   query7	0.02	0.02	0.01
   query8	0.03	0.02	0.02
   query9	0.55	0.53	0.52
   query10	0.56	0.56	0.58
   query11	0.13	0.09	0.09
   query12	0.12	0.11	0.11
   query13	0.64	0.61	0.60
   query14	0.80	0.82	0.78
   query15	0.85	0.82	0.82
   query16	0.37	0.36	0.39
   query17	0.99	0.95	0.99
   query18	0.25	0.26	0.26
   query19	1.83	1.74	1.74
   query20	0.01	0.01	0.02
   query21	15.40	0.67	0.68
   query22	3.28	3.26	2.49
   query23	17.22	1.13	1.06
   query24	2.16	0.35	0.19
   query25	0.30	0.03	0.03
   query26	0.15	0.13	0.14
   query27	0.04	0.03	0.04
   query28	12.54	0.85	0.84
   query29	12.73	3.28	3.21
   query30	0.60	0.53	0.60
   query31	2.81	0.34	0.35
   query32	3.36	0.49	0.48
   query33	3.24	3.26	3.20
   query34	15.39	4.78	4.73
   query35	4.74	4.76	4.71
   query36	1.14	1.08	1.07
   query37	0.07	0.05	0.05
   query38	0.04	0.03	0.03
   query39	0.03	0.01	0.01
   query40	0.16	0.12	0.12
   query41	0.07	0.02	0.01
   query42	0.02	0.01	0.02
   query43	0.02	0.02	0.02
   Total cold run time: 106.47 s
   Total hot run time: 32.18 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970428047

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7 with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       18.1 seconds inserted 10000000 Rows, about 552K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31390:
URL: https://github.com/apache/doris/pull/31390#discussion_r1502002159


##########
be/src/runtime/descriptors.h:
##########
@@ -238,6 +238,8 @@
     const std::string region() const { return _region; }
     const std::string project() const { return _project; }
     const std::string table() const { return _table; }
+    const std::string odps_url() const { return _odps_url; }
+    const std::string tunnel_url() const { return _tunnel_url; }

Review Comment:
   warning: return type 'const int' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]
   
   ```suggestion
       std::string tunnel_url() const { return _tunnel_url; }
   ```
   



##########
be/src/runtime/descriptors.h:
##########
@@ -238,6 +238,8 @@ class MaxComputeTableDescriptor : public TableDescriptor {
     const std::string region() const { return _region; }
     const std::string project() const { return _project; }
     const std::string table() const { return _table; }
+    const std::string odps_url() const { return _odps_url; }

Review Comment:
   warning: return type 'const int' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]
   
   ```suggestion
       std::string odps_url() const { return _odps_url; }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970357208

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]

Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #31390:
URL: https://github.com/apache/doris/pull/31390#discussion_r1507065893


##########
be/src/runtime/descriptors.h:
##########
@@ -238,6 +238,8 @@ class MaxComputeTableDescriptor : public TableDescriptor {
     const std::string region() const { return _region; }
     const std::string project() const { return _project; }
     const std::string table() const { return _table; }
+    const std::string odps_url() const { return _odps_url; }

Review Comment:
   ignore



##########
be/src/runtime/descriptors.h:
##########
@@ -238,6 +238,8 @@
     const std::string region() const { return _region; }
     const std::string project() const { return _project; }
     const std::string table() const { return _table; }
+    const std::string odps_url() const { return _odps_url; }
+    const std::string tunnel_url() const { return _tunnel_url; }

Review Comment:
   ignore



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org