You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wsjz (via GitHub)" <gi...@apache.org> on 2024/02/26 03:09:23 UTC
[PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
wsjz opened a new pull request, #31390:
URL: https://github.com/apache/doris/pull/31390
## Proposed changes
add max compute custom odps and tunnel url
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970360998
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970416763
<details>
<summary>TPC-DS: <b>Total hot run time: 169514 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7, data reload: false
query1 943 347 337 337
query2 6517 1769 1845 1769
query3 6699 217 204 204
query4 23076 20573 20594 20573
query5 4322 368 353 353
query6 262 178 182 178
query7 4598 309 289 289
query8 264 193 204 193
query9 8449 2239 2218 2218
query10 423 224 229 224
query11 14573 14201 14108 14108
query12 138 80 82 80
query13 1640 429 426 426
query14 8414 6251 6662 6251
query15 217 181 184 181
query16 7096 265 259 259
query17 1027 522 496 496
query18 1898 303 275 275
query19 196 148 144 144
query20 85 83 84 83
query21 200 133 123 123
query22 4877 4637 4569 4569
query23 30571 29993 30017 29993
query24 11813 3157 3141 3141
query25 670 352 354 352
query26 1888 154 166 154
query27 3066 319 316 316
query28 7043 1823 1804 1804
query29 1264 564 563 563
query30 287 131 152 131
query31 860 689 708 689
query32 95 62 54 54
query33 721 223 217 217
query34 1051 484 481 481
query35 849 751 743 743
query36 976 816 813 813
query37 179 62 61 61
query38 3107 2994 2988 2988
query39 1321 1255 1254 1254
query40 276 104 102 102
query41 39 38 37 37
query42 101 95 104 95
query43 497 435 446 435
query44 1066 703 715 703
query45 193 175 178 175
query46 1031 785 795 785
query47 1534 1478 1437 1437
query48 414 362 339 339
query49 1165 291 279 279
query50 765 368 373 368
query51 4325 4351 4315 4315
query52 101 84 94 84
query53 328 263 263 263
query54 293 224 222 222
query55 83 86 87 86
query56 220 202 194 194
query57 942 872 908 872
query58 217 194 201 194
query59 2238 2160 2215 2160
query60 237 218 205 205
query61 92 86 86 86
query62 587 368 380 368
query63 277 258 257 257
query64 6592 3023 3137 3023
query65 3289 3236 3231 3231
query66 1431 323 315 315
query67 14485 14279 14239 14239
query68 4992 572 558 558
query69 527 374 368 368
query70 1221 1219 1189 1189
query71 400 242 275 242
query72 6229 2829 2673 2673
query73 708 314 321 314
query74 6288 6042 5949 5949
query75 3021 2450 2395 2395
query76 2846 1033 1146 1033
query77 324 243 234 234
query78 9089 8647 8379 8379
query79 1624 531 511 511
query80 1098 357 344 344
query81 504 195 196 195
query82 949 86 87 86
query83 229 123 122 122
query84 276 78 82 78
query85 1368 376 354 354
query86 424 275 283 275
query87 3332 3086 3124 3086
query88 2692 2244 2233 2233
query89 384 318 322 318
query90 1964 163 163 163
query91 158 123 126 123
query92 59 49 46 46
query93 1827 532 512 512
query94 1314 187 180 180
query95 439 334 344 334
query96 575 260 258 258
query97 4303 4145 4147 4145
query98 219 204 192 192
query99 1104 669 703 669
Total cold run time: 264980 ms
Total hot run time: 169514 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970356799
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970468480
TeamCity be ut coverage result:
Function Coverage: 35.68% (8548/23958)
Line Coverage: 27.49% (69387/252421)
Region Coverage: 26.64% (35980/135070)
Branch Coverage: 23.45% (18393/78430)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7_a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1963241231
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970357240
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #31390:
URL: https://github.com/apache/doris/pull/31390#discussion_r1505222600
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -1974,6 +1974,18 @@ public class Config extends ConfigBase {
@ConfField(mutable = true)
public static boolean enable_query_hive_views = true;
+ /**
+ * ODPS server url of max compute, set it to replace default url.
+ */
+ @ConfField(mutable = true)
+ public static String max_compute_odps_url = "";
Review Comment:
No need, remove these 2 config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970557839
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #31390:
URL: https://github.com/apache/doris/pull/31390
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970402907
<details>
<summary>TPC-H: <b>Total hot run time: 38056 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7, data reload: false
------ Round 1 ----------------------------------
q1 17685 4115 4086 4086
q2 2050 144 134 134
q3 10639 943 943 943
q4 4659 951 960 951
q5 7755 2866 3012 2866
q6 182 120 121 120
q7 1247 773 770 770
q8 9316 2054 2036 2036
q9 7265 6289 6289 6289
q10 8227 2518 2518 2518
q11 422 198 196 196
q12 722 318 306 306
q13 17929 3217 3235 3217
q14 278 264 269 264
q15 529 505 499 499
q16 498 397 399 397
q17 953 853 826 826
q18 6780 6078 6074 6074
q19 1546 1505 1517 1505
q20 553 283 282 282
q21 6358 3503 3580 3503
q22 805 292 274 274
Total cold run time: 106398 ms
Total hot run time: 38056 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4006 4005 4041 4005
q2 309 214 214 214
q3 2983 2943 2954 2943
q4 1849 1842 1826 1826
q5 5154 5167 5208 5167
q6 202 115 115 115
q7 2201 1718 1723 1718
q8 3140 3208 3227 3208
q9 8239 8181 8192 8181
q10 6099 3661 3670 3661
q11 510 444 427 427
q12 696 541 524 524
q13 8676 3050 3104 3050
q14 268 269 258 258
q15 539 504 490 490
q16 522 462 470 462
q17 1712 1687 1700 1687
q18 7936 7511 7474 7474
q19 1674 1642 1639 1639
q20 2104 1900 1907 1900
q21 4787 4630 4772 4630
q22 523 411 448 411
Total cold run time: 64129 ms
Total hot run time: 53990 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970423685
<details>
<summary>ClickBench: <b>Total hot run time: 32.18 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.03
query3 0.23 0.06 0.06
query4 1.67 0.09 0.10
query5 0.52 0.52 0.52
query6 1.29 0.67 0.71
query7 0.02 0.02 0.01
query8 0.03 0.02 0.02
query9 0.55 0.53 0.52
query10 0.56 0.56 0.58
query11 0.13 0.09 0.09
query12 0.12 0.11 0.11
query13 0.64 0.61 0.60
query14 0.80 0.82 0.78
query15 0.85 0.82 0.82
query16 0.37 0.36 0.39
query17 0.99 0.95 0.99
query18 0.25 0.26 0.26
query19 1.83 1.74 1.74
query20 0.01 0.01 0.02
query21 15.40 0.67 0.68
query22 3.28 3.26 2.49
query23 17.22 1.13 1.06
query24 2.16 0.35 0.19
query25 0.30 0.03 0.03
query26 0.15 0.13 0.14
query27 0.04 0.03 0.04
query28 12.54 0.85 0.84
query29 12.73 3.28 3.21
query30 0.60 0.53 0.60
query31 2.81 0.34 0.35
query32 3.36 0.49 0.48
query33 3.24 3.26 3.20
query34 15.39 4.78 4.73
query35 4.74 4.76 4.71
query36 1.14 1.08 1.07
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.03 0.01 0.01
query40 0.16 0.12 0.12
query41 0.07 0.02 0.01
query42 0.02 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 106.47 s
Total hot run time: 32.18 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970428047
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit a551560b6e8eb0ccd2466e76ad59ba86bdff1eb7 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 18.1 seconds inserted 10000000 Rows, about 552K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31390:
URL: https://github.com/apache/doris/pull/31390#discussion_r1502002159
##########
be/src/runtime/descriptors.h:
##########
@@ -238,6 +238,8 @@
const std::string region() const { return _region; }
const std::string project() const { return _project; }
const std::string table() const { return _table; }
+ const std::string odps_url() const { return _odps_url; }
+ const std::string tunnel_url() const { return _tunnel_url; }
Review Comment:
warning: return type 'const int' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]
```suggestion
std::string tunnel_url() const { return _tunnel_url; }
```
##########
be/src/runtime/descriptors.h:
##########
@@ -238,6 +238,8 @@ class MaxComputeTableDescriptor : public TableDescriptor {
const std::string region() const { return _region; }
const std::string project() const { return _project; }
const std::string table() const { return _table; }
+ const std::string odps_url() const { return _odps_url; }
Review Comment:
warning: return type 'const int' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]
```suggestion
std::string odps_url() const { return _odps_url; }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31390:
URL: https://github.com/apache/doris/pull/31390#issuecomment-1970357208
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-catalog) add max compute custom odps and tunnel url [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #31390:
URL: https://github.com/apache/doris/pull/31390#discussion_r1507065893
##########
be/src/runtime/descriptors.h:
##########
@@ -238,6 +238,8 @@ class MaxComputeTableDescriptor : public TableDescriptor {
const std::string region() const { return _region; }
const std::string project() const { return _project; }
const std::string table() const { return _table; }
+ const std::string odps_url() const { return _odps_url; }
Review Comment:
ignore
##########
be/src/runtime/descriptors.h:
##########
@@ -238,6 +238,8 @@
const std::string region() const { return _region; }
const std::string project() const { return _project; }
const std::string table() const { return _table; }
+ const std::string odps_url() const { return _odps_url; }
+ const std::string tunnel_url() const { return _tunnel_url; }
Review Comment:
ignore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org