You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "luwei16 (via GitHub)" <gi...@apache.org> on 2024/01/24 16:06:31 UTC
[PR] [feature](merge-cloud) add available clusters for load [doris]
luwei16 opened a new pull request, #30340:
URL: https://github.com/apache/doris/pull/30340
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1911337716
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 716bc4f53295f79e4695ffe58c272fb9f37f5bf1 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.4 seconds inserted 10000000 Rows, about 694K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1911297386
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1910573843
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit f922cead339d7c764976df5974eba6f5762c61de with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.4 seconds inserted 10000000 Rows, about 746K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1910484744
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1909509444
<details>
<summary>TPC-H: <b>Total hot run time: 39481 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc5c9118b96d65ceb96c0f57406f700e14c472cf, data reload: false
------ Round 1 ----------------------------------
q1 17598 5497 5375 5375
q2 2050 144 133 133
q3 10648 1160 1145 1145
q4 10263 794 832 794
q5 7816 3302 3235 3235
q6 205 121 122 121
q7 891 500 490 490
q8 9796 1991 1991 1991
q9 7867 6527 6486 6486
q10 8956 3038 3032 3032
q11 417 210 193 193
q12 356 189 187 187
q13 17990 3346 3367 3346
q14 242 214 222 214
q15 555 497 506 497
q16 416 363 380 363
q17 930 559 552 552
q18 7641 7005 6926 6926
q19 1548 1462 1406 1406
q20 584 304 282 282
q21 2799 2439 2423 2423
q22 343 312 290 290
Total cold run time: 109911 ms
Total hot run time: 39481 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5402 5226 5173 5173
q2 330 214 212 212
q3 3330 3246 3225 3225
q4 2032 2062 2057 2057
q5 6076 5959 5892 5892
q6 195 114 116 114
q7 2314 1870 1928 1870
q8 3219 3381 3385 3381
q9 9051 8858 8797 8797
q10 3934 3825 3823 3823
q11 552 457 455 455
q12 813 617 620 617
q13 16922 3186 3160 3160
q14 280 249 256 249
q15 544 502 505 502
q16 510 444 480 444
q17 1920 1812 1827 1812
q18 9604 20019 9677 9677
q19 24197 1622 1526 1526
q20 4619 1964 1928 1928
q21 14394 5398 5337 5337
q22 977 531 547 531
Total cold run time: 111215 ms
Total hot run time: 60782 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1910568455
<details>
<summary>ClickBench: <b>Total hot run time: 30.93 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f922cead339d7c764976df5974eba6f5762c61de, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.03 0.02
query3 0.23 0.05 0.05
query4 1.70 0.06 0.08
query5 0.54 0.52 0.53
query6 1.35 0.65 0.65
query7 0.02 0.01 0.02
query8 0.04 0.02 0.02
query9 0.55 0.50 0.49
query10 0.58 0.55 0.55
query11 0.11 0.09 0.08
query12 0.11 0.09 0.09
query13 0.60 0.61 0.60
query14 0.79 0.81 0.80
query15 0.80 0.77 0.79
query16 0.39 0.37 0.37
query17 0.98 1.01 1.02
query18 0.25 0.25 0.26
query19 1.89 1.80 1.79
query20 0.02 0.01 0.01
query21 15.39 0.58 0.58
query22 2.85 2.51 1.89
query23 17.28 0.90 0.70
query24 2.47 1.20 0.91
query25 0.38 0.15 0.19
query26 0.58 0.13 0.13
query27 0.05 0.06 0.06
query28 11.66 0.79 0.77
query29 12.53 3.23 3.16
query30 0.55 0.49 0.50
query31 2.78 0.34 0.36
query32 3.37 0.49 0.49
query33 3.23 3.22 3.23
query34 15.65 4.30 4.27
query35 4.30 4.28 4.33
query36 1.12 1.07 1.07
query37 0.06 0.05 0.05
query38 0.03 0.02 0.02
query39 0.02 0.02 0.02
query40 0.16 0.13 0.13
query41 0.07 0.01 0.02
query42 0.02 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.62 s
Total hot run time: 30.93 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #30340:
URL: https://github.com/apache/doris/pull/30340#discussion_r1466371096
##########
fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java:
##########
@@ -1034,14 +1037,63 @@ public String getCurrentCloudCluster() {
return cluster;
}
- // Set cloud cluster by `use @clusterName`
public void setCloudCluster(String cluster) {
this.cloudCluster = cluster;
}
- // The returned cluster is set by `use @clusterName`
+ // 1 Use an explicitly specified cluster
+ // 2 If no cluster is specified, the user's default cluster is used.
+ // 3 If the user does not have a default cluster, select a cluster with permissions for the user.
public String getCloudCluster() {
- return cloudCluster;
+ String cluster = null;
+ if (!Strings.isNullOrEmpty(this.cloudCluster)) {
+ cluster = this.cloudCluster;
+ }
+
+ String defaultCluster = getDefaultCloudCluster();
+ if (!Strings.isNullOrEmpty(defaultCluster)) {
+ cluster = defaultCluster;
+ }
+
+ String authorizedCluster = getAuthorizedCloudCluster();
+ if (!Strings.isNullOrEmpty(authorizedCluster)) {
+ cluster = authorizedCluster;
+ }
+
+ if (Strings.isNullOrEmpty(cluster)) {
+ LOG.warn("cant get a valid cluster for user {} to use", getCurrentUserIdentity());
+ getState().setError(ErrorCode.ERR_NO_CLUSTER_ERROR,
+ "Cant get a Valid cluster for you to use, plz connect admin");
+ } else {
+ this.cloudCluster = cluster;
+ LOG.info("finally set context cluster name {}", cloudCluster);
+ }
+
+ return cluster;
+ }
+
+ public String getDefaultCloudCluster() {
Review Comment:
Add comment
##########
fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java:
##########
@@ -1034,14 +1037,63 @@ public String getCurrentCloudCluster() {
return cluster;
}
- // Set cloud cluster by `use @clusterName`
public void setCloudCluster(String cluster) {
this.cloudCluster = cluster;
}
- // The returned cluster is set by `use @clusterName`
+ // 1 Use an explicitly specified cluster
+ // 2 If no cluster is specified, the user's default cluster is used.
+ // 3 If the user does not have a default cluster, select a cluster with permissions for the user.
public String getCloudCluster() {
- return cloudCluster;
+ String cluster = null;
+ if (!Strings.isNullOrEmpty(this.cloudCluster)) {
+ cluster = this.cloudCluster;
+ }
+
+ String defaultCluster = getDefaultCloudCluster();
+ if (!Strings.isNullOrEmpty(defaultCluster)) {
+ cluster = defaultCluster;
+ }
+
+ String authorizedCluster = getAuthorizedCloudCluster();
+ if (!Strings.isNullOrEmpty(authorizedCluster)) {
+ cluster = authorizedCluster;
+ }
+
+ if (Strings.isNullOrEmpty(cluster)) {
+ LOG.warn("cant get a valid cluster for user {} to use", getCurrentUserIdentity());
+ getState().setError(ErrorCode.ERR_NO_CLUSTER_ERROR,
+ "Cant get a Valid cluster for you to use, plz connect admin");
+ } else {
+ this.cloudCluster = cluster;
+ LOG.info("finally set context cluster name {}", cloudCluster);
+ }
+
+ return cluster;
+ }
+
+ public String getDefaultCloudCluster() {
+ return null;
+ }
+
+ public String getAuthorizedCloudCluster() {
+ List<String> cloudClusterNames = Env.getCurrentSystemInfo().getCloudClusterNames();
+ // get all available cluster of the user
+ for (String cloudClusterName : cloudClusterNames) {
+ // find a cluster has more than one alive be
+ List<Backend> bes = Env.getCurrentSystemInfo().getBackendsByClusterName(cloudClusterName);
+ AtomicBoolean hasAliveBe = new AtomicBoolean(false);
Review Comment:
why atomic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1909457871
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1908587072
<details>
<summary>TPC-DS: <b>Total hot run time: 186804 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 83f5caf072ccdfc8ee5e4c5878b37804eb3cb783, data reload: false
query1 935 336 327 327
query2 6571 1964 1998 1964
query3 6722 202 199 199
query4 30723 22220 22062 22062
query5 4452 378 450 378
query6 250 166 156 156
query7 4627 255 259 255
query8 227 176 181 176
query9 8269 2578 2554 2554
query10 420 218 217 217
query11 16907 15600 15560 15560
query12 121 74 66 66
query13 1664 386 383 383
query14 10613 7027 6952 6952
query15 210 180 192 180
query16 5782 254 254 254
query17 959 473 473 473
query18 1802 257 252 252
query19 177 128 130 128
query20 71 66 65 65
query21 197 124 120 120
query22 4881 4734 4448 4448
query23 31484 31023 30948 30948
query24 5857 2887 2857 2857
query25 406 321 313 313
query26 765 146 153 146
query27 2260 285 282 282
query28 3817 1851 1816 1816
query29 917 645 632 632
query30 195 137 135 135
query31 935 739 749 739
query32 65 56 56 56
query33 415 216 207 207
query34 823 451 472 451
query35 881 758 766 758
query36 1223 1189 1173 1173
query37 96 55 60 55
query38 3377 3301 3241 3241
query39 1327 1266 1262 1262
query40 199 84 83 83
query41 36 35 36 35
query42 88 80 80 80
query43 549 459 468 459
query44 1028 681 684 681
query45 195 177 176 176
query46 1045 639 640 639
query47 1671 1599 1538 1538
query48 399 310 316 310
query49 632 296 287 287
query50 673 311 325 311
query51 5359 5224 5224 5224
query52 100 77 85 77
query53 320 262 258 258
query54 222 193 193 193
query55 79 75 77 75
query56 177 166 170 166
query57 1000 945 957 945
query58 180 156 161 156
query59 2940 2745 2726 2726
query60 205 191 187 187
query61 83 82 80 80
query62 452 378 345 345
query63 287 253 263 253
query64 2578 1770 1766 1766
query65 3335 3276 3249 3249
query66 1156 320 315 315
query67 15686 15422 15189 15189
query68 12678 526 508 508
query69 614 294 299 294
query70 1625 1499 1480 1480
query71 10445 10206 10201 10201
query72 5041 2870 2865 2865
query73 2906 317 317 317
query74 7059 6400 6441 6400
query75 5010 2357 2285 2285
query76 6447 1040 1035 1035
query77 689 276 235 235
query78 9234 8728 8631 8631
query79 1005 505 508 505
query80 608 317 318 317
query81 474 210 201 201
query82 199 78 77 77
query83 144 118 129 118
query84 272 68 67 67
query85 1072 356 337 337
query86 388 372 387 372
query87 3565 3393 3300 3300
query88 3039 2170 2151 2151
query89 431 363 344 344
query90 2239 189 182 182
query91 161 126 123 123
query92 56 46 43 43
query93 1697 428 416 416
query94 1496 159 158 158
query95 498 459 451 451
query96 616 319 311 311
query97 4313 4165 4152 4152
query98 209 194 189 189
query99 978 702 696 696
Total cold run time: 287305 ms
Total hot run time: 186804 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1909477915
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1909554190
<details>
<summary>ClickBench: <b>Total hot run time: 30.89 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dc5c9118b96d65ceb96c0f57406f700e14c472cf, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.22 0.05 0.05
query4 1.70 0.07 0.06
query5 0.54 0.52 0.52
query6 1.27 0.63 0.61
query7 0.02 0.02 0.02
query8 0.04 0.03 0.02
query9 0.53 0.51 0.49
query10 0.55 0.55 0.54
query11 0.12 0.09 0.08
query12 0.11 0.10 0.09
query13 0.61 0.61 0.61
query14 0.78 0.81 0.78
query15 0.79 0.78 0.76
query16 0.37 0.37 0.39
query17 1.03 1.04 0.99
query18 0.23 0.26 0.24
query19 1.85 1.79 1.78
query20 0.01 0.01 0.01
query21 15.42 0.57 0.56
query22 2.52 2.25 1.93
query23 17.25 1.03 0.75
query24 3.05 1.17 0.97
query25 0.38 0.17 0.14
query26 0.58 0.13 0.13
query27 0.06 0.05 0.05
query28 11.25 0.77 0.76
query29 12.57 3.16 3.14
query30 0.52 0.47 0.47
query31 2.78 0.35 0.34
query32 3.38 0.48 0.48
query33 3.21 3.23 3.21
query34 15.76 4.33 4.29
query35 4.34 4.32 4.30
query36 1.09 1.07 1.07
query37 0.06 0.04 0.04
query38 0.03 0.03 0.03
query39 0.02 0.02 0.01
query40 0.19 0.13 0.13
query41 0.07 0.02 0.01
query42 0.02 0.01 0.01
query43 0.02 0.01 0.02
Total cold run time: 105.43 s
Total hot run time: 30.89 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1910558413
<details>
<summary>TPC-DS: <b>Total hot run time: 186701 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f922cead339d7c764976df5974eba6f5762c61de, data reload: false
query1 940 336 328 328
query2 6536 1873 2035 1873
query3 6695 200 195 195
query4 30917 22362 22221 22221
query5 4448 440 391 391
query6 249 164 163 163
query7 4600 261 264 261
query8 220 212 181 181
query9 8303 2607 2588 2588
query10 436 232 236 232
query11 16848 15619 15498 15498
query12 116 66 69 66
query13 1684 375 375 375
query14 10577 6964 7100 6964
query15 215 190 186 186
query16 5787 249 268 249
query17 936 468 485 468
query18 1784 258 251 251
query19 176 132 123 123
query20 73 70 70 70
query21 196 121 129 121
query22 4968 4665 4771 4665
query23 31654 31000 30862 30862
query24 12706 2825 2831 2825
query25 565 320 306 306
query26 1831 149 148 148
query27 3204 291 287 287
query28 7366 1847 1831 1831
query29 2166 632 635 632
query30 290 137 135 135
query31 982 746 771 746
query32 81 50 54 50
query33 713 212 209 209
query34 1153 469 476 469
query35 930 793 788 788
query36 1315 1126 1210 1126
query37 138 59 58 58
query38 3313 3267 3239 3239
query39 1321 1264 1270 1264
query40 345 89 82 82
query41 38 38 34 34
query42 101 77 81 77
query43 536 479 483 479
query44 1089 682 690 682
query45 193 179 166 166
query46 1069 653 659 653
query47 1587 1578 1583 1578
query48 405 310 311 310
query49 1220 300 287 287
query50 677 315 313 313
query51 5363 5163 5232 5163
query52 89 87 77 77
query53 339 257 259 257
query54 238 192 194 192
query55 80 76 73 73
query56 184 169 170 169
query57 1040 952 943 943
query58 187 158 152 152
query59 2942 2811 2759 2759
query60 206 183 186 183
query61 83 81 81 81
query62 620 384 378 378
query63 282 264 266 264
query64 6092 1746 1721 1721
query65 3333 3248 3251 3248
query66 1439 309 321 309
query67 15707 15092 15185 15092
query68 10555 520 493 493
query69 603 310 293 293
query70 1704 1437 1471 1437
query71 10476 10205 10201 10201
query72 4697 2853 2812 2812
query73 2004 313 304 304
query74 6949 6474 6352 6352
query75 4934 2363 2357 2357
query76 6698 1047 1035 1035
query77 857 221 229 221
query78 9291 8920 8578 8578
query79 1035 504 493 493
query80 515 328 317 317
query81 444 209 203 203
query82 236 79 77 77
query83 142 117 110 110
query84 273 67 67 67
query85 1060 353 317 317
query86 402 365 392 365
query87 3479 3363 3323 3323
query88 2729 2154 2152 2152
query89 429 360 358 358
query90 2080 186 185 185
query91 151 133 126 126
query92 51 45 43 43
query93 953 415 411 411
query94 1172 158 157 157
query95 504 460 457 457
query96 604 297 305 297
query97 4272 4123 4130 4123
query98 203 203 182 182
query99 1001 689 732 689
Total cold run time: 302419 ms
Total hot run time: 186701 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1908566484
<details>
<summary>TPC-H: <b>Total hot run time: 39242 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 83f5caf072ccdfc8ee5e4c5878b37804eb3cb783, data reload: false
------ Round 1 ----------------------------------
q1 18545 5463 5364 5364
q2 2051 143 132 132
q3 10800 1187 1206 1187
q4 10441 826 794 794
q5 8048 3229 3193 3193
q6 205 122 120 120
q7 890 517 489 489
q8 10288 1987 1942 1942
q9 9288 6460 6486 6460
q10 9171 3055 3005 3005
q11 426 210 204 204
q12 356 183 187 183
q13 18025 3414 3395 3395
q14 240 209 215 209
q15 538 511 500 500
q16 481 372 392 372
q17 930 628 578 578
q18 7285 6950 6709 6709
q19 2611 1400 1366 1366
q20 573 288 287 287
q21 2777 2459 2456 2456
q22 376 297 314 297
Total cold run time: 114345 ms
Total hot run time: 39242 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5439 5320 5195 5195
q2 323 215 210 210
q3 3344 3254 3164 3164
q4 2032 2074 2047 2047
q5 6007 5866 5952 5866
q6 200 115 117 115
q7 2341 1828 1905 1828
q8 3196 3356 3377 3356
q9 8922 8851 8812 8812
q10 3986 3832 3884 3832
q11 556 471 439 439
q12 846 593 614 593
q13 16936 3163 3149 3149
q14 294 267 267 267
q15 562 510 513 510
q16 504 458 489 458
q17 2290 1746 1844 1746
q18 9497 16437 9560 9560
q19 21940 1563 1559 1559
q20 4604 1936 1932 1932
q21 14598 5361 5453 5361
q22 982 536 553 536
Total cold run time: 109399 ms
Total hot run time: 60535 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1909540426
<details>
<summary>TPC-DS: <b>Total hot run time: 186318 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dc5c9118b96d65ceb96c0f57406f700e14c472cf, data reload: false
query1 935 333 320 320
query2 6556 1818 2034 1818
query3 6694 197 198 197
query4 30145 22146 22181 22146
query5 4454 362 440 362
query6 243 164 154 154
query7 4599 264 254 254
query8 209 177 170 170
query9 8399 2597 2594 2594
query10 408 222 265 222
query11 16739 15428 15538 15428
query12 125 71 63 63
query13 1677 374 391 374
query14 10417 7054 7033 7033
query15 218 186 186 186
query16 5787 265 242 242
query17 952 467 462 462
query18 1789 256 247 247
query19 164 131 131 131
query20 71 76 72 72
query21 195 140 138 138
query22 4980 4737 4732 4732
query23 31677 30895 30901 30895
query24 12462 2762 2821 2762
query25 546 316 297 297
query26 1815 149 140 140
query27 3266 286 281 281
query28 7364 1840 1833 1833
query29 2164 607 621 607
query30 283 138 141 138
query31 945 731 760 731
query32 77 54 50 50
query33 717 210 208 208
query34 1156 454 448 448
query35 893 779 788 779
query36 1361 1197 1157 1157
query37 177 58 59 58
query38 3344 3258 3234 3234
query39 1327 1256 1273 1256
query40 346 85 78 78
query41 37 34 36 34
query42 87 79 81 79
query43 526 470 477 470
query44 1103 697 689 689
query45 192 179 179 179
query46 1094 648 638 638
query47 1656 1584 1595 1584
query48 398 306 316 306
query49 1219 291 298 291
query50 669 313 300 300
query51 5326 5229 5226 5226
query52 89 83 79 79
query53 323 258 260 258
query54 235 189 187 187
query55 87 74 75 74
query56 174 166 153 153
query57 1020 916 947 916
query58 184 159 156 156
query59 2765 2730 2671 2671
query60 208 183 180 180
query61 88 84 82 82
query62 602 360 358 358
query63 293 252 264 252
query64 6173 1786 1762 1762
query65 3318 3265 3263 3263
query66 1435 317 306 306
query67 15603 14938 15236 14938
query68 13147 521 526 521
query69 612 300 283 283
query70 1832 1548 1467 1467
query71 10429 10194 10203 10194
query72 4789 2817 2840 2817
query73 3423 317 316 316
query74 6951 6494 6386 6386
query75 4909 2348 2305 2305
query76 6252 1040 1033 1033
query77 669 222 227 222
query78 9049 8801 8545 8545
query79 1053 502 485 485
query80 533 314 311 311
query81 447 204 206 204
query82 194 81 77 77
query83 140 110 115 110
query84 273 67 76 67
query85 1081 322 307 307
query86 384 376 385 376
query87 3503 3365 3312 3312
query88 2885 2186 2200 2186
query89 435 355 351 351
query90 1948 185 183 183
query91 151 126 123 123
query92 49 44 45 44
query93 1551 426 436 426
query94 1244 167 157 157
query95 497 457 454 454
query96 627 321 322 321
query97 4274 4099 4141 4099
query98 209 182 186 182
query99 996 738 726 726
Total cold run time: 305116 ms
Total hot run time: 186318 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1911283254
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1911332608
<details>
<summary>TPC-DS: <b>Total hot run time: 187005 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 716bc4f53295f79e4695ffe58c272fb9f37f5bf1, data reload: false
query1 922 336 332 332
query2 6579 1957 1959 1957
query3 6702 203 199 199
query4 31568 22135 22188 22135
query5 4465 440 428 428
query6 241 149 147 147
query7 4611 267 258 258
query8 217 169 175 169
query9 8388 2610 2584 2584
query10 416 221 228 221
query11 16763 15613 15513 15513
query12 118 67 60 60
query13 1673 364 366 364
query14 10535 7051 7075 7051
query15 205 184 189 184
query16 5795 258 249 249
query17 950 474 461 461
query18 1785 251 255 251
query19 172 126 129 126
query20 74 72 70 70
query21 192 134 126 126
query22 4889 4836 4782 4782
query23 31490 30903 31017 30903
query24 11542 2855 2828 2828
query25 555 305 294 294
query26 1398 141 140 140
query27 3235 286 285 285
query28 7356 1848 1821 1821
query29 1248 628 605 605
query30 272 138 144 138
query31 929 744 753 744
query32 77 47 48 47
query33 691 205 201 201
query34 1163 453 456 453
query35 892 748 742 742
query36 1303 1258 1206 1206
query37 97 59 58 58
query38 3374 3273 3180 3180
query39 1303 1270 1257 1257
query40 204 93 77 77
query41 46 34 34 34
query42 96 83 86 83
query43 507 448 490 448
query44 1098 689 685 685
query45 194 175 173 173
query46 1088 642 636 636
query47 1667 1566 1485 1485
query48 389 315 328 315
query49 1146 289 290 289
query50 683 307 311 307
query51 5278 5261 5248 5248
query52 97 75 73 73
query53 335 254 258 254
query54 239 183 181 181
query55 89 75 71 71
query56 175 161 158 158
query57 1013 913 916 913
query58 170 155 182 155
query59 3027 2755 2810 2755
query60 212 173 187 173
query61 79 84 83 83
query62 622 353 344 344
query63 280 258 257 257
query64 4993 1773 1737 1737
query65 3349 3256 3247 3247
query66 1178 322 321 321
query67 15716 15181 15386 15181
query68 12220 534 527 527
query69 586 299 294 294
query70 1664 1530 1466 1466
query71 10421 10211 10194 10194
query72 5008 2868 2833 2833
query73 2719 320 315 315
query74 7044 6429 6486 6429
query75 4818 2372 2303 2303
query76 6237 1041 1030 1030
query77 662 220 225 220
query78 9160 9041 8633 8633
query79 1569 504 496 496
query80 568 320 308 308
query81 471 211 202 202
query82 201 80 75 75
query83 138 119 110 110
query84 271 66 66 66
query85 1081 334 322 322
query86 382 369 385 369
query87 3517 3330 3376 3330
query88 3176 2211 2194 2194
query89 435 372 347 347
query90 1958 186 187 186
query91 155 122 125 122
query92 49 46 43 43
query93 1876 453 433 433
query94 1245 155 151 151
query95 493 454 447 447
query96 622 313 321 313
query97 4248 4138 4125 4125
query98 204 207 189 189
query99 975 681 753 681
Total cold run time: 302328 ms
Total hot run time: 187005 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1910473457
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1908601740
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 83f5caf072ccdfc8ee5e4c5878b37804eb3cb783 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.9 seconds inserted 10000000 Rows, about 671K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1910474903
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1910534648
<details>
<summary>TPC-H: <b>Total hot run time: 38498 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f922cead339d7c764976df5974eba6f5762c61de, data reload: false
------ Round 1 ----------------------------------
q1 17677 5158 5249 5158
q2 2038 143 128 128
q3 10644 1106 1176 1106
q4 10234 822 783 783
q5 7729 3148 3141 3141
q6 197 118 119 118
q7 854 496 492 492
q8 9212 1907 1936 1907
q9 7192 6352 6308 6308
q10 8205 3071 3057 3057
q11 416 195 194 194
q12 355 187 187 187
q13 17986 3373 3349 3349
q14 241 208 220 208
q15 539 508 503 503
q16 423 366 392 366
q17 926 581 481 481
q18 7424 6856 6657 6657
q19 1744 1372 1430 1372
q20 611 313 277 277
q21 2768 2424 2407 2407
q22 348 299 303 299
Total cold run time: 107763 ms
Total hot run time: 38498 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5470 5305 5230 5230
q2 323 216 208 208
q3 3347 3243 3165 3165
q4 2106 2114 2039 2039
q5 5873 5950 5763 5763
q6 203 118 117 117
q7 2330 1892 1855 1855
q8 3196 3358 3368 3358
q9 8895 8878 8840 8840
q10 4090 3766 3808 3766
q11 560 452 469 452
q12 782 617 631 617
q13 16930 3185 3172 3172
q14 283 258 258 258
q15 551 506 506 506
q16 514 483 466 466
q17 1855 1836 1826 1826
q18 9496 9611 9406 9406
q19 23800 1516 1506 1506
q20 4564 1943 1911 1911
q21 14327 5535 5422 5422
q22 1004 532 561 532
Total cold run time: 110499 ms
Total hot run time: 60415 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1908503237
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1908595943
<details>
<summary>ClickBench: <b>Total hot run time: 30.26 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 83f5caf072ccdfc8ee5e4c5878b37804eb3cb783, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.03 0.02
query3 0.23 0.04 0.05
query4 1.70 0.07 0.06
query5 0.53 0.53 0.53
query6 1.30 0.62 0.64
query7 0.02 0.01 0.02
query8 0.04 0.02 0.02
query9 0.56 0.49 0.49
query10 0.58 0.55 0.57
query11 0.11 0.09 0.08
query12 0.11 0.09 0.09
query13 0.61 0.61 0.60
query14 0.79 0.80 0.81
query15 0.80 0.78 0.76
query16 0.38 0.38 0.37
query17 0.99 1.00 1.01
query18 0.22 0.26 0.25
query19 1.76 1.78 1.74
query20 0.01 0.01 0.02
query21 15.43 0.58 0.57
query22 2.64 2.22 1.74
query23 17.14 0.94 0.79
query24 2.55 0.70 1.10
query25 0.46 0.22 0.09
query26 0.51 0.14 0.13
query27 0.05 0.04 0.05
query28 11.87 0.78 0.73
query29 12.52 3.10 3.08
query30 0.54 0.50 0.49
query31 2.78 0.33 0.34
query32 3.39 0.48 0.49
query33 3.21 3.26 3.23
query34 16.07 4.21 4.31
query35 4.39 4.26 4.33
query36 1.12 1.08 1.06
query37 0.07 0.05 0.05
query38 0.03 0.03 0.03
query39 0.02 0.02 0.01
query40 0.15 0.12 0.13
query41 0.08 0.02 0.01
query42 0.03 0.02 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.91 s
Total hot run time: 30.26 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1911325374
<details>
<summary>TPC-H: <b>Total hot run time: 38996 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 716bc4f53295f79e4695ffe58c272fb9f37f5bf1, data reload: false
------ Round 1 ----------------------------------
q1 17673 5315 5237 5237
q2 2041 139 127 127
q3 10640 1152 1146 1146
q4 10249 770 792 770
q5 7751 3144 3155 3144
q6 201 120 120 120
q7 849 507 477 477
q8 9197 1958 1949 1949
q9 7331 6399 6391 6391
q10 8214 3051 3077 3051
q11 414 205 216 205
q12 348 190 187 187
q13 17994 3423 3354 3354
q14 237 218 212 212
q15 544 502 500 500
q16 441 381 384 381
q17 956 541 560 541
q18 7457 6982 6795 6795
q19 2149 1373 1379 1373
q20 603 299 306 299
q21 2809 2441 2464 2441
q22 369 301 296 296
Total cold run time: 108467 ms
Total hot run time: 38996 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5474 5200 5136 5136
q2 329 222 217 217
q3 3395 3289 3282 3282
q4 2036 2043 2002 2002
q5 6183 5882 5951 5882
q6 193 115 115 115
q7 2264 1891 1937 1891
q8 3228 3394 3389 3389
q9 8947 8909 8719 8719
q10 3907 3833 3873 3833
q11 544 456 442 442
q12 819 618 649 618
q13 16933 3167 3150 3150
q14 278 250 258 250
q15 554 507 500 500
q16 533 473 477 473
q17 1880 1811 1825 1811
q18 9617 12509 9554 9554
q19 20778 1594 1510 1510
q20 4621 1952 1929 1929
q21 15457 5484 5337 5337
q22 2065 563 576 563
Total cold run time: 110035 ms
Total hot run time: 60603 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1911335966
<details>
<summary>ClickBench: <b>Total hot run time: 30.61 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 716bc4f53295f79e4695ffe58c272fb9f37f5bf1, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.23 0.05 0.05
query4 1.70 0.07 0.08
query5 0.54 0.53 0.52
query6 1.18 0.62 0.67
query7 0.02 0.02 0.01
query8 0.03 0.03 0.03
query9 0.55 0.53 0.50
query10 0.59 0.57 0.57
query11 0.11 0.09 0.08
query12 0.11 0.10 0.09
query13 0.62 0.62 0.62
query14 0.79 0.83 0.80
query15 0.82 0.79 0.79
query16 0.38 0.38 0.38
query17 1.05 1.04 1.06
query18 0.26 0.25 0.23
query19 1.90 1.82 1.86
query20 0.01 0.02 0.01
query21 15.62 0.57 0.57
query22 2.96 2.13 1.43
query23 17.49 0.74 0.89
query24 2.37 1.24 0.91
query25 0.52 0.18 0.17
query26 0.50 0.13 0.14
query27 0.05 0.05 0.06
query28 11.30 0.79 0.80
query29 12.53 3.15 3.20
query30 0.56 0.49 0.48
query31 2.78 0.34 0.36
query32 3.37 0.48 0.48
query33 3.24 3.22 3.24
query34 15.77 4.30 4.24
query35 4.26 4.32 4.33
query36 1.10 1.07 1.07
query37 0.06 0.05 0.05
query38 0.03 0.02 0.02
query39 0.02 0.01 0.02
query40 0.16 0.13 0.14
query41 0.07 0.02 0.01
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 105.78 s
Total hot run time: 30.61 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #30340:
URL: https://github.com/apache/doris/pull/30340#discussion_r1465281801
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/BrokerLoadJob.java:
##########
@@ -193,6 +193,23 @@ private void onPendingTaskFinished(BrokerPendingTaskAttachment attachment) {
loadStartTimestamp = System.currentTimeMillis();
}
+ private LoadLoadingTask createAndInit(Database db, OlapTable table, List<BrokerFileGroup> brokerFileGroups,
Review Comment:
just call it createLoadingTask()
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/load/loadv2/CloudBrokerLoadJob.java:
##########
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.cloud.load.loadv2;
+
+import org.apache.doris.analysis.BrokerDesc;
+import org.apache.doris.analysis.UserIdentity;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.common.UserException;
+import org.apache.doris.load.BrokerFileGroup;
+import org.apache.doris.load.BrokerFileGroupAggInfo.FileGroupAggKey;
+import org.apache.doris.load.loadv2.BrokerLoadJob;
+import org.apache.doris.load.loadv2.BrokerPendingTaskAttachment;
+import org.apache.doris.load.loadv2.LoadLoadingTask;
+import org.apache.doris.qe.AutoCloseConnectContext;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.OriginStatement;
+import org.apache.doris.thrift.TUniqueId;
+
+import com.google.common.base.Strings;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.UUID;
+
+public class CloudBrokerLoadJob extends BrokerLoadJob {
+ private static final Logger LOG = LogManager.getLogger(CloudBrokerLoadJob.class);
+
+ protected static final String CLUSTER_ID = "clusterId";
Review Comment:
it seems to be confusing with the existing clusterid stored on BE.
is it better using cloudClusterId instead?
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/planner/CloudStreamLoadPlanner.java:
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.cloud.planner;
+
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.common.UserException;
+import org.apache.doris.planner.StreamLoadPlanner;
+import org.apache.doris.qe.AutoCloseConnectContext;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.task.LoadTaskInfo;
+import org.apache.doris.thrift.TExecPlanFragmentParams;
+import org.apache.doris.thrift.TUniqueId;
+
+import com.google.common.base.Strings;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+public class CloudStreamLoadPlanner extends StreamLoadPlanner {
+ private static final Logger LOG = LogManager.getLogger(CloudStreamLoadPlanner.class);
+
+ private String cloudClusterName;
+
+ public CloudStreamLoadPlanner(Database db, OlapTable destTable, LoadTaskInfo taskInfo, String cloudClusterName) {
+ super(db, destTable, taskInfo);
+ this.cloudClusterName = cloudClusterName;
+ }
+
+ private AutoCloseConnectContext buildConnectContext() throws UserException {
+ if (ConnectContext.get() == null) {
+ ConnectContext connectContext = new ConnectContext();
+ connectContext.setCloudCluster();
+ if (Strings.isNullOrEmpty(cloudClusterName)) {
+ connectContext.setCloudCluster();
+ } else {
+ connectContext.setCloudCluster(cloudClusterName);
+ }
Review Comment:
whats the difference between setCloudCluster() and setCloudCluster(cluster)
can we merge them into one method?
if not, changing the former one to another name like chooseCloudCluster() may avoid the confusion
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/load/loadv2/CloudBrokerLoadJob.java:
##########
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.cloud.load.loadv2;
+
+import org.apache.doris.analysis.BrokerDesc;
+import org.apache.doris.analysis.UserIdentity;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.common.UserException;
+import org.apache.doris.load.BrokerFileGroup;
+import org.apache.doris.load.BrokerFileGroupAggInfo.FileGroupAggKey;
+import org.apache.doris.load.loadv2.BrokerLoadJob;
+import org.apache.doris.load.loadv2.BrokerPendingTaskAttachment;
+import org.apache.doris.load.loadv2.LoadLoadingTask;
+import org.apache.doris.qe.AutoCloseConnectContext;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.OriginStatement;
+import org.apache.doris.thrift.TUniqueId;
+
+import com.google.common.base.Strings;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.UUID;
+
+public class CloudBrokerLoadJob extends BrokerLoadJob {
+ private static final Logger LOG = LogManager.getLogger(CloudBrokerLoadJob.class);
+
+ protected static final String CLUSTER_ID = "clusterId";
+ protected String clusterId;
+
+ public CloudBrokerLoadJob() {
+ super();
Review Comment:
this call is redundant
##########
fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java:
##########
@@ -1044,6 +1047,33 @@ public String getCloudCluster() {
return cloudCluster;
}
+ public void setCloudCluster() {
Review Comment:
pls add some commment here to desc what is going on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30340:
URL: https://github.com/apache/doris/pull/30340#issuecomment-1909561994
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit dc5c9118b96d65ceb96c0f57406f700e14c472cf with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.7 seconds inserted 10000000 Rows, about 680K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](merge-cloud) add available cloud cluster for load and query [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #30340:
URL: https://github.com/apache/doris/pull/30340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org