You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "w41ter (via GitHub)" <gi...@apache.org> on 2024/07/18 07:30:05 UTC
[PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
w41ter opened a new pull request, #38064:
URL: https://github.com/apache/doris/pull/38064
Cherry-pick #33858
Many domestic cloud vendors are compatible with the s3 protocol. However, early versions of s3 client will only generate path style http requests (https://github.com/aws/aws-sdk-java-v2/pull/763) when encountering endpoints that do not start with s3, while some cloud vendors only support virtual host style http request.
Therefore, Doris used `forceVirtualHosted` in `S3URI` to convert it into a virtual hosted path and implemented it through path style. For example:
For s3 uri `s3://my-bucket/data/file.txt`, It will eventually be parsed into:
- virtualBucket: my-bucket
- Bucket: data (bucket must be set, otherwise the s3 client will report an error) Especially this step is particularly tricky because of the limitations of the s3 client.
- Key: file.txt
The path style mode is used to generate an http request similar to the virtual host by setting the endpoint to virtualBucket + original endpoint, setting the bucket and key.
**However, the bucket and key here are inconsistent with the original concepts of s3, but the aws client happens to be able to generate an http request similar to the virtual host through the path style mode.**
However, after #30799 we have upgrade the aws sdk version from 2.17.257 to 2.20.131. The current aws s3 client can already generate a virtual host by third party by default style of http request. So in #31111 need to set the path style option, let the s3 client use doris' virtual bucket mechanism to continue working.
**Finally, the virtual bucket mechanism is too confusing and tricky, and we no longer need it with the new version of s3 client.**
Rewrite `S3URI` to remove tricky virtual bucket mechanism and support different uri styles by flags.
This class represents a fully qualified location in S3 for input/output operations expressed as as URI.
#### For AWS S3, URI common styles:
- AWS Client Style(Hadoop S3 Style): `s3://my-bucket/path/to/file?versionId=abc123&partNumber=77&partNumber=88`
- Virtual Host Style: `https://my-bucket.s3.us-west-1.amazonaws.com/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88`
- Path Style: `https://s3.us-west-1.amazonaws.com/my-bucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88`
Regarding the above-mentioned common styles, we can use <code>isPathStyle</code> to control whether to use path style
or virtual host style.
"Virtual host style" is the currently mainstream and recommended approach to use, so the default value of
<code>isPathStyle</code> is false.
#### Other Styles:
- Virtual Host AWS Client (Hadoop S3) Mixed Style: `s3://my-bucket.s3.us-west-1.amazonaws.com/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88`
- Path AWS Client (Hadoop S3) Mixed Style: `s3://s3.us-west-1.amazonaws.com/my-bucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88`
For these two styles, we can use <code>isPathStyle</code> and <code>forceParsingByStandardUri</code>
to control whether to use.
Virtual Host AWS Client (Hadoop S3) Mixed Style: <code>isPathStyle = false && forceParsingByStandardUri = true</code>
Path AWS Client (Hadoop S3) Mixed Style: <code>isPathStyle = true && forceParsingByStandardUri = true</code>
When the incoming location is url encoded, the encoded string will be returned.
For <code>getKey()</code>, <code>getQueryParams()</code> will return the encoding string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2236128130
<details>
<summary>TPC-DS: <b>Total hot run time: 203242 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bfcdd018a7e6a07c095c3e3d2e3880b7fc9bf1e3, data reload: false
query1 934 427 376 376
query2 6551 2825 2708 2708
query3 6921 212 204 204
query4 19835 17936 17877 17877
query5 19744 6518 6581 6518
query6 306 217 236 217
query7 4227 300 311 300
query8 510 437 435 435
query9 3099 2674 2574 2574
query10 424 293 303 293
query11 11272 10690 10687 10687
query12 122 75 72 72
query13 5595 693 705 693
query14 17973 13463 13910 13463
query15 367 245 251 245
query16 6453 288 263 263
query17 1685 1467 873 873
query18 2304 420 426 420
query19 208 155 159 155
query20 84 77 77 77
query21 200 101 94 94
query22 5217 4999 4997 4997
query23 32586 31966 32040 31966
query24 6913 6537 6531 6531
query25 510 422 441 422
query26 533 170 166 166
query27 1862 302 294 294
query28 6160 2373 2309 2309
query29 2915 2791 2876 2791
query30 245 169 168 168
query31 909 743 763 743
query32 70 66 59 59
query33 409 265 263 263
query34 848 479 504 479
query35 1125 948 970 948
query36 1427 1199 1131 1131
query37 90 59 64 59
query38 3119 2917 2971 2917
query39 1378 1328 1336 1328
query40 203 95 98 95
query41 48 44 45 44
query42 90 90 82 82
query43 819 660 622 622
query44 1127 725 729 725
query45 244 236 234 234
query46 1260 959 980 959
query47 1754 1745 1695 1695
query48 1005 723 720 720
query49 642 380 381 380
query50 848 574 603 574
query51 4763 4663 4643 4643
query52 95 87 78 78
query53 451 324 316 316
query54 2634 2473 2469 2469
query55 97 82 87 82
query56 247 206 207 206
query57 1235 1135 1030 1030
query58 222 205 209 205
query59 4184 3749 3686 3686
query60 217 188 224 188
query61 95 96 97 96
query62 834 520 471 471
query63 492 356 346 346
query64 2579 1554 1436 1436
query65 3658 3549 3539 3539
query66 792 384 370 370
query67 15817 15454 15289 15289
query68 10475 654 666 654
query69 577 339 369 339
query70 1951 1395 1485 1395
query71 421 304 318 304
query72 6547 3411 3483 3411
query73 739 318 317 317
query74 6290 5805 5868 5805
query75 5370 3663 3706 3663
query76 6787 1127 1187 1127
query77 1183 257 253 253
query78 12581 11522 12127 11522
query79 11101 638 629 629
query80 891 398 395 395
query81 499 235 234 234
query82 1487 97 98 97
query83 159 138 134 134
query84 263 72 73 72
query85 889 327 329 327
query86 335 306 298 298
query87 3275 3039 3036 3036
query88 4497 2324 2344 2324
query89 477 307 285 285
query90 2016 199 211 199
query91 174 140 146 140
query92 56 50 55 50
query93 6434 596 580 580
query94 721 205 214 205
query95 1094 1057 1061 1057
query96 638 332 330 330
query97 6442 6346 6364 6346
query98 188 181 178 178
query99 2905 841 889 841
Total cold run time: 319044 ms
Total hot run time: 203242 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2235819232
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2236102598
<details>
<summary>TPC-H: <b>Total hot run time: 49668 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bfcdd018a7e6a07c095c3e3d2e3880b7fc9bf1e3, data reload: false
------ Round 1 ----------------------------------
q1 17696 4368 4325 4325
q2 2073 154 145 145
q3 10437 1872 1864 1864
q4 10336 1259 1307 1259
q5 8633 3896 3908 3896
q6 251 124 125 124
q7 2045 1593 1605 1593
q8 9313 2719 2687 2687
q9 10582 10296 10342 10296
q10 8626 3485 3500 3485
q11 421 240 256 240
q12 469 305 298 298
q13 18386 3955 4053 3955
q14 357 331 326 326
q15 500 466 448 448
q16 679 577 581 577
q17 1109 958 981 958
q18 7188 6905 6760 6760
q19 1778 1671 1643 1643
q20 553 309 296 296
q21 4436 4152 4045 4045
q22 519 455 448 448
Total cold run time: 116387 ms
Total hot run time: 49668 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4321 4344 4329 4329
q2 322 225 219 219
q3 4155 4127 4166 4127
q4 2749 2754 2752 2752
q5 7110 7062 7117 7062
q6 234 121 120 120
q7 3247 2918 2790 2790
q8 4325 4474 4509 4474
q9 16960 16624 16729 16624
q10 4229 4248 4277 4248
q11 750 668 678 668
q12 1035 857 860 857
q13 6910 3773 3756 3756
q14 455 424 437 424
q15 490 466 457 457
q16 731 708 682 682
q17 3901 3859 3856 3856
q18 8884 8762 8838 8762
q19 1698 1737 1643 1643
q20 2366 2132 2111 2111
q21 8521 8557 8516 8516
q22 1070 978 971 971
Total cold run time: 84463 ms
Total hot run time: 79448 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2235925110
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 71574850ade4be8f99c15f2d3820f10ff5dc233a with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.3 seconds inserted 10000000 Rows, about 469K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter closed pull request #38064: [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags.
URL: https://github.com/apache/doris/pull/38064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2235906910
<details>
<summary>TPC-DS: <b>Total hot run time: 203802 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 71574850ade4be8f99c15f2d3820f10ff5dc233a, data reload: false
query1 932 423 385 385
query2 6544 2771 2601 2601
query3 6927 207 205 205
query4 20244 18041 18019 18019
query5 19738 6530 6549 6530
query6 387 219 231 219
query7 4895 294 315 294
query8 434 408 411 408
query9 3082 2605 2561 2561
query10 427 294 302 294
query11 11533 10933 11110 10933
query12 121 88 77 77
query13 5610 690 693 690
query14 18847 13531 13321 13321
query15 368 248 238 238
query16 6466 281 262 262
query17 1358 1514 856 856
query18 2309 411 401 401
query19 207 150 143 143
query20 78 80 78 78
query21 184 106 98 98
query22 5256 5067 5089 5067
query23 32313 31863 32112 31863
query24 6819 6546 6530 6530
query25 521 431 427 427
query26 512 164 162 162
query27 1753 295 298 295
query28 6131 2357 2294 2294
query29 2915 2847 2839 2839
query30 245 168 172 168
query31 935 719 727 719
query32 73 60 59 59
query33 411 244 249 244
query34 858 483 497 483
query35 1098 908 909 908
query36 1240 1139 1072 1072
query37 95 59 58 58
query38 3107 2948 2917 2917
query39 1360 1339 1330 1330
query40 207 96 91 91
query41 47 44 45 44
query42 82 85 82 82
query43 772 638 773 638
query44 1155 714 716 714
query45 247 231 239 231
query46 1217 940 965 940
query47 2123 1700 1815 1700
query48 1009 712 699 699
query49 632 377 371 371
query50 868 596 607 596
query51 4753 4657 4681 4657
query52 98 83 88 83
query53 446 322 329 322
query54 2636 2487 2440 2440
query55 90 75 89 75
query56 242 227 197 197
query57 1125 1171 1061 1061
query58 220 198 192 192
query59 4291 4039 4247 4039
query60 240 195 193 193
query61 99 95 119 95
query62 849 500 488 488
query63 482 341 343 341
query64 2542 1435 1525 1435
query65 3608 3497 3587 3497
query66 766 377 382 377
query67 15791 15404 15271 15271
query68 10199 640 666 640
query69 593 343 348 343
query70 2020 1406 1490 1406
query71 423 315 314 314
query72 6558 3489 3529 3489
query73 740 329 312 312
query74 6476 5760 5812 5760
query75 5311 3589 3665 3589
query76 6553 1142 1192 1142
query77 1156 265 254 254
query78 12711 11947 12336 11947
query79 8177 643 660 643
query80 1150 398 409 398
query81 487 231 235 231
query82 1300 98 95 95
query83 159 129 140 129
query84 255 71 70 70
query85 887 327 330 327
query86 330 300 298 298
query87 3200 2992 3046 2992
query88 4310 2295 2323 2295
query89 418 301 299 299
query90 1953 211 212 211
query91 182 141 139 139
query92 56 51 49 49
query93 5814 550 570 550
query94 669 209 206 206
query95 1092 1058 1048 1048
query96 649 323 323 323
query97 6475 6412 6359 6359
query98 188 190 180 180
query99 3101 847 911 847
Total cold run time: 316562 ms
Total hot run time: 203802 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2236138768
<details>
<summary>ClickBench: <b>Total hot run time: 30.3 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bfcdd018a7e6a07c095c3e3d2e3880b7fc9bf1e3, data reload: false
query1 0.02 0.02 0.02
query2 0.06 0.03 0.02
query3 0.24 0.04 0.05
query4 1.82 0.07 0.06
query5 0.52 0.53 0.52
query6 1.25 0.62 0.61
query7 0.01 0.02 0.01
query8 0.04 0.02 0.02
query9 0.53 0.47 0.49
query10 0.54 0.54 0.54
query11 0.12 0.09 0.09
query12 0.12 0.09 0.09
query13 0.62 0.62 0.62
query14 0.78 0.77 0.78
query15 0.77 0.76 0.77
query16 0.36 0.36 0.38
query17 1.03 1.01 0.99
query18 0.21 0.26 0.24
query19 1.92 1.85 1.83
query20 0.01 0.00 0.00
query21 15.46 0.54 0.54
query22 1.89 1.95 1.69
query23 17.32 1.02 0.86
query24 7.98 0.68 0.57
query25 0.43 0.13 0.04
query26 0.68 0.17 0.18
query27 0.05 0.04 0.04
query28 5.91 0.74 0.75
query29 12.71 2.38 2.24
query30 0.62 0.53 0.53
query31 2.81 0.38 0.37
query32 3.37 0.50 0.49
query33 3.10 3.08 3.07
query34 15.26 4.78 4.79
query35 4.83 4.84 4.85
query36 1.11 1.01 1.02
query37 0.06 0.05 0.04
query38 0.03 0.03 0.03
query39 0.02 0.01 0.02
query40 0.16 0.14 0.15
query41 0.07 0.02 0.01
query42 0.02 0.01 0.01
query43 0.02 0.02 0.02
Total cold run time: 104.88 s
Total hot run time: 30.3 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2236038316
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2235918644
<details>
<summary>ClickBench: <b>Total hot run time: 31.2 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 71574850ade4be8f99c15f2d3820f10ff5dc233a, data reload: false
query1 0.02 0.02 0.02
query2 0.08 0.02 0.02
query3 0.24 0.05 0.05
query4 1.78 0.06 0.07
query5 0.54 0.52 0.52
query6 1.23 0.60 0.62
query7 0.01 0.02 0.02
query8 0.03 0.03 0.02
query9 0.52 0.50 0.48
query10 0.55 0.54 0.51
query11 0.12 0.08 0.09
query12 0.10 0.09 0.09
query13 0.62 0.61 0.61
query14 0.78 0.78 0.78
query15 0.78 0.77 0.77
query16 0.37 0.35 0.38
query17 1.02 1.02 1.00
query18 0.21 0.23 0.26
query19 1.86 1.86 1.87
query20 0.02 0.01 0.01
query21 15.46 0.54 0.56
query22 1.92 2.12 2.18
query23 17.42 0.95 0.89
query24 6.12 1.03 1.54
query25 0.39 0.10 0.04
query26 0.70 0.15 0.15
query27 0.05 0.04 0.03
query28 6.49 0.74 0.71
query29 12.75 2.28 2.26
query30 0.63 0.54 0.54
query31 2.80 0.40 0.36
query32 3.38 0.50 0.51
query33 3.09 3.06 3.10
query34 15.23 4.81 4.78
query35 4.84 4.86 4.85
query36 1.06 1.02 1.02
query37 0.06 0.04 0.05
query38 0.03 0.02 0.02
query39 0.02 0.01 0.02
query40 0.16 0.14 0.14
query41 0.07 0.01 0.01
query42 0.02 0.01 0.01
query43 0.03 0.02 0.01
Total cold run time: 103.6 s
Total hot run time: 31.2 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2235821657
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2236144681
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit bfcdd018a7e6a07c095c3e3d2e3880b7fc9bf1e3 with default session variables
Stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.4 seconds inserted 10000000 Rows, about 467K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](multi-catalog) Rewrite `S3URI` to remove tricky virtualbucket mechanism and support different uri styles by flags. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38064:
URL: https://github.com/apache/doris/pull/38064#issuecomment-2235881782
<details>
<summary>TPC-H: <b>Total hot run time: 49609 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 71574850ade4be8f99c15f2d3820f10ff5dc233a, data reload: false
------ Round 1 ----------------------------------
q1 17582 4367 4323 4323
q2 2064 153 148 148
q3 10270 1905 2160 1905
q4 10294 1253 1304 1253
q5 8516 3866 3890 3866
q6 236 148 123 123
q7 2027 1568 1617 1568
q8 9299 2725 2703 2703
q9 10743 10407 10189 10189
q10 8647 3542 3517 3517
q11 422 244 261 244
q12 468 298 301 298
q13 18346 3940 4034 3940
q14 349 323 325 323
q15 499 465 456 456
q16 674 570 571 570
q17 1121 953 962 953
q18 7235 6891 6795 6795
q19 1805 1659 1591 1591
q20 558 293 284 284
q21 4409 4106 4109 4106
q22 533 458 454 454
Total cold run time: 116097 ms
Total hot run time: 49609 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4301 4326 4381 4326
q2 323 226 228 226
q3 4172 4185 4156 4156
q4 2755 2751 2782 2751
q5 7155 7167 7172 7167
q6 239 119 122 119
q7 3177 2906 2884 2884
q8 4417 4508 4479 4479
q9 16934 16877 16913 16877
q10 4421 4423 4402 4402
q11 813 736 715 715
q12 1085 921 940 921
q13 7884 3836 3817 3817
q14 463 432 439 432
q15 513 471 468 468
q16 739 707 672 672
q17 3905 3904 3855 3855
q18 8840 8799 8800 8799
q19 1730 1688 1657 1657
q20 2384 2110 2133 2110
q21 8567 8458 8445 8445
q22 1029 1031 977 977
Total cold run time: 85846 ms
Total hot run time: 80255 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org