You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "GoGoWen (via GitHub)" <gi...@apache.org> on 2023/11/10 09:53:13 UTC
[PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
GoGoWen opened a new pull request, #26769:
URL: https://github.com/apache/doris/pull/26769
## Proposed changes
create catalog in local test/dev env like below:
create catalog iceberg PROPERTIES ('type'='iceberg','iceberg.catalog.type' = 'hadoop', 'warehouse' = '/export/workspace/warehouse');
we will get error: "Unrecognized 'warehouse' location format because name service is required."
expect:
should be OK
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1807142923
@GoGoWen Should we use prefix with 'file://' to specify local file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #26769:
URL: https://github.com/apache/doris/pull/26769
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1805713169
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.24 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162186466 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1907458342
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1907503514
<details>
<summary>ClickBench: <b>Total hot run time: 30.94 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8312dbfbd61ca646ae8e7ae32d51576e081f0680, data reload: false
query1 0.04 0.03 0.02
query2 0.05 0.02 0.02
query3 0.22 0.04 0.05
query4 1.70 0.06 0.07
query5 0.54 0.52 0.53
query6 1.32 0.63 0.65
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.54 0.51 0.51
query10 0.57 0.55 0.55
query11 0.12 0.09 0.09
query12 0.11 0.10 0.09
query13 0.60 0.61 0.61
query14 0.80 0.81 0.80
query15 0.79 0.78 0.78
query16 0.39 0.40 0.38
query17 1.00 1.04 1.02
query18 0.23 0.26 0.22
query19 1.86 1.80 1.78
query20 0.02 0.01 0.01
query21 15.41 0.59 0.58
query22 2.48 2.03 2.01
query23 17.55 0.85 0.84
query24 2.33 0.81 0.98
query25 0.43 0.23 0.16
query26 0.47 0.14 0.15
query27 0.06 0.04 0.06
query28 12.26 0.76 0.76
query29 12.51 3.12 3.25
query30 0.52 0.51 0.46
query31 2.78 0.35 0.36
query32 3.37 0.49 0.48
query33 3.24 3.20 3.23
query34 16.01 4.28 4.24
query35 4.35 4.30 4.26
query36 1.10 1.07 1.08
query37 0.06 0.04 0.04
query38 0.04 0.03 0.03
query39 0.03 0.01 0.01
query40 0.16 0.13 0.13
query41 0.07 0.01 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 106.23 s
Total hot run time: 30.94 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1907453471
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1907497439
<details>
<summary>TPC-DS: <b>Total hot run time: 187412 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8312dbfbd61ca646ae8e7ae32d51576e081f0680, data reload: false
query1 933 342 333 333
query2 6577 1972 2044 1972
query3 7071 207 219 207
query4 33773 22620 22346 22346
query5 4481 390 451 390
query6 240 158 151 151
query7 4830 275 274 274
query8 229 199 193 193
query9 9288 2604 2591 2591
query10 419 230 235 230
query11 17155 16049 15811 15811
query12 136 74 70 70
query13 1715 500 420 420
query14 11885 7052 6967 6967
query15 223 172 176 172
query16 6256 263 261 261
query17 2497 478 478 478
query18 3046 251 248 248
query19 196 134 132 132
query20 69 69 69 69
query21 189 134 116 116
query22 5026 4839 4770 4770
query23 31671 30961 30936 30936
query24 12751 2778 2851 2778
query25 558 308 308 308
query26 1828 142 148 142
query27 3314 279 289 279
query28 7477 1823 1807 1807
query29 2152 614 607 607
query30 276 136 134 134
query31 924 736 748 736
query32 73 52 48 48
query33 713 218 216 216
query34 1257 458 476 458
query35 918 776 746 746
query36 1383 1232 1276 1232
query37 178 56 61 56
query38 3375 3232 3217 3217
query39 1335 1267 1249 1249
query40 344 88 82 82
query41 36 35 35 35
query42 90 79 85 79
query43 492 478 468 468
query44 1079 689 696 689
query45 192 172 177 172
query46 1071 648 665 648
query47 1669 1512 1576 1512
query48 393 312 305 305
query49 1211 277 279 277
query50 695 315 321 315
query51 5327 5151 5165 5151
query52 86 79 78 78
query53 324 263 251 251
query54 242 185 185 185
query55 80 88 76 76
query56 179 168 164 164
query57 1017 922 913 913
query58 195 160 165 160
query59 3006 2688 2753 2688
query60 209 191 189 189
query61 85 89 81 81
query62 607 366 326 326
query63 275 252 269 252
query64 6084 1779 1757 1757
query65 3345 3278 3241 3241
query66 1425 323 312 312
query67 15499 15176 14966 14966
query68 10436 534 511 511
query69 590 305 304 304
query70 1640 1471 1481 1471
query71 10474 10195 10199 10195
query72 4349 2804 2806 2804
query73 1781 320 315 315
query74 7072 6390 6406 6390
query75 4907 2339 2264 2264
query76 6881 1037 994 994
query77 848 227 237 227
query78 9086 8932 8860 8860
query79 988 504 495 495
query80 536 325 316 316
query81 473 205 209 205
query82 235 79 79 79
query83 141 118 119 118
query84 272 70 68 68
query85 1122 339 329 329
query86 386 408 382 382
query87 3502 3354 3300 3300
query88 2691 2217 2206 2206
query89 436 373 337 337
query90 2157 189 188 188
query91 155 121 123 121
query92 52 45 43 43
query93 959 433 452 433
query94 1209 161 159 159
query95 497 451 444 444
query96 605 315 323 315
query97 4246 4126 4136 4126
query98 202 193 194 193
query99 1030 709 698 698
Total cold run time: 311632 ms
Total hot run time: 187412 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1805699417
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit ed30564db702e49f58a5951d4dc1fd432e22acdd, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5305 5083 5060 5060
q2 366 190 199 190
q3 2072 2064 2026 2026
q4 1480 1428 1418 1418
q5 4118 4160 4118 4118
q6 253 130 138 130
q7 2088 1620 1595 1595
q8 2773 2747 2758 2747
q9 10344 10268 10253 10253
q10 3484 3559 3564 3559
q11 369 245 258 245
q12 461 302 307 302
q13 4528 4136 4117 4117
q14 323 310 297 297
q15 634 560 589 560
q16 691 614 581 581
q17 1156 1096 1073 1073
q18 7773 7437 7421 7421
q19 1687 1711 1713 1711
q20 587 362 347 347
q21 4914 4571 4541 4541
q22 544 446 423 423
Total cold run time: 55950 ms
Total hot run time: 52714 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4984 5005 5026 5005
q2 347 245 238 238
q3 4104 3919 3908 3908
q4 2819 2756 2758 2756
q5 6560 6498 6480 6480
q6 250 127 130 127
q7 3163 2680 2749 2680
q8 4784 4831 4779 4779
q9 17782 17581 17669 17581
q10 4091 4140 4156 4140
q11 769 669 634 634
q12 1031 841 805 805
q13 4333 3895 3864 3864
q14 381 354 350 350
q15 618 537 571 537
q16 800 679 683 679
q17 3957 3978 3912 3912
q18 9430 9345 9382 9345
q19 1807 1770 1756 1756
q20 2408 2088 2040 2040
q21 8704 8883 8729 8729
q22 974 864 890 864
Total cold run time: 84096 ms
Total hot run time: 81209 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1907511083
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 8312dbfbd61ca646ae8e7ae32d51576e081f0680 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.7 seconds inserted 10000000 Rows, about 729K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1809439795
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1805657674
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "GoGoWen (via GitHub)" <gi...@apache.org>.
GoGoWen commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1807428733
> @GoGoWen Should we use prefix with 'file://' to specify local file?
I do not think prefix "file://" is needed， as path like "/export/workspace/warehouse" will be processed as local path in hdfs like Iceberg. in iceberg， we can use "/export/workspace/warehouse" instead of "file://export/workspace/warehouse".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26769:
URL: https://github.com/apache/doris/pull/26769#issuecomment-1907483169
<details>
<summary>TPC-H: <b>Total hot run time: 38470 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8312dbfbd61ca646ae8e7ae32d51576e081f0680, data reload: false
------ Round 1 ----------------------------------
q1 17632 5030 5291 5030
q2 2040 151 138 138
q3 10781 1127 1205 1127
q4 10410 774 802 774
q5 7736 3174 3118 3118
q6 192 120 120 120
q7 830 493 474 474
q8 9207 1903 1925 1903
q9 7212 6296 6301 6296
q10 8165 3028 3023 3023
q11 408 201 209 201
q12 353 194 189 189
q13 17988 3365 3323 3323
q14 249 216 215 215
q15 569 506 503 503
q16 420 369 362 362
q17 929 553 519 519
q18 7472 6934 6825 6825
q19 1553 1362 1348 1348
q20 589 307 275 275
q21 2772 2431 2400 2400
q22 359 307 310 307
Total cold run time: 107866 ms
Total hot run time: 38470 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5435 5362 5245 5245
q2 330 219 212 212
q3 3343 3248 3189 3189
q4 2093 2057 2052 2052
q5 5979 5880 5920 5880
q6 195 117 114 114
q7 2303 1845 1912 1845
q8 3205 3347 3348 3347
q9 8799 8793 8820 8793
q10 3871 3763 3815 3763
q11 551 447 441 441
q12 830 639 648 639
q13 16917 3174 3146 3146
q14 277 262 270 262
q15 550 510 491 491
q16 517 463 470 463
q17 1842 1907 1841 1841
q18 9372 18714 9628 9628
q19 23857 1528 1517 1517
q20 4638 1938 1922 1922
q21 14350 5458 5475 5458
q22 971 576 584 576
Total cold run time: 110225 ms
Total hot run time: 60824 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org