You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wuwenchi (via GitHub)" <gi...@apache.org> on 2024/06/06 04:10:30 UTC
[PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
wuwenchi opened a new pull request, #35957:
URL: https://github.com/apache/doris/pull/35957
## Proposed changes
<!--Describe your changes.-->
Doris will convert the schema in FIleSystem to achieve the function of natively reading files. For example, `s3a` will be converted to `s3`.
The position delete file of iceberg will record the full path of the datafile, which includes the schema.
When comparing datafile with position delete, the converted path cannot be used, but the original datafile path must be used.
So, added variables for the original path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35957:
URL: https://github.com/apache/doris/pull/35957#issuecomment-2154356463
<details>
<summary>TPC-H: <b>Total hot run time: 40355 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 198e8634918f79075b6219eea79ceab7ec01ccfb, data reload: false
------ Round 1 ----------------------------------
q1 17691 4540 4374 4374
q2 3010 203 211 203
q3 11327 1161 1110 1110
q4 10515 827 790 790
q5 7767 2745 2809 2745
q6 231 137 143 137
q7 1000 620 617 617
q8 9717 2129 2114 2114
q9 9088 6505 6516 6505
q10 8923 3726 3722 3722
q11 445 242 239 239
q12 502 229 226 226
q13 17786 2978 2967 2967
q14 269 226 221 221
q15 537 473 487 473
q16 538 389 382 382
q17 990 755 709 709
q18 8202 7487 7363 7363
q19 2502 1507 1511 1507
q20 662 315 316 315
q21 4970 3302 3960 3302
q22 412 334 341 334
Total cold run time: 117084 ms
Total hot run time: 40355 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4410 4225 4168 4168
q2 373 271 255 255
q3 3036 2771 2765 2765
q4 1954 1616 1620 1616
q5 5325 5325 5496 5325
q6 217 124 126 124
q7 2111 1692 1755 1692
q8 3221 3349 3321 3321
q9 8422 8369 8508 8369
q10 3898 3654 3686 3654
q11 594 477 487 477
q12 766 611 599 599
q13 16755 2993 2985 2985
q14 293 284 283 283
q15 516 477 486 477
q16 492 434 411 411
q17 1781 1470 1477 1470
q18 7704 7682 7382 7382
q19 1705 1612 1611 1611
q20 1964 1804 1768 1768
q21 4965 4603 4636 4603
q22 626 531 534 531
Total cold run time: 71128 ms
Total hot run time: 53886 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35957:
URL: https://github.com/apache/doris/pull/35957#issuecomment-2154403160
TeamCity be ut coverage result:
Function Coverage: 36.44% (8976/24630)
Line Coverage: 27.98% (73361/262193)
Region Coverage: 27.38% (38023/138862)
Branch Coverage: 24.01% (19314/80456)
Coverage Report: http://coverage.selectdb-in.cc/coverage/198e8634918f79075b6219eea79ceab7ec01ccfb_198e8634918f79075b6219eea79ceab7ec01ccfb/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35957:
URL: https://github.com/apache/doris/pull/35957#issuecomment-2154270166
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35957:
URL: https://github.com/apache/doris/pull/35957#issuecomment-2151379025
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #35957:
URL: https://github.com/apache/doris/pull/35957
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35957:
URL: https://github.com/apache/doris/pull/35957#issuecomment-2154817827
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35957:
URL: https://github.com/apache/doris/pull/35957#issuecomment-2154817763
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #35957:
URL: https://github.com/apache/doris/pull/35957#issuecomment-2154261461
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35957:
URL: https://github.com/apache/doris/pull/35957#issuecomment-2154375120
<details>
<summary>TPC-DS: <b>Total hot run time: 172081 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 198e8634918f79075b6219eea79ceab7ec01ccfb, data reload: false
query1 936 382 364 364
query2 6456 2479 2289 2289
query3 6654 202 205 202
query4 20815 17462 17352 17352
query5 4123 470 438 438
query6 242 159 182 159
query7 4589 292 290 290
query8 334 299 299 299
query9 8492 2398 2363 2363
query10 450 308 271 271
query11 10566 10282 10249 10249
query12 138 93 86 86
query13 1673 356 354 354
query14 9176 7558 6811 6811
query15 254 182 182 182
query16 8061 262 250 250
query17 1868 515 523 515
query18 1949 270 288 270
query19 196 147 146 146
query20 88 88 82 82
query21 210 143 130 130
query22 4522 4047 4072 4047
query23 33476 32967 32961 32961
query24 11819 2773 2873 2773
query25 611 352 352 352
query26 1670 153 153 153
query27 2892 330 324 324
query28 7562 2060 2024 2024
query29 975 603 585 585
query30 286 147 150 147
query31 936 727 738 727
query32 93 50 54 50
query33 766 279 268 268
query34 1001 480 467 467
query35 761 659 598 598
query36 1079 946 934 934
query37 156 66 69 66
query38 2895 2716 2744 2716
query39 870 793 792 792
query40 278 124 117 117
query41 53 52 54 52
query42 117 97 97 97
query43 611 576 546 546
query44 1168 721 735 721
query45 200 163 166 163
query46 1086 707 706 706
query47 1875 1744 1814 1744
query48 377 294 292 292
query49 1146 411 419 411
query50 790 387 374 374
query51 6779 6686 6641 6641
query52 106 93 91 91
query53 351 281 281 281
query54 1055 446 436 436
query55 74 74 71 71
query56 299 249 250 249
query57 1145 1064 1062 1062
query58 268 263 235 235
query59 3488 3155 3120 3120
query60 291 275 258 258
query61 88 83 90 83
query62 652 455 435 435
query63 325 292 290 290
query64 9827 2250 1698 1698
query65 3215 3085 3089 3085
query66 1322 338 330 330
query67 15599 14989 15057 14989
query68 4609 539 541 539
query69 453 286 298 286
query70 1089 1014 1142 1014
query71 394 261 273 261
query72 7105 5506 5479 5479
query73 746 317 315 315
query74 6060 5564 5530 5530
query75 3375 2650 2661 2650
query76 2752 922 882 882
query77 422 291 290 290
query78 10237 9798 9652 9652
query79 2399 501 503 501
query80 934 454 444 444
query81 590 216 220 216
query82 764 103 98 98
query83 233 167 168 167
query84 248 83 81 81
query85 2061 264 257 257
query86 526 339 321 321
query87 3278 3079 3086 3079
query88 4118 2349 2334 2334
query89 480 376 396 376
query90 1825 188 188 188
query91 137 172 97 97
query92 59 46 50 46
query93 2370 510 509 509
query94 1289 183 177 177
query95 404 300 314 300
query96 584 269 259 259
query97 3179 3016 3012 3012
query98 222 208 202 202
query99 1186 860 856 856
Total cold run time: 276353 ms
Total hot run time: 172081 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35957:
URL: https://github.com/apache/doris/pull/35957#issuecomment-2154384348
<details>
<summary>ClickBench: <b>Total hot run time: 29.91 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 198e8634918f79075b6219eea79ceab7ec01ccfb, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.67 0.07 0.07
query5 0.50 0.47 0.50
query6 1.13 0.71 0.72
query7 0.02 0.02 0.02
query8 0.04 0.04 0.04
query9 0.54 0.49 0.48
query10 0.55 0.56 0.54
query11 0.14 0.11 0.11
query12 0.15 0.12 0.12
query13 0.59 0.60 0.59
query14 0.78 0.76 0.79
query15 0.84 0.82 0.82
query16 0.36 0.36 0.36
query17 1.05 1.04 0.98
query18 0.21 0.26 0.25
query19 1.82 1.78 1.75
query20 0.01 0.01 0.01
query21 15.61 0.66 0.64
query22 4.43 8.50 1.34
query23 18.29 1.44 1.25
query24 2.12 0.23 0.21
query25 0.15 0.09 0.08
query26 0.26 0.18 0.18
query27 0.08 0.07 0.07
query28 13.24 1.02 1.01
query29 13.19 3.26 3.25
query30 0.25 0.05 0.06
query31 2.85 0.39 0.38
query32 3.28 0.48 0.47
query33 2.86 2.97 2.85
query34 17.01 4.46 4.39
query35 4.50 4.49 4.50
query36 0.64 0.48 0.47
query37 0.17 0.16 0.16
query38 0.16 0.15 0.14
query39 0.04 0.04 0.03
query40 0.16 0.14 0.15
query41 0.09 0.05 0.04
query42 0.06 0.04 0.04
query43 0.05 0.03 0.04
Total cold run time: 110.24 s
Total hot run time: 29.91 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](iceberg)Fix the datafile path error issue [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35957:
URL: https://github.com/apache/doris/pull/35957#issuecomment-2151374772
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org