You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hubgeter (via GitHub)" <gi...@apache.org> on 2025/10/10 14:34:15 UTC
[PR] [fix](hdfs)Fix be coredump in HDFS reader during profile collection. [doris]
hubgeter opened a new pull request, #56806:
URL: https://github.com/apache/doris/pull/56806
### What problem does this PR solve?
Related PR: #54926
Issue Number: close #55747
Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read failed, but HDFS reader still held such a dangling pointer, which led to issue #55747
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hdfs)Fix be coredump in HDFS reader during profile collection. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56806:
URL: https://github.com/apache/doris/pull/56806#issuecomment-3390864597
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hdfs)Fix be coredump in HDFS reader during profile collection. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56806:
URL: https://github.com/apache/doris/pull/56806#issuecomment-3390864350
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hdfs)Fix be coredump in HDFS reader during profile collection. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56806:
URL: https://github.com/apache/doris/pull/56806#issuecomment-3390721141
<details>
<summary>TPC-DS: <b>Total hot run time: 190230 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c23464e1782cabe74ae14124319333b947bb49e5, data reload: false
query1 1048 433 403 403
query2 6580 1714 1681 1681
query3 6773 222 222 222
query4 25963 23974 23018 23018
query5 5201 643 465 465
query6 365 242 240 240
query7 4658 496 295 295
query8 305 251 255 251
query9 8702 2564 2544 2544
query10 519 339 291 291
query11 15520 15646 15524 15524
query12 196 129 125 125
query13 2337 561 441 441
query14 11566 9495 9507 9495
query15 250 213 175 175
query16 8283 738 542 542
query17 1838 804 675 675
query18 2334 443 365 365
query19 239 209 185 185
query20 141 145 134 134
query21 222 139 127 127
query22 4630 4646 4566 4566
query23 34908 33901 33151 33151
query24 8395 2350 2367 2350
query25 555 497 441 441
query26 1231 265 153 153
query27 2729 483 364 364
query28 4382 2192 2167 2167
query29 757 592 486 486
query30 291 224 196 196
query31 905 810 758 758
query32 80 70 71 70
query33 581 371 325 325
query34 794 861 520 520
query35 767 825 762 762
query36 996 999 936 936
query37 126 112 89 89
query38 3509 3596 3536 3536
query39 1488 1403 1409 1403
query40 222 128 116 116
query41 60 60 62 60
query42 123 116 108 108
query43 480 487 470 470
query44 1303 842 811 811
query45 186 178 172 172
query46 863 979 627 627
query47 1763 1836 1744 1744
query48 381 414 317 317
query49 772 489 419 419
query50 638 687 405 405
query51 3914 3979 4022 3979
query52 119 107 97 97
query53 234 268 195 195
query54 593 616 521 521
query55 85 84 87 84
query56 322 333 311 311
query57 1200 1219 1116 1116
query58 290 280 268 268
query59 2520 2658 2506 2506
query60 350 344 332 332
query61 159 159 154 154
query62 777 733 666 666
query63 231 197 192 192
query64 4405 1168 894 894
query65 4040 3955 3935 3935
query66 1083 457 346 346
query67 15408 15461 15251 15251
query68 7832 934 609 609
query69 509 344 303 303
query70 1354 1345 1281 1281
query71 455 340 332 332
query72 6137 4870 4851 4851
query73 617 586 364 364
query74 8926 9059 8969 8969
query75 3780 3329 2919 2919
query76 3498 1152 742 742
query77 803 415 321 321
query78 9569 9880 8911 8911
query79 2821 816 585 585
query80 679 565 487 487
query81 509 260 234 234
query82 452 166 134 134
query83 310 285 256 256
query84 309 120 91 91
query85 893 550 425 425
query86 386 325 302 302
query87 3772 3750 3677 3677
query88 3727 2205 2233 2205
query89 382 322 290 290
query90 2013 223 217 217
query91 163 172 148 148
query92 89 69 68 68
query93 2301 971 634 634
query94 712 454 342 342
query95 403 319 321 319
query96 481 588 277 277
query97 2947 2949 2867 2867
query98 249 214 214 214
query99 1444 1401 1289 1289
Total cold run time: 281207 ms
Total hot run time: 190230 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hdfs)Fix be coredump in HDFS reader during profile collection. [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56806:
URL: https://github.com/apache/doris/pull/56806#issuecomment-3390503137
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hdfs)Fix be coredump in HDFS reader during profile collection. [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #56806:
URL: https://github.com/apache/doris/pull/56806
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hdfs)Fix be coredump in HDFS reader during profile collection. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56806:
URL: https://github.com/apache/doris/pull/56806#issuecomment-3390747758
<details>
<summary>ClickBench: <b>Total hot run time: 30.41 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c23464e1782cabe74ae14124319333b947bb49e5, data reload: false
query1 0.06 0.05 0.05
query2 0.10 0.05 0.07
query3 0.25 0.08 0.08
query4 1.61 0.12 0.12
query5 0.28 0.27 0.24
query6 1.20 0.65 0.63
query7 0.04 0.03 0.03
query8 0.06 0.04 0.04
query9 0.64 0.52 0.52
query10 0.58 0.59 0.58
query11 0.17 0.11 0.12
query12 0.15 0.12 0.12
query13 0.63 0.61 0.61
query14 1.02 1.05 1.04
query15 0.87 0.87 0.87
query16 0.43 0.40 0.41
query17 1.07 1.06 1.06
query18 0.21 0.20 0.20
query19 1.97 1.89 1.84
query20 0.02 0.01 0.02
query21 15.43 0.92 0.58
query22 0.76 1.22 0.73
query23 14.83 1.39 0.69
query24 7.53 1.86 0.46
query25 0.56 0.28 0.13
query26 0.59 0.15 0.13
query27 0.07 0.05 0.05
query28 9.71 1.36 0.95
query29 12.55 3.98 3.37
query30 0.29 0.15 0.12
query31 2.83 0.60 0.39
query32 3.25 0.54 0.47
query33 3.11 3.14 3.11
query34 15.99 5.55 4.87
query35 4.92 4.90 4.93
query36 0.70 0.53 0.50
query37 0.10 0.07 0.07
query38 0.06 0.04 0.04
query39 0.03 0.02 0.03
query40 0.17 0.16 0.14
query41 0.09 0.04 0.03
query42 0.03 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 105 s
Total hot run time: 30.41 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hdfs)Fix be coredump in HDFS reader during profile collection. [doris]
Posted by "hubgeter (via GitHub)" <gi...@apache.org>.
hubgeter commented on PR #56806:
URL: https://github.com/apache/doris/pull/56806#issuecomment-3390503365
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hdfs)Fix be coredump in HDFS reader during profile collection. [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56806:
URL: https://github.com/apache/doris/pull/56806#issuecomment-3390843133
# BE UT Coverage Report
Increment line coverage `0.00% (0/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/c23464e1782cabe74ae14124319333b947bb49e5_c23464e1782cabe74ae14124319333b947bb49e5/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/c23464e1782cabe74ae14124319333b947bb49e5_c23464e1782cabe74ae14124319333b947bb49e5/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.53% (17730/33755) |
| Line Coverage | 37.69% (160951/427003) |
| Region Coverage | 32.19% (122979/382045) |
| Branch Coverage | 33.57% (53913/160576) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hdfs)Fix be coredump in HDFS reader during profile collection. [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56806:
URL: https://github.com/apache/doris/pull/56806#issuecomment-3391301004
# BE Regression && UT Coverage Report
Increment line coverage `0.00% (0/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/56806_c23464e1782cabe74ae14124319333b947bb49e5_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/56806_c23464e1782cabe74ae14124319333b947bb49e5_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.09% (23511/33073) |
| Line Coverage | 57.53% (245409/426575) |
| Region Coverage | 52.54% (203261/386873) |
| Branch Coverage | 54.43% (87835/161387) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org