You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Yukang-Lian (via GitHub)" <gi...@apache.org> on 2025/05/12 12:20:10 UTC
[PR] [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps [doris]
Yukang-Lian opened a new pull request, #50819:
URL: https://github.com/apache/doris/pull/50819
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #49882
Problem Summary:
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1746727905 (unix time) try "date -d @1746727905" if you are using GNU date ***
*** Current BE git commitID: ace825afc5 ***
*** SIGSEGV address not mapped to object (@0x8) received by PID 3151893 (TID 3152363 OR 0x7f1186c00640) from PID 8; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
3# 0x00007F12D9FEE520 in /lib/x86_64-linux-gnu/libc.so.6
4# std::_Hashtable<long, std::pair<long const, std::shared_ptr<doris::CloudBaseCompaction> >, std::allocator<std::pair<long const, std::shared_ptr<doris::CloudBaseCompaction> > >, std::__detail::_Select1st, std::equal_to<long>, std::hash<long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node(unsigned long, long const&, unsigned long) const at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/hashtable.h:1817
5# std::pair<std::__detail::_Node_iterator<std::pair<long const, std::shared_ptr<doris::CloudBaseCompaction> >, false, false>, bool> std::_Hashtable<long, std::pair<long const, std::shared_ptr<doris::CloudBaseCompaction> >, std::allocator<std::pair<long const, std::shared_ptr<doris::CloudBaseCompaction> > >, std::__detail::_Select1st, std::equal_to<long>, std::hash<long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_emplace<long, decltype(nullptr)>(std::integral_constant<bool, true>, long&&, decltype(nullptr)&&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/hashtable.h:1947
6# doris::CloudStorageEngine::_submit_base_compaction_task(std::shared_ptr<doris::CloudTablet> const&) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
7# doris::CloudStorageEngine::submit_compaction_task(std::shared_ptr<doris::CloudTablet> const&, doris::CompactionType) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/cloud/cloud_storage_engine.cpp:917
8# doris::CloudStorageEngine::_compaction_tasks_producer_callback() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/cloud/cloud_storage_engine.cpp:494
9# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499
10# start_thread at ./nptl/pthread_create.c:442
11# 0x00007F12DA0D2850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50819:
URL: https://github.com/apache/doris/pull/50819#issuecomment-2872552417
<details>
<summary>ClickBench: <b>Total hot run time: 29.07 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a9a1bdee79e7ccce7beb56d52721c2dcf7b7202c, data reload: false
query1 0.03 0.04 0.03
query2 0.12 0.10 0.11
query3 0.25 0.19 0.19
query4 1.59 0.19 0.19
query5 0.58 0.58 0.58
query6 1.20 0.72 0.72
query7 0.02 0.02 0.02
query8 0.04 0.04 0.03
query9 0.58 0.52 0.51
query10 0.56 0.57 0.57
query11 0.16 0.11 0.11
query12 0.14 0.11 0.11
query13 0.61 0.60 0.59
query14 0.79 0.81 0.81
query15 0.88 0.84 0.86
query16 0.39 0.38 0.37
query17 1.01 1.08 1.03
query18 0.23 0.22 0.22
query19 1.93 1.84 1.78
query20 0.01 0.01 0.02
query21 15.40 0.91 0.55
query22 0.73 1.22 0.65
query23 14.97 1.40 0.61
query24 7.13 1.31 0.56
query25 0.53 0.07 0.13
query26 0.66 0.16 0.15
query27 0.05 0.05 0.06
query28 9.54 0.94 0.47
query29 12.55 3.91 3.31
query30 0.25 0.09 0.07
query31 2.82 0.60 0.38
query32 3.23 0.54 0.47
query33 3.04 3.11 3.14
query34 15.78 5.10 4.53
query35 4.48 4.57 4.48
query36 0.65 0.50 0.48
query37 0.08 0.07 0.06
query38 0.06 0.04 0.04
query39 0.03 0.03 0.03
query40 0.17 0.14 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.03
query43 0.03 0.03 0.04
Total cold run time: 103.41 s
Total hot run time: 29.07 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #50819:
URL: https://github.com/apache/doris/pull/50819#issuecomment-2874839346
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50819:
URL: https://github.com/apache/doris/pull/50819#issuecomment-2872716619
# BE UT Coverage Report
Increment line coverage `0.00% (0/2)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/a9a1bdee79e7ccce7beb56d52721c2dcf7b7202c_a9a1bdee79e7ccce7beb56d52721c2dcf7b7202c/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/a9a1bdee79e7ccce7beb56d52721c2dcf7b7202c_a9a1bdee79e7ccce7beb56d52721c2dcf7b7202c/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 55.76% (14894/26709) |
| Line Coverage | 44.57% (131780/295647) |
| Region Coverage | 43.63% (66266/151882) |
| Branch Coverage | 38.23% (33956/88814) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #50819:
URL: https://github.com/apache/doris/pull/50819#issuecomment-2872339245
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50819:
URL: https://github.com/apache/doris/pull/50819#issuecomment-2874976065
# BE Regression && UT Coverage Report
Increment line coverage `0.00% (0/2)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/50819_a9a1bdee79e7ccce7beb56d52721c2dcf7b7202c_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/50819_a9a1bdee79e7ccce7beb56d52721c2dcf7b7202c_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.46% (20877/26275) |
| Line Coverage | 72.69% (214873/295583) |
| Region Coverage | 70.83% (126352/178382) |
| Branch Coverage | 64.45% (65367/101422) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50819:
URL: https://github.com/apache/doris/pull/50819#issuecomment-2872535050
<details>
<summary>TPC-DS: <b>Total hot run time: 193080 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a9a1bdee79e7ccce7beb56d52721c2dcf7b7202c, data reload: false
query1 1379 1081 1068 1068
query2 6211 1824 1842 1824
query3 11014 4635 4543 4543
query4 54814 25487 23084 23084
query5 5020 502 446 446
query6 332 229 206 206
query7 4870 502 295 295
query8 307 255 243 243
query9 5362 2692 2665 2665
query10 434 329 257 257
query11 14947 15007 14875 14875
query12 157 114 106 106
query13 1028 521 408 408
query14 10227 6323 6313 6313
query15 205 208 187 187
query16 7067 659 503 503
query17 1121 733 615 615
query18 1548 406 320 320
query19 205 192 175 175
query20 133 126 121 121
query21 211 137 112 112
query22 4318 4575 4160 4160
query23 34301 33603 33731 33603
query24 6612 2460 2450 2450
query25 473 471 411 411
query26 707 273 157 157
query27 2328 514 353 353
query28 2855 2185 2168 2168
query29 617 604 441 441
query30 275 215 188 188
query31 833 838 763 763
query32 80 62 60 60
query33 446 357 321 321
query34 774 854 551 551
query35 812 837 760 760
query36 984 996 918 918
query37 111 102 79 79
query38 4310 4220 4359 4220
query39 1517 1457 1431 1431
query40 220 120 112 112
query41 56 54 50 50
query42 126 113 105 105
query43 504 511 481 481
query44 1383 849 840 840
query45 177 176 175 175
query46 891 1060 659 659
query47 1862 1902 1803 1803
query48 408 421 333 333
query49 680 512 431 431
query50 680 704 420 420
query51 4202 4230 4237 4230
query52 112 111 102 102
query53 239 273 196 196
query54 597 579 530 530
query55 88 85 89 85
query56 296 314 299 299
query57 1186 1172 1130 1130
query58 275 265 263 263
query59 2630 2752 2596 2596
query60 340 351 315 315
query61 132 149 123 123
query62 738 778 676 676
query63 238 188 198 188
query64 1877 1062 712 712
query65 4292 4242 4227 4227
query66 738 391 310 310
query67 15875 15600 15464 15464
query68 7583 891 515 515
query69 555 308 270 270
query70 1229 1137 1103 1103
query71 515 340 297 297
query72 5998 4868 4844 4844
query73 1527 697 361 361
query74 8860 9064 8749 8749
query75 3836 3228 2731 2731
query76 4180 1194 748 748
query77 621 373 286 286
query78 10082 10321 9349 9349
query79 2764 790 574 574
query80 665 494 439 439
query81 476 256 218 218
query82 440 129 135 129
query83 377 250 227 227
query84 295 96 86 86
query85 783 352 304 304
query86 389 321 289 289
query87 4409 4431 4329 4329
query88 3409 2288 2293 2288
query89 412 310 285 285
query90 1804 202 205 202
query91 138 150 111 111
query92 76 61 65 61
query93 2032 922 580 580
query94 666 407 312 312
query95 375 287 280 280
query96 492 573 280 280
query97 3167 3192 3093 3093
query98 232 213 213 213
query99 1443 1419 1272 1272
Total cold run time: 299617 ms
Total hot run time: 193080 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50819:
URL: https://github.com/apache/doris/pull/50819#issuecomment-2872478635
<details>
<summary>TPC-H: <b>Total hot run time: 33908 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a9a1bdee79e7ccce7beb56d52721c2dcf7b7202c, data reload: false
------ Round 1 ----------------------------------
q1 25508 5079 4995 4995
q2 2056 300 198 198
q3 10362 1226 713 713
q4 10238 993 509 509
q5 7541 2347 2350 2347
q6 182 163 135 135
q7 920 740 615 615
q8 9308 1301 1081 1081
q9 6792 5112 5099 5099
q10 6870 2295 1898 1898
q11 500 294 274 274
q12 348 353 225 225
q13 17756 3825 3117 3117
q14 234 221 212 212
q15 526 464 496 464
q16 423 427 385 385
q17 636 858 386 386
q18 7498 7166 7065 7065
q19 1411 960 557 557
q20 343 324 217 217
q21 4406 3371 2489 2489
q22 1019 1017 927 927
Total cold run time: 114877 ms
Total hot run time: 33908 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5106 5063 5053 5053
q2 235 321 225 225
q3 2144 2672 2261 2261
q4 1326 1760 1367 1367
q5 4440 4378 4336 4336
q6 220 171 130 130
q7 1996 1898 1738 1738
q8 2570 2594 2458 2458
q9 7165 7185 6958 6958
q10 3070 3199 2761 2761
q11 578 519 497 497
q12 721 738 626 626
q13 3464 3801 3284 3284
q14 301 309 292 292
q15 522 480 482 480
q16 452 490 435 435
q17 1140 1498 1396 1396
q18 7694 7515 7296 7296
q19 809 806 866 806
q20 1983 2026 1842 1842
q21 5082 4735 4739 4735
q22 1037 1048 1044 1044
Total cold run time: 52055 ms
Total hot run time: 50020 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50819:
URL: https://github.com/apache/doris/pull/50819#issuecomment-2872338716
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org