You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Yukang-Lian (via GitHub)" <gi...@apache.org> on 2023/12/28 15:42:00 UTC
[PR] [Fix](core) Fix file system scan deleted file [doris]
Yukang-Lian opened a new pull request, #29266:
URL: https://github.com/apache/doris/pull/29266
## Proposed changes
Issue Number: close #xxx
Fix core:
terminate called after throwing an instance of 'std::filesystem::__cxx11::filesystem_error'
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121220&linesState=121220&logView=linear) what(): filesystem error: cannot get file size: No such file or directory [/mnt/ssd01/cluster_storage/doris.SSD/P0/cluster0/wal/192735/194510/9158_group_commit_414e0ba38971bc59_16f57becc4aeca90]
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121221&linesState=121221&logView=linear) *** Query id: 0-0 ***
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121222&linesState=121222&logView=linear) *** tablet id: 0 ***
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121223&linesState=121223&logView=linear) *** Aborted at 1703763396 (unix time) try "date -d @1703763396" if you are using GNU date ***
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121224&linesState=121224&logView=linear) *** Current BE git commitID: b4cc7c5 ***
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121225&linesState=121225&logView=linear) *** SIGABRT unknown detail explain (@0x15274) received by PID 86644 (TID 87963 OR 0x7f9cb0044700) from PID 86644; stack trace: ***
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121226&linesState=121226&logView=linear) 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:417
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121227&linesState=121227&logView=linear) 1# 0x00007FA32E3BD090 in /lib/x86_64-linux-gnu/libc.so.6
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121228&linesState=121228&logView=linear) 2# raise at ../sysdeps/unix/sysv/linux/raise.c:51
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121229&linesState=121229&logView=linear) 3# abort at /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:81
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121230&linesState=121230&logView=linear) 4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121231&linesState=121231&logView=linear) 5# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121232&linesState=121232&logView=linear) 6# 0x0000562DE4732E61 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121233&linesState=121233&logView=linear) 7# 0x0000562DE4732FB4 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121234&linesState=121234&logView=linear) 8# std::filesystem::file_size(std::filesystem::__cxx11::path const&) [clone .cold] at ../../../../../libstdc++-v3/src/c++17/fs_ops.cc:919
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121235&linesState=121235&logView=linear) 9# doris::io::LocalFileSystem::directory_size(std::filesystem::__cxx11::path const&, unsigned long*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121236&linesState=121236&logView=linear) 10# doris::WalDirInfo::update_wal_dir_limit(unsigned long) at /root/doris/be/src/olap/wal_dirs_info.cpp:93
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121237&linesState=121237&logView=linear) 11# doris::WalDirsInfo::update_all_wal_dir_limit() at /root/doris/be/src/olap/wal_dirs_info.cpp:181
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121238&linesState=121238&logView=linear) 12# doris::WalManager::_update_wal_dir_info_thread() in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121239&linesState=121239&logView=linear) 13# std::_Function_handler<void (), doris::WalManager::_init_wal_dirs_info()::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121240&linesState=121240&logView=linear) 14# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:499
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121241&linesState=121241&logView=linear) 15# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
[19:53:53 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_P0Regression/319209?buildTab=log&focusLine=121242&linesState=121242&logView=linear) 16# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](core) Fix file system scan deleted file [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #29266:
URL: https://github.com/apache/doris/pull/29266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](core) Fix file system scan deleted file [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29266:
URL: https://github.com/apache/doris/pull/29266#issuecomment-1871314318
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.64 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17188991728 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](core) Fix file system scan deleted file [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29266:
URL: https://github.com/apache/doris/pull/29266#issuecomment-1871293035
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](core) Fix file system scan deleted file [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #29266:
URL: https://github.com/apache/doris/pull/29266#issuecomment-1871286085
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](core) Fix file system scan deleted file [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29266:
URL: https://github.com/apache/doris/pull/29266#issuecomment-1871293068
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](core) Fix file system scan deleted file [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #29266:
URL: https://github.com/apache/doris/pull/29266#discussion_r1437768355
##########
be/src/io/fs/local_file_system.cpp:
##########
@@ -182,7 +183,11 @@ Status LocalFileSystem::directory_size(const Path& dir_path, size_t* dir_size) {
if (std::filesystem::exists(dir_path) && std::filesystem::is_directory(dir_path)) {
for (const auto& entry : std::filesystem::recursive_directory_iterator(dir_path)) {
if (std::filesystem::is_regular_file(entry)) {
- *dir_size += std::filesystem::file_size(entry);
+ try {
+ *dir_size += std::filesystem::file_size(entry);
+ } catch (const std::exception& e) {
+ LOG(INFO) << "{}", e.what();
Review Comment:
Print more context in the log e.g.
```suggestion
LOG(INFO) << "failed to get file size, err: {}", e.what();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](core) Fix file system scan deleted file [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29266:
URL: https://github.com/apache/doris/pull/29266#issuecomment-1871304756
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 4235aafaa9abd41a675b4a8364b63111d0501738, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5364 5130 5062 5062
q2 400 151 152 151
q3 1479 1154 1245 1154
q4 1102 855 938 855
q5 3170 3107 3177 3107
q6 233 163 157 157
q7 996 579 538 538
q8 2161 2285 2232 2232
q9 6877 6841 6848 6841
q10 3241 3128 3116 3116
q11 344 215 213 213
q12 392 244 242 242
q13 4416 3643 3636 3636
q14 250 218 233 218
q15 626 569 558 558
q16 460 428 387 387
q17 1045 572 516 516
q18 7074 6743 6753 6743
q19 1643 1528 1617 1528
q20 604 352 359 352
q21 2876 2507 2481 2481
q22 395 313 323 313
Total cold run time: 45148 ms
Total hot run time: 40400 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5109 5052 5038 5038
q2 338 241 264 241
q3 3366 3275 3267 3267
q4 2171 2067 2041 2041
q5 5933 5918 5918 5918
q6 240 133 130 130
q7 2373 1931 1909 1909
q8 3533 3642 3632 3632
q9 9020 8951 8962 8951
q10 3871 3914 3946 3914
q11 567 465 493 465
q12 801 637 660 637
q13 3850 3235 3204 3204
q14 298 278 266 266
q15 608 574 566 566
q16 569 511 519 511
q17 2027 1822 1789 1789
q18 8796 8330 8299 8299
q19 1761 1689 1708 1689
q20 2273 2009 1987 1987
q21 5694 5394 5300 5300
q22 551 498 525 498
Total cold run time: 63749 ms
Total hot run time: 60252 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](core) Fix file system scan deleted file [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29266:
URL: https://github.com/apache/doris/pull/29266#issuecomment-1871302598
TeamCity be ut coverage result:
Function Coverage: 36.58% (8588/23478)
Line Coverage: 28.65% (69854/243799)
Region Coverage: 27.65% (36147/130744)
Branch Coverage: 24.36% (18472/75826)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4235aafaa9abd41a675b4a8364b63111d0501738_4235aafaa9abd41a675b4a8364b63111d0501738/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](core) Fix file system scan deleted file [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29266:
URL: https://github.com/apache/doris/pull/29266#issuecomment-1871291524
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org