You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "haoyan19881215 (via GitHub)" <gi...@apache.org> on 2023/12/22 09:46:35 UTC
[I] [Bug] doris be 自动退出 [doris]
haoyan19881215 opened a new issue, #28881:
URL: https://github.com/apache/doris/issues/28881
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
2.0.2
### What's Wrong?
there are 8 be in my doris cluster,but 2 be Automatic stop when they running for 1~2 hours。but I cannot find any exception in logs
**be.out**
`
start time: Fri Dec 22 09:47:07 CST 2023
INFO: java_cmd /opt/jdk1.8.0_191/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/hadoop_hdfs/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
`
**be.INFO**
`
I1222 11:57:49.784793 24196 olap_server.cpp:1064] cooldown producer get tablet num: 0
I1222 11:57:54.344010 24947 heartbeat_server.cpp:61] get heartbeat from FE.host:10.237.22.118, port:9020, cluster id:1914286766, counter:1561, BE start time: 1703209629783
I1222 11:57:57.641873 24283 task_worker_pool.cpp:1068] successfully report TASK|host=10.237.22.118|port=9020
I1222 11:58:09.578222 24038 load_channel_mgr.cpp:250] cleaning timed out load channels
I1222 11:58:09.578332 24038 load_channel_mgr.cpp:282] load mem consumption(bytes). limit: 13420866764, current: 0, peak: 0, total running load channels: 0
I1222 11:58:09.785163 24196 olap_server.cpp:1064] cooldown producer get tablet num: 0
I1222 11:58:10.643011 24283 task_worker_pool.cpp:1068] successfully report TASK|host=10.237.22.118|port=9020
I1222 11:58:23.643994 24283 task_worker_pool.cpp:1068] successfully report TASK|host=10.237.22.118|port=9020
I1222 11:58:29.785368 24196 olap_server.cpp:1064] cooldown producer get tablet num: 0
I1222 11:58:31.761829 24285 tablet_manager.cpp:1016] find expired transactions for 0 tablets
I1222 11:58:31.761873 24285 tablet_manager.cpp:1048] success to build all report tablets info. tablet_count=0
I1222 11:58:31.762550 24285 task_worker_pool.cpp:1068] successfully report TABLET|host=10.237.22.118|port=9020
I1222 11:58:36.864832 23487 daemon.cpp:397] doris start to exit
I1222 11:58:38.206190 24284 data_dir.cpp:810] path: /opt/apache-doris-2.0.2-bin-x64-noavx2/be/storage total capacity: 1022174953472, available capacity: 1013264003072
I1222 11:58:38.206262 24284 storage_engine.cpp:383] get root path info cost: 0 ms. tablet counter: 0
I1222 11:58:38.207151 24284 task_worker_pool.cpp:1068] successfully report DISK|host=10.237.22.118|port=9020
I1222 11:58:38.644892 24283 task_worker_pool.cpp:1068] successfully report TASK|host=10.237.22.118|port=9020
I1222 11:58:39.867292 23487 server.cpp:1167] Server[doris::PInternalServiceImpl] is going to quit
I1222 11:58:39.892066 24882 thrift_server.cpp:170] ThriftServer heartbeat exited
I1222 11:58:39.892594 24289 thrift_server.cpp:170] ThriftServer backend exited
I1222 11:58:39.892784 23487 storage_engine.cpp:546] begin stopping storage engine
I1222 11:58:39.892843 24190 olap_server.cpp:364] try to perform path gc by rowsetid!
I1222 11:58:39.893136 23487 storage_engine.cpp:566] start join garbage sweeper thread
I1222 11:58:39.893158 23487 storage_engine.cpp:568] end join garbage sweeper thread
I1222 11:58:39.893258 23487 storage_engine.cpp:588] end stopping storage engine
I1222 11:58:40.893028 24284 data_dir.cpp:810] path: /opt/apache-doris-2.0.2-bin-x64-noavx2/be/storage total capacity: 1022174953472, available capacity: 1013264003072
I1222 11:58:40.893131 24284 storage_engine.cpp:383] get root path info cost: 0 ms. tablet counter: 0
I1222 11:58:40.893143 24285 tablet_manager.cpp:1016] find expired transactions for 0 tablets
I1222 11:58:40.893206 24285 tablet_manager.cpp:1048] success to build all report tablets info. tablet_count=0
I1222 11:58:40.893572 24283 task_worker_pool.cpp:1068] successfully report TASK|host=10.237.22.118|port=9020
I1222 11:58:40.893594 24284 task_worker_pool.cpp:1068] successfully report DISK|host=10.237.22.118|port=9020
I1222 11:58:40.893640 24285 task_worker_pool.cpp:1068] successfully report TABLET|host=10.237.22.118|port=9020
I1222 11:58:40.898120 23487 task_scheduler.cpp:63] Start shutdown BlockedTaskScheduler
I1222 11:58:40.898229 23800 task_scheduler.cpp:193] BlockedTaskScheduler schedule thread stop
I1222 11:58:40.898555 23487 task_scheduler.cpp:63] Start shutdown BlockedTaskScheduler
I1222 11:58:40.898597 23809 task_scheduler.cpp:193] BlockedTaskScheduler schedule thread stop
I1222 11:58:40.899576 23810 fragment_mgr.cpp:1080] FragmentMgr cancel worker is going to exit.
I1222 11:58:40.901443 23888 result_buffer_mgr.cpp:172] result buffer manager cancel thread finish.
I1222 11:58:40.904731 23487 routine_load_task_executor.cpp:83] 0 not executed tasks left, cleanup
I1222 11:58:40.921046 23487 olap_meta.cpp:68] [Rocksdb] [db/db_impl.cc:252] Shutdown: canceling all background work
I1222 11:58:40.921134 23487 olap_meta.cpp:68] [Rocksdb] [db/db_impl.cc:252] Shutdown: canceling all background work
I1222 11:58:40.921468 23487 olap_meta.cpp:68] [Rocksdb] [db/db_impl.cc:398] Shutdown complete
I1222 11:58:40.921481 23487 olap_meta.cpp:106] finish close rocksdb for OlapMeta
I1222 11:58:40.929057 23487 stream_load_recorder.cpp:56] finish close rocksdb for ~StreamLoadRecorder
`
**be.WARNING**
`
W1222 11:47:09.999689 23889 status.h:383] meet error status: [IO_ERROR]failed to list /opt/apache-doris-2.0.2-bin-x64-noavx2/be/storage/mini_download: (2), No such file or directory
0. /root/src/doris-2.0/be/src/common/stack_trace.cpp:302: StackTrace::tryCapture() @ 0x000000000b9e64c7 in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
1. /root/src/doris-2.0/be/src/common/stack_trace.h:0: doris::get_stack_trace[abi:cxx11]() @ 0x000000000b9e4ae5 in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
2. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173: doris::Status doris::Status::Error<true, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_
_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(int, std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_stri
ng<char, std::char_traits<char>, std::allocator<char> >&&) @ 0x000000000aecc168 in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
3. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::io::LocalFileSystem::list_impl(std::filesystem::__cxx11::path const&, bool, std::vector<doris::io::FileInfo, std::allocator<do
ris::io::FileInfo> >*, bool*) @ 0x000000000aec6eac in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
4. /root/src/doris-2.0/be/src/common/status.h:348: doris::io::FileSystem::list(std::filesystem::__cxx11::path const&, bool, std::vector<doris::io::FileInfo, std::allocator<doris::io::FileInfo> >*, bool*) @ 0x000000000aec0f6c in /opt/apache-doris-2.
0.2-bin-x64-noavx2/be/lib/doris_be
5. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:360: doris::LoadPathMgr::clean_one_path(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) @ 0x00000000
0b83cd40 in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
6. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_iterator.h:1034: std::_Function_handler<void (), doris::LoadPathMgr::init()::$_0>::_M_invoke(std::_Any_data const&) @ 0x000000000b83e218 in /opt/apac
he-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
7. /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562: doris::Thread::supervise_thread(void*) @ 0x000000000ba1819a in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
8. start_thread @ 0x0000000000007dd5 in /usr/lib64/libpthread-2.17.so
9. clone @ 0x00000000000fdead in /usr/lib64/libc-2.17.so
W1222 11:47:09.999835 23889 file_system.cpp:72] [IO_ERROR]failed to list /opt/apache-doris-2.0.2-bin-x64-noavx2/be/storage/mini_download: (2), No such file or directory
0. /root/src/doris-2.0/be/src/common/stack_trace.cpp:302: StackTrace::tryCapture() @ 0x000000000b9e64c7 in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
1. /root/src/doris-2.0/be/src/common/stack_trace.h:0: doris::get_stack_trace[abi:cxx11]() @ 0x000000000b9e4ae5 in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
2. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173: doris::Status doris::Status::Error<true, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_
_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(int, std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_stri
ng<char, std::char_traits<char>, std::allocator<char> >&&) @ 0x000000000aecc168 in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
3. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::io::LocalFileSystem::list_impl(std::filesystem::__cxx11::path const&, bool, std::vector<doris::io::FileInfo, std::allocator<do
ris::io::FileInfo> >*, bool*) @ 0x000000000aec6eac in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
4. /root/src/doris-2.0/be/src/common/status.h:348: doris::io::FileSystem::list(std::filesystem::__cxx11::path const&, bool, std::vector<doris::io::FileInfo, std::allocator<doris::io::FileInfo> >*, bool*) @ 0x000000000aec0f6c in /opt/apache-doris-2.
0.2-bin-x64-noavx2/be/lib/doris_be
5. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:360: doris::LoadPathMgr::clean_one_path(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) @ 0x00000000
0b83cd40 in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
6. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_iterator.h:1034: std::_Function_handler<void (), doris::LoadPathMgr::init()::$_0>::_M_invoke(std::_Any_data const&) @ 0x000000000b83e218 in /opt/apac
he-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
7. /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562: doris::Thread::supervise_thread(void*) @ 0x000000000ba1819a in /opt/apache-doris-2.0.2-bin-x64-noavx2/be/lib/doris_be
8. start_thread @ 0x0000000000007dd5 in /usr/lib64/libpthread-2.17.so
9. clone @ 0x00000000000fdead in /usr/lib64/libc-2.17.so
`
I run command 'dmesg -T',there was no any information about doris.
### What You Expected?
I want to know the reason and how to fix it
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] doris be 自动退出 [doris]
Posted by "haoyan19881215 (via GitHub)" <gi...@apache.org>.
haoyan19881215 commented on issue #28881:
URL: https://github.com/apache/doris/issues/28881#issuecomment-1869965243
> > > Are the above logs for the two BE nodes that stopped working
> >
> >
> > yes, these are all recent log before the nodes stoped
>
> Can you check the mini_ Does the download directory exist or is there a permission issue?
Thanks for replying.There is not the directory,and I checked other running nodes,this sentence is in all the log of these nodes .So I think it is not the fatal error ,but I still can not find other information.
> > > Are the above logs for the two BE nodes that stopped working
> >
> >
> > yes, these are all recent log before the nodes stoped
>
> Can you check the mini_ Does the download directory exist or is there a permission issue?
I checked it,there is no this directory ,but on other running be,there is not this directory either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] doris be 自动退出 [doris]
Posted by "haoyan19881215 (via GitHub)" <gi...@apache.org>.
haoyan19881215 commented on issue #28881:
URL: https://github.com/apache/doris/issues/28881#issuecomment-1868627604
> Are the above logs for the two BE nodes that stopped working
yes, these are all recent log before the nodes stoped
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] doris be 自动退出 [doris]
Posted by "cjj2010 (via GitHub)" <gi...@apache.org>.
cjj2010 commented on issue #28881:
URL: https://github.com/apache/doris/issues/28881#issuecomment-1868664379
> > Are the above logs for the two BE nodes that stopped working
>
> yes, these are all recent log before the nodes stoped
Can you check the mini_ Does the download directory exist or is there a permission issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] doris be 自动退出 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28881:
URL: https://github.com/apache/doris/issues/28881#issuecomment-3287228408
We're closing this issue because it hasn't been updated in a while.
If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] doris be 自动退出 [doris]
Posted by "cjj2010 (via GitHub)" <gi...@apache.org>.
cjj2010 commented on issue #28881:
URL: https://github.com/apache/doris/issues/28881#issuecomment-1868447536
Are the above logs for the two BE nodes that stopped working
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] doris be 自动退出 [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman closed issue #28881: [Bug] doris be 自动退出
URL: https://github.com/apache/doris/issues/28881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org