You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Jaap Wiggelinkhuizen (Jira)" <ji...@apache.org> on 2021/07/19 07:29:00 UTC

[jira] [Commented] (PROTON-2411) Simultaneous idle timeout sequencing errors

    [ https://issues.apache.org/jira/browse/PROTON-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383088#comment-17383088 ] 

Jaap Wiggelinkhuizen commented on PROTON-2411:
----------------------------------------------

Interestingly the coredumps of the incidents show that the abort occurred at the EPOLL_FATAL in replace_timer_deadline() although stderr shows the error-message of the EPOLL_FATAL in pni_timer_set() ???
{code:java}
Program terminated with signal SIGABRT, Aborted.
#0  0xf77d0430 in __kernel_vsyscall ()
[Current thread is 1 (Thread 0xf17ffb40 (LWP 57506))]
Missing separate debuginfos, use: debuginfo-install expat-2.1.0-10.el7_3.i686 fontconfig-2.10.95-11.el7.i686 freetype-2.4.11-15.el7.i686 glibc-2.17-222.el7.i686 keyutils-libs-1.5.8-3.el7.i686 krb5-libs-1.15.1-18.el7.i686 libICE-1.0.9-9.el7.i686 libSM-1.2.2-2.el7.i686 libX11-1.6.5-1.el7.i686 libXau-1.0.8-2.1.el7.i686 libXext-1.3.3-3.el7.i686 libXft-2.3.2-2.el7.i686 libXmu-1.1.2-2.el7.i686 libXp-1.0.2-2.1.el7.i686 libXrender-0.9.10-1.el7.i686 libXt-1.1.5-3.el7.i686 libcom_err-1.42.9-11.el7.i686 libgcc-4.8.5-28.el7.i686 libjpeg-turbo-1.2.90-5.el7.i686 libpng-1.5.13-7.el7_2.i686 libselinux-2.5-12.el7.i686 libstdc++-4.8.5-28.el7.i686 libunwind-1.2-2.el7.i686 libuuid-2.23.2-52.el7.i686 libxcb-1.12-1.el7.i686 motif-2.3.4-12.el7_4.i686 ncurses-libs-5.9-14.20130511.el7_4.i686 openssl-libs-1.0.2k-12.el7.i686 pcre-8.32-17.el7.i686 protobuf-2.5.0-8.el7.i686 xerces-c-3.1.1-8.el7_2.i686
(gdb) where
#0  0xf77d0430 in __kernel_vsyscall ()
#1  0xf6af8147 in raise () from /lib/libc.so.6
#2  0xf6af9a52 in abort () from /lib/libc.so.6
#3  0xf6964bac in replace_timer_deadline (timer=<optimized out>, tm=<optimized out>)
    at /opt/jenkins_home/workspace/PRL/Qpid_release_build/qpid-proton-src/c/src/proactor/epoll_timer.c:375
#4  pni_timer_set (timer=<optimized out>, deadline=<optimized out>) at /opt/jenkins_home/workspace/PRL/Qpid_release_build/qpid-proton-src/c/src/proactor/epoll_timer.c:267
#5  0xf6964f7d in pconnection_tick (pc=pc@entry=0xf3a06a00) at /opt/jenkins_home/workspace/PRL/Qpid_release_build/qpid-proton-src/c/src/proactor/epoll.c:1433
#6  0xf696692e in pconnection_process (pc=pc@entry=0xf3a06a00, events=<optimized out>, events@entry=1, sched_ready=sched_ready@entry=false, topup=false)
    at /opt/jenkins_home/workspace/PRL/Qpid_release_build/qpid-proton-src/c/src/proactor/epoll.c:1211
#7  0xf6968237 in process (tsk=0xf3a06a00) at /opt/jenkins_home/workspace/PRL/Qpid_release_build/qpid-proton-src/c/src/proactor/epoll.c:2213
#8  next_event_batch (p=0xa4ec218, can_block=true) at /opt/jenkins_home/workspace/PRL/Qpid_release_build/qpid-proton-src/c/src/proactor/epoll.c:2423
#9  0xf6e14b9d in proton::container::impl::thread() (this=0xa4ec168)
    at /opt/jenkins_home/workspace/PRL/Qpid_release_build/qpid-proton-src/cpp/src/proactor_container_impl.cpp:760
#10 0xf6e1536d in proton::container::impl::run(int) (this=0xa4ec168, threads=1)
    at /opt/jenkins_home/workspace/PRL/Qpid_release_build/qpid-proton-src/cpp/src/proactor_container_impl.cpp:812
#11 0xf6e020be in proton::container::run() (this=0xa4ec034) at /opt/jenkins_home/workspace/PRL/Qpid_release_build/qpid-proton-src/cpp/src/container.cpp:92
#12 0x0868eeb2 in pbamq::Communicator::ProtonContainerThread (this=0xa4ec008) at /usr/src/prl/src_0/vptlib/postbus/amq/communicator.cxx:657
#13 0xf6e28f5d in execute_native_thread_routine () from /opt/qpid-itr/lib/libqpid-proton-cpp.so.12
#14 0xf77aeb1c in start_thread () from /lib/libpthread.so.0
#15 0xf6bc745e in clone () from /lib/libc.so.6
{code}

> Simultaneous idle timeout sequencing errors
> -------------------------------------------
>
>                 Key: PROTON-2411
>                 URL: https://issues.apache.org/jira/browse/PROTON-2411
>             Project: Qpid Proton
>          Issue Type: Bug
>            Reporter: Jaap Wiggelinkhuizen
>            Priority: Major
>
> In our mission critical software we use Qpid proton 0.34.0 in our C++-client software together with the Qpid dispatch router 1.16.0. We updated to these versions not so long ago, before we used proton 0.25.0 and dispatch 1.3.0. Our application runs on several VM’s with a router on each VM. All clients connect to the local router only and the routers connect to eachother in a hub spoke pattern. In both the client configuration as the router configuration we have configured an idle timeout of 30 seconds.
> On July 4th we were confronted with an incident in production where a lot of our client processes reported problems regarding the idle timeouts. These client processes were already running stable for more than 3 weeks. The problem appeared in two flavors:
>  # Transport error “error: amqp:resource-limit-exceeded: local-idle-timeout expired”
>  # epoll proactor failure in epoll_timer.c:263: “idle timeout sequencing error”
> On each VM at least 3 processes showed one of these problems in a total time window of less than a minute. We haven’t found any cause in the underlying hardware, hypervisor, network or operating system until now.
> Although we don’t know the root cause of the problems, we can solve the first situation by using the proper reconnect settings (by mistake we handled on_transport_error() as a fatal situation and will correct that so that only on_transport_close() will be handled as fatal). However the second situation is more odd because it results in an abort within proton itself. The comments in epoll_timer.c explain that this error occurs when a connection timer is moved backwards a second time. We don’t understand how this can happen suddenly.
>  
> Last sunday the problem occurred again on two more production sites where our software was operational just over 3 weeks now. And again it has happened on all VM's within a short timeframe. It's interesting that it only occurs on sunday mornings until now. Maybe it has something to do with how long the software is running and the fact that on sunday mornings there is less messaging traffic, i.e. more heartbeats?...
>  
> Unfortunately we haven't been able to reproduce the issue at our test facilities and hence can not provide a reproducer.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org