You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by wolfwolfswinkel <wo...@objectplus.nl> on 2011/12/22 14:59:26 UTC
Heartbeats in C++ broker on Windows
Hi,
I am using the 0.10 c++ broker and client libraries on windows and
performing some experiments with the reconnect behaviour of the async sender
in the c++ messaging client API. Specifically, I try to enable heartbeats on
the connection, however until now without success.
I set Connection options "reconnect" to "true" and "heartbeat" to 6, open
the Connection, create a Session and a Sender, then send a few messages
which are delivered as expected. Then I leave the connection idle for a
while, the message "debug Traffic timeout" appears in the client's log after
12 seconds (probably twice the heartbeat interval). This seems to be the
indication that the broker did not sent a heartbeat in time.
Then I start sending messages using the Sender again. It appears the
messages do not reach the broker. Inspection on the broker side shows the
connection with the client still exists, but is idle. At some point the
Sender reaches capacity and then waits indefinitely for capacity inside a
blocking sync() call.
It seems there are two issues here:
1. the broker does not send a heartbeat
2. when the client detects the missing heartbeat, it gets in a state where
it cannot deliver messages to the broker. No exceptions are thrown, no
indications in the log that the connection is lost or of reconnection
attempts
Regarding the first issue:
If I turn on tracing on the broker side, I get:
2011-12-22 11:49:52 trace SENT [192.168.169.27:5672-192.168.169.115:4518]:
Frame[BEbe; channel=0; {ConnectionTuneBody: channel-max=32767;
max-frame-size=65535; heartbeat-min=0; heartbeat-max=0; }]
2011-12-22 11:49:52 trace RECV [192.168.169.27:5672-192.168.169.115:4518]:
Frame[BEbe; channel=0; {ConnectionTuneOkBody: channel-max=32767;
max-frame-size=65535; heartbeat=0; }]
I am not familiar with the protocol at this level, but I suspect this is the
broker telling the client its min/max supported heartbeat intervals (both
0), and then the client agreeing to use heartbeat interval 0.
Does anyone recognize these issues? Does the C++ broker on windows support
heartbeats at all?
Thanks,
Wolf Wolfswinkel
--
View this message in context: http://qpid.2158936.n2.nabble.com/Heartbeats-in-C-broker-on-Windows-tp7118702p7118702.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Heartbeats in C++ broker on Windows
Posted by Gordon Sim <gs...@redhat.com>.
On 01/05/2012 11:20 PM, Chuck Rolke wrote:
>>
>> That will require a little more debugging however.
>>
>
> I've looked at this a little bit.
Thanks, Chuck!
> I have a windows client running the
> main.cpp from this thread. The test I run is to let the client send
> the first two messages. Then during the pause I pull the network
> cable. After the pause the client sends messages 3..51. Finally, plug
> the network cable back in and see what happens.
The issue raised by Wolf (as I understand it) is that with AMQP
heartbeats enabled on the client, the client should detect the loss of
connectivity while the cable is unplugged, should close the socket and
since reconnect is enabled it should then try to reconnect.
However on windows it appeared as if this was not happening. The sending
thread was blocked waiting for completion rather than having detected
the loss of connection and going into the reconnection logic.
> With a LINUX broker the client restarts shortly after restoring the
> connection. With a (patched) WINDOWS broker the client appears to
> hang. However, if you wait five minutes the client restarts.
>
> A Wireshark sniffer analysis shows:
>
> With the LINUX broker: after the connection is restored the broker
> sends a TCP frame. The client ACKs the frame, sends a RST, and opens
> a new connection.
>
> With the WINDOWS broker: after the connection is restored the broker
> sends a RST frame to the client and the client ACKs the RST. Then the
> client sits for 300 seconds. During this time netstat shows that the
> TCP connection is in ESTABLISHED state. Next the client sends a RST
> frame to the broker. Immediately thereafter the client opens a new
> connection and messages start flowing again.
>
> Somehow the tcpconnector code is not processing the received RST in a
> timely manner. I'll look at that next.
This sounds like(?) the differences at the TCP level in detecting the
loss of connectivity, which is a different question from the AMQP
heartbeating (though interesting nonetheless).
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Heartbeats in C++ broker on Windows
Posted by Chuck Rolke <cr...@redhat.com>.
>
> That will require a little more debugging however.
>
I've looked at this a little bit. I have a windows client running the main.cpp from this thread. The test I run is to let the client send the first two messages. Then during the pause I pull the network cable. After the pause the client sends messages 3..51. Finally, plug the network cable back in and see what happens.
With a LINUX broker the client restarts shortly after restoring the connection.
With a (patched) WINDOWS broker the client appears to hang. However, if you wait five minutes the client restarts.
A Wireshark sniffer analysis shows:
With the LINUX broker: after the connection is restored the broker sends a TCP frame. The client ACKs the frame, sends a RST, and opens a new connection.
With the WINDOWS broker: after the connection is restored the broker sends a RST frame to the client and the client ACKs the RST. Then the client sits for 300 seconds. During this time netstat shows that the TCP connection is in ESTABLISHED state. Next the client sends a RST frame to the broker. Immediately thereafter the client opens a new connection and messages start flowing again.
Somehow the tcpconnector code is not processing the received RST in a timely manner. I'll look at that next.
Regards,
Chuck
----- Original Message -----
> From: "Gordon Sim" <gs...@redhat.com>
> To: users@qpid.apache.org
> Sent: Thursday, January 5, 2012 8:00:27 AM
> Subject: Re: Heartbeats in C++ broker on Windows
>
> On 01/04/2012 12:03 PM, wolfwolfswinkel wrote:
> > The patch fixes my first issue (the broker not sending heartbeats).
> >
> > Unfortunately the second issue persists. Now I trigger a heartbeat
> > timeout
> > in the client by disconnecting a network cable. The client notices
> > the
> > missing heartbeat (shows the "debug Traffic timeout" message), but
> > then does
> > not do any reconnect attempts. When the Sender is up to capacity,
> > it seems
> > to block indefinitely in a sync() call. Stack trace below.
>
> That looks like a problem on the client side. When the heartbeat
> fires
> it asks the AsynchIO part of the library to callback on an IO thread
> (AsynchIO::requestCallback()). The callback function is
> TCPConnector::eof() which just calls TCPConnector::close() which then
> calls AsynchIO::queueWriteClose(). That should result in the socket
> being closed and the application being notified of the closed socket.
>
> As far as I can see, the only part that differs between windows and
> linux here is the AsyncIO implementation. My guess is that either the
> callback mechanism on AsynchIO isn't working as expected on windows,
> or
> somehow the close that it triggers isn't actually closing the socket
> and
> notifying the application as expected.
>
> That will require a little more debugging however.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Heartbeats in C++ broker on Windows
Posted by Gordon Sim <gs...@redhat.com>.
On 01/04/2012 12:03 PM, wolfwolfswinkel wrote:
> The patch fixes my first issue (the broker not sending heartbeats).
>
> Unfortunately the second issue persists. Now I trigger a heartbeat timeout
> in the client by disconnecting a network cable. The client notices the
> missing heartbeat (shows the "debug Traffic timeout" message), but then does
> not do any reconnect attempts. When the Sender is up to capacity, it seems
> to block indefinitely in a sync() call. Stack trace below.
That looks like a problem on the client side. When the heartbeat fires
it asks the AsynchIO part of the library to callback on an IO thread
(AsynchIO::requestCallback()). The callback function is
TCPConnector::eof() which just calls TCPConnector::close() which then
calls AsynchIO::queueWriteClose(). That should result in the socket
being closed and the application being notified of the closed socket.
As far as I can see, the only part that differs between windows and
linux here is the AsyncIO implementation. My guess is that either the
callback mechanism on AsynchIO isn't working as expected on windows, or
somehow the close that it triggers isn't actually closing the socket and
notifying the application as expected.
That will require a little more debugging however.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Heartbeats in C++ broker on Windows
Posted by wolfwolfswinkel <wo...@objectplus.nl>.
Hi Gordon,
The patch fixes my first issue (the broker not sending heartbeats).
Unfortunately the second issue persists. Now I trigger a heartbeat timeout
in the client by disconnecting a network cable. The client notices the
missing heartbeat (shows the "debug Traffic timeout" message), but then does
not do any reconnect attempts. When the Sender is up to capacity, it seems
to block indefinitely in a sync() call. Stack trace below.
--Wolf
kernel32.dll!7c809590()
boost_thread-vc90-mt-gd-1_46_1.dll!boost::this_thread::interruptible_wait(void
* handle_to_wait_for=0x00001730, boost::detail::timeout target_time={...})
Line 454 + 0x34 bytes C++
qpidclientd.dll!boost::detail::basic_cv_list_entry::wait(boost::detail::timeout
wait_until={...}) Line 82 + 0x20 bytes C++
qpidclientd.dll!boost::detail::basic_condition_variable::do_wait<boost::recursive_mutex>(boost::recursive_mutex
& lock={...}, boost::detail::timeout wait_until={...}) Line 218 + 0x1e
bytes C++
qpidclientd.dll!boost::condition_variable_any::wait<boost::recursive_mutex>(boost::recursive_mutex
& m={...}) Line 369 C++
qpidclientd.dll!qpid::sys::Condition::wait(qpid::sys::Mutex & mutex={...})
Line 62 C++
qpidclientd.dll!qpid::sys::Monitor::wait() Line 42 C++
qpidclientd.dll!qpid::sys::Waitable::wait() Line 92 C++
qpidclientd.dll!qpid::client::SessionImpl::waitForCompletionImpl(const
qpid::framing::SequenceNumber & id={...}) Line 181 C++
qpidclientd.dll!qpid::client::SessionImpl::waitForCompletion(const
qpid::framing::SequenceNumber & id={...}) Line 174 C++
qpidclientd.dll!qpid::client::Future::wait(qpid::client::SessionImpl &
session={...}) Line 33 C++
qpidclientd.dll!qpid::client::SessionBase_0_10::sync() Line 50 + 0x35 bytes
C++
qpidmessagingd.dll!qpid::client::amqp0_10::SenderImpl::waitForCapacity()
Line 106 + 0xe bytes C++
qpidmessagingd.dll!qpid::client::amqp0_10::SenderImpl::Send::operator()()
Line 111 C++
qpidmessagingd.dll!qpid::client::amqp0_10::SessionImpl::execute<qpid::client::amqp0_10::SenderImpl::Send>(qpid::client::amqp0_10::SenderImpl::Send
& f={...}) Line 100 C++
qpidmessagingd.dll!qpid::client::amqp0_10::SenderImpl::send(const
qpid::messaging::Message & message={...}, bool sync=false) Line 44 + 0x1e
bytes C++
qpidmessagingd.dll!qpid::messaging::Sender::send(const
qpid::messaging::Message & message={...}, bool sync=false) Line 35 + 0x2a
bytes C++
--
View this message in context: http://qpid.2158936.n2.nabble.com/Heartbeats-in-C-broker-on-Windows-tp7118702p7150140.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Heartbeats in C++ broker on Windows
Posted by Gordon Sim <gs...@redhat.com>.
On 12/22/2011 01:59 PM, wolfwolfswinkel wrote:
> Hi,
>
> I am using the 0.10 c++ broker and client libraries on windows and
> performing some experiments with the reconnect behaviour of the async sender
> in the c++ messaging client API. Specifically, I try to enable heartbeats on
> the connection, however until now without success.
>
> I set Connection options "reconnect" to "true" and "heartbeat" to 6, open
> the Connection, create a Session and a Sender, then send a few messages
> which are delivered as expected. Then I leave the connection idle for a
> while, the message "debug Traffic timeout" appears in the client's log after
> 12 seconds (probably twice the heartbeat interval). This seems to be the
> indication that the broker did not sent a heartbeat in time.
>
> Then I start sending messages using the Sender again. It appears the
> messages do not reach the broker. Inspection on the broker side shows the
> connection with the client still exists, but is idle. At some point the
> Sender reaches capacity and then waits indefinitely for capacity inside a
> blocking sync() call.
>
> It seems there are two issues here:
>
> 1. the broker does not send a heartbeat
>
> 2. when the client detects the missing heartbeat, it gets in a state where
> it cannot deliver messages to the broker. No exceptions are thrown, no
> indications in the log that the connection is lost or of reconnection
> attempts
>
> Regarding the first issue:
>
> If I turn on tracing on the broker side, I get:
>
> 2011-12-22 11:49:52 trace SENT [192.168.169.27:5672-192.168.169.115:4518]:
> Frame[BEbe; channel=0; {ConnectionTuneBody: channel-max=32767;
> max-frame-size=65535; heartbeat-min=0; heartbeat-max=0; }]
> 2011-12-22 11:49:52 trace RECV [192.168.169.27:5672-192.168.169.115:4518]:
> Frame[BEbe; channel=0; {ConnectionTuneOkBody: channel-max=32767;
> max-frame-size=65535; heartbeat=0; }]
>
> I am not familiar with the protocol at this level, but I suspect this is the
> broker telling the client its min/max supported heartbeat intervals (both
> 0), and then the client agreeing to use heartbeat interval 0.
>
> Does anyone recognize these issues? Does the C++ broker on windows support
> heartbeats at all?
I think this is an issue with the c++ broker on windows where it is
incorrectly communicating the acceptable limits (thus causing the client
to believe heartbeats are not supported).
Unfortunately the sending of the tune control, in which the heartbeat
limits are conveyed, is triggered from the authentication logic which
differs between the platforms even though the heartbeat implementation
does not.
The attached patch should fix this.
RE: Heartbeats in C++ broker on Windows
Posted by wolfwolfswinkel <wo...@objectplus.nl>.
Hi Steve,
Attached is the test code. Currently I am running this on 0.10 broker
and client libs.
Wolf
________________________________
From: Steve Huston [via Qpid]
[mailto:ml-node+s2158936n7119191h29@n2.nabble.com]
Sent: Thursday, December 22, 2011 5:43 PM
To: Wolf Wolfswinkel
Subject: RE: Heartbeats in C++ broker on Windows
Hi Wolf,
Could you please post your test code? I'll try this on 0.14 (shortly to
be
released) and see if the issue persists.
-Steve
--
View this message in context: http://qpid.2158936.n2.nabble.com/Heartbeats-in-C-broker-on-Windows-tp7118702p7129839.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
RE: Heartbeats in C++ broker on Windows
Posted by Steve Huston <sh...@riverace.com>.
Hi Wolf,
Could you please post your test code? I'll try this on 0.14 (shortly to be
released) and see if the issue persists.
-Steve
> -----Original Message-----
> From: wolfwolfswinkel [mailto:wolf.wolfswinkel@objectplus.nl]
> Sent: Thursday, December 22, 2011 8:59 AM
> To: users@qpid.apache.org
> Subject: Heartbeats in C++ broker on Windows
>
> Hi,
>
> I am using the 0.10 c++ broker and client libraries on windows and
performing
> some experiments with the reconnect behaviour of the async sender in the
> c++ messaging client API. Specifically, I try to enable heartbeats on
the
> connection, however until now without success.
>
> I set Connection options "reconnect" to "true" and "heartbeat" to 6,
open
> the Connection, create a Session and a Sender, then send a few messages
> which are delivered as expected. Then I leave the connection idle for a
while,
> the message "debug Traffic timeout" appears in the client's log after
> 12 seconds (probably twice the heartbeat interval). This seems to be the
> indication that the broker did not sent a heartbeat in time.
>
> Then I start sending messages using the Sender again. It appears the
> messages do not reach the broker. Inspection on the broker side shows
the
> connection with the client still exists, but is idle. At some point the
Sender
> reaches capacity and then waits indefinitely for capacity inside a
blocking
> sync() call.
>
> It seems there are two issues here:
>
> 1. the broker does not send a heartbeat
>
> 2. when the client detects the missing heartbeat, it gets in a state
where it
> cannot deliver messages to the broker. No exceptions are thrown, no
> indications in the log that the connection is lost or of reconnection
attempts
>
> Regarding the first issue:
>
> If I turn on tracing on the broker side, I get:
>
> 2011-12-22 11:49:52 trace SENT
[192.168.169.27:5672-192.168.169.115:4518]:
> Frame[BEbe; channel=0; {ConnectionTuneBody: channel-max=32767; max-
> frame-size=65535; heartbeat-min=0; heartbeat-max=0; }]
> 2011-12-22 11:49:52 trace RECV
[192.168.169.27:5672-192.168.169.115:4518]:
> Frame[BEbe; channel=0; {ConnectionTuneOkBody: channel-max=32767;
> max-frame-size=65535; heartbeat=0; }]
>
> I am not familiar with the protocol at this level, but I suspect this is
the broker
> telling the client its min/max supported heartbeat intervals (both 0),
and
> then the client agreeing to use heartbeat interval 0.
>
> Does anyone recognize these issues? Does the C++ broker on windows
> support heartbeats at all?
>
> Thanks,
>
> Wolf Wolfswinkel
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Heartbeats-in-C-broker-on-Windows-
> tp7118702p7118702.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org