You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by NeilOwens <ne...@raymarine.com> on 2013/09/27 13:08:46 UTC
C++ Version 0.20: Rescuing a failed connection
This is a cross-compiled version running on an embedded ARM processor.
I'm seeing odd behaviour in reacquiring a connection to a remote unit that's
rebooted. I have created a MessageSender object that's managing all the
qpid objects and it's created a connection, session and sender to a topic on
a broker on a remote node packaged up as a BrokerSession object. However,
due to the system it's running on, the remote node can be restarted, and I'm
trying to get my qpid objects to recover. I've got a separate agent
monitoring heartbeats to other nodes in the system, so it's been notified to
deallocate the BrokerSession object.
The effect I'm seeing is that when I try to create a new connection object
and open the connection I get a TransportFailure exception thrown. I
attempt to back off and wait a few seconds before trying to open it again,
but then get the same exception thrown again. The odd thing is that if I
kill and restart the whole process, the process works perfectly. I think
that even though the qpid::messaging::connection object goes out of scope,
under the hood the Protocol Registry factory object is giving me the same
underlying ConnectionImpl object back again, and it's giving the same error.
Nothing I've tried short of that (deleting all the other qpid amalgam
objects I've got floating around).
Please note that my own reconnection implementation is needed because of
other system knowledge the higher level objects have so they can recreate
things when they know they're going to be up.
My questions:
a) Is there anything I can do to purge the ProtocolRegistry factory entry
for that ConnectionImpl object (well, short of changing the Connection
constructor). The Registry seems to be a static singleton, so only process
kill will delete those objects.
b) Is there anything I can call on the Connection object to get it out of
its current state? The BrokerSession object closes the connection in its
destructor, so is there any other cleanup I can do to stop this happening?
c) Is this fixed by a later release? I can always upgrade to a later
version of qpid of course, but I only want to do that if I can be sure it'll
fix the problems I'm seeing, I don't want that hassle (I have a bunch of
buildroot patch files I'd have to update to integrate the newer release).
Thanks in advance,
Neil
--
View this message in context: http://qpid.2158936.n2.nabble.com/C-Version-0-20-Rescuing-a-failed-connection-tp7598658.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: C++ Version 0.20: Rescuing a failed connection
Posted by Gordon Sim <gs...@redhat.com>.
On 10/07/2013 09:38 AM, NeilOwens wrote:
> Yes, I apply a 1 second heartbeat to the connection. We're currently seeing
> a 4 second connection timeout (this is elapsed time within connection.open()
> when the destination IP address doesn't exist). We don't really want to
> have to post the connect to a different thread, so we're looking at other
> options before we do that.
Hmm, I wonder why that is... I see the timeout firing after two seconds
as expected, e.g.
./src/tests/qpid-send --connection-options '{heartbeat:1}' --address abc
--broker <bad ip> --log-enable trace+
Gives:
> 2013-10-07 10:35:41 [Client] debug Created connection 192.56.0.111 with {heartbeat:1}
> 2013-10-07 10:35:41 [Client] debug Starting connection, urls=[192.56.0.111]
> 2013-10-07 10:35:41 [Client] info Trying to connect to 192.56.0.111...
> 2013-10-07 10:35:41 [Client] debug Created IO thread: 0
> 2013-10-07 10:35:41 [Network] debug TCPConnector created for 0-10
> 2013-10-07 10:35:41 [Client] info Set TCP_NODELAY
> 2013-10-07 10:35:41 [System] info Connecting: 192.56.0.111:5672
> 2013-10-07 10:35:43 [Client] debug Traffic timeout
> 2013-10-07 10:35:43 [Network] warning Connect failed: Connection timedout
> 2013-10-07 10:35:43 [Client] warning Connection closed
> 2013-10-07 10:35:43 [System] debug Exception constructed: Connection closed
> 2013-10-07 10:35:44 [Client] info Failed to connect to 192.56.0.111: Connection closed
> qpid-send: Failed to connect (reconnect disabled)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: C++ Version 0.20: Rescuing a failed connection
Posted by NeilOwens <ne...@raymarine.com>.
Yes, I apply a 1 second heartbeat to the connection. We're currently seeing
a 4 second connection timeout (this is elapsed time within connection.open()
when the destination IP address doesn't exist). We don't really want to
have to post the connect to a different thread, so we're looking at other
options before we do that.
--
View this message in context: http://qpid.2158936.n2.nabble.com/C-Version-0-20-Rescuing-a-failed-connection-tp7598658p7599102.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: C++ Version 0.20: Rescuing a failed connection
Posted by Gordon Sim <gs...@redhat.com>.
On 10/07/2013 08:27 AM, NeilOwens wrote:
> We used the debug code to resolve our issue. Reinstantiating the
> connection, session and senders seems less reliable in this scenario than
> keeping everything constructed, and just reopening the connection object. I
> don't know why letting the old objects go out of scope and creating them
> again didn't work.
>
> The next thing we're addressing is the connection time. There is a fairly
> long timeout on the ::connect function. Has anyone tried changing BSDSocket
> to do the set non-blocking -> connect -> select statement loop until socket
> is selected for write ->set non-blocking trick to reduce the timeout?
Have you tried setting a heartbeat? That should at least bound the
connect time as it will give up if it doesn't connect within two
heartbeat intervals. (That may of course not help, but just in case...)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: C++ Version 0.20: Rescuing a failed connection
Posted by NeilOwens <ne...@raymarine.com>.
We used the debug code to resolve our issue. Reinstantiating the
connection, session and senders seems less reliable in this scenario than
keeping everything constructed, and just reopening the connection object. I
don't know why letting the old objects go out of scope and creating them
again didn't work.
The next thing we're addressing is the connection time. There is a fairly
long timeout on the ::connect function. Has anyone tried changing BSDSocket
to do the set non-blocking -> connect -> select statement loop until socket
is selected for write ->set non-blocking trick to reduce the timeout?
--
View this message in context: http://qpid.2158936.n2.nabble.com/C-Version-0-20-Rescuing-a-failed-connection-tp7598658p7599096.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: C++ Version 0.20: Rescuing a failed connection
Posted by Gordon Sim <gs...@redhat.com>.
On 09/30/2013 11:46 AM, NeilOwens wrote:
> How do I enable the client-side logging?
The easiest is to set an environment variable: export QPID_LOG_ENABLE=debug+
If that isn't possible for whatever reason, you can also do it in code[1]:
qpid::log::Options opts;
opts.selectors.push_back("debug+");
qpid::log::Logger::instance().configure(opts);
[1] At least on older versions. On trunk these headers are currently not
accessible.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: C++ Version 0.20: Rescuing a failed connection
Posted by NeilOwens <ne...@raymarine.com>.
How do I enable the client-side logging?
--
View this message in context: http://qpid.2158936.n2.nabble.com/C-Version-0-20-Rescuing-a-failed-connection-tp7598658p7598802.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: C++ Version 0.20: Rescuing a failed connection
Posted by Gordon Sim <gs...@redhat.com>.
On 09/30/2013 11:24 AM, NeilOwens wrote:
> Thanks for your reply, Gordon.
>
> I hadn't confirmed using instrumentation that the same ConnectionImpl is
> being returned, it was just a (apparently wrong) guess looking at the code.
> It's somewhat difficult to isolate this scenario in a simpler situation. I
> have my code I'm calling:
>
> m_Connection = qpid::messaging::Connection("10.0.0.5:5672");
> m_Connection.setOption("heartbeat", 1);
> m_Connection.open();
>
> after 10.0.0.5 has been rebooted. I get a TransportFailure exception
> ("Failed to connect (reconnect disabled)") thrown from m_Connection.open(),
> and continues to do so until I restart the application. Destroying all of
> my amalgam objects doesn't seem to clear the fault, only restarting the
> whole linux process. This made me think that some kind of bad state was
> sticking around through all my objects going out of scope.
It could well be.
Another possibility that occurred to me (without knowledge of your code
of course), is that the reconnect took a different path resulting in the
open() being called on an object that wasn't initialised as expected for
some reason (e.g. a connection handle held in a map or something). From
the snippet above, that doesn't seem to be the case though.
> Looking at the Connection.cpp source made me think that the ConnectionImpl
> was a function of the options and protocol. However, I'm not supplying any
> options, so looking at ProtocolRegistry::create it should be returning 0 and
> so it should be creating a new ConnectionImpl object each time.
>
> However, the empirical observation is that something somewhere is continuing
> to maintain state that isn't backing off properly. I have an older version
> which still has the startup bug if you try and start the broker without
> --log-enable, so I've got that set to warning+ at the moment. I'll try
> increasing the logging level, but it's a busy broker so there's a lot of
> spam to pick through.
Actually I meant turn on logging on the client side. The broker logging
won't be that useful. What I really want to see is the logs from the
connection attempts.
> My next guess is that there's something in the Connector object that is
> failing to connect. I'm guessing the factory stuff in there will end up
> getting a TCPConnector, and thence doing stuff in posix::AsynchConnector and
> BSDSocket::connect. I'll add some more logging and get back to you.
>
> Thanks again
>
>
>
> --
> View this message in context: http://qpid.2158936.n2.nabble.com/C-Version-0-20-Rescuing-a-failed-connection-tp7598658p7598800.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: C++ Version 0.20: Rescuing a failed connection
Posted by NeilOwens <ne...@raymarine.com>.
Thanks for your reply, Gordon.
I hadn't confirmed using instrumentation that the same ConnectionImpl is
being returned, it was just a (apparently wrong) guess looking at the code.
It's somewhat difficult to isolate this scenario in a simpler situation. I
have my code I'm calling:
m_Connection = qpid::messaging::Connection("10.0.0.5:5672");
m_Connection.setOption("heartbeat", 1);
m_Connection.open();
after 10.0.0.5 has been rebooted. I get a TransportFailure exception
("Failed to connect (reconnect disabled)") thrown from m_Connection.open(),
and continues to do so until I restart the application. Destroying all of
my amalgam objects doesn't seem to clear the fault, only restarting the
whole linux process. This made me think that some kind of bad state was
sticking around through all my objects going out of scope.
Looking at the Connection.cpp source made me think that the ConnectionImpl
was a function of the options and protocol. However, I'm not supplying any
options, so looking at ProtocolRegistry::create it should be returning 0 and
so it should be creating a new ConnectionImpl object each time.
However, the empirical observation is that something somewhere is continuing
to maintain state that isn't backing off properly. I have an older version
which still has the startup bug if you try and start the broker without
--log-enable, so I've got that set to warning+ at the moment. I'll try
increasing the logging level, but it's a busy broker so there's a lot of
spam to pick through.
My next guess is that there's something in the Connector object that is
failing to connect. I'm guessing the factory stuff in there will end up
getting a TCPConnector, and thence doing stuff in posix::AsynchConnector and
BSDSocket::connect. I'll add some more logging and get back to you.
Thanks again
--
View this message in context: http://qpid.2158936.n2.nabble.com/C-Version-0-20-Rescuing-a-failed-connection-tp7598658p7598800.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: C++ Version 0.20: Rescuing a failed connection
Posted by Gordon Sim <gs...@redhat.com>.
On 09/27/2013 12:08 PM, NeilOwens wrote:
> This is a cross-compiled version running on an embedded ARM processor.
>
> I'm seeing odd behaviour in reacquiring a connection to a remote unit that's
> rebooted. I have created a MessageSender object that's managing all the
> qpid objects and it's created a connection, session and sender to a topic on
> a broker on a remote node packaged up as a BrokerSession object. However,
> due to the system it's running on, the remote node can be restarted, and I'm
> trying to get my qpid objects to recover. I've got a separate agent
> monitoring heartbeats to other nodes in the system, so it's been notified to
> deallocate the BrokerSession object.
>
> The effect I'm seeing is that when I try to create a new connection object
> and open the connection I get a TransportFailure exception thrown. I
> attempt to back off and wait a few seconds before trying to open it again,
> but then get the same exception thrown again. The odd thing is that if I
> kill and restart the whole process, the process works perfectly. I think
> that even though the qpid::messaging::connection object goes out of scope,
> under the hood the Protocol Registry factory object is giving me the same
> underlying ConnectionImpl object back again, and it's giving the same error.
> Nothing I've tried short of that (deleting all the other qpid amalgam
> objects I've got floating around).
>
> Please note that my own reconnection implementation is needed because of
> other system knowledge the higher level objects have so they can recreate
> things when they know they're going to be up.
>
> My questions:
>
> a) Is there anything I can do to purge the ProtocolRegistry factory entry
> for that ConnectionImpl object (well, short of changing the Connection
> constructor). The Registry seems to be a static singleton, so only process
> kill will delete those objects.
The ProtocolRegistry[1] simply holds function pointers to functions that
return a new object. There is nothing in the code that holds on to these
objects from with the factory function or registry.
The only factory function that is actually registered is for the 1.0
implementation. The 0-10 version is the default and handling for that is
built directly into the ProtocolRegistry. However again, a new object is
created every time and no reference or pointer to it is retained by the
registry.
Did you do some debugging to lead you to the suspicion that you are
getting the same impl object? I don't see how that could happen myself.
How are you creating the new connection object? Forgive the silly
question, but are you passing in the URL to use? If you use the default
constructor the url defaults to 127.0.0.1 which may not be what you want.
Have you tried turning up logging? E.g. export QPID_LOG_ENABLE=info+ or
even debug+?
> b) Is there anything I can call on the Connection object to get it out of
> its current state? The BrokerSession object closes the connection in its
> destructor, so is there any other cleanup I can do to stop this happening?
> c) Is this fixed by a later release? I can always upgrade to a later
> version of qpid of course, but I only want to do that if I can be sure it'll
> fix the problems I'm seeing, I don't want that hassle (I have a bunch of
> buildroot patch files I'd have to update to integrate the newer release).
--Gordon.
[1] There are actually two different ProtocolRegistry definitions, for
two different purposes. The first is for choosing the transport for an
0-10 connection (e.g. tcp, ssl, rdma), the second is for choosing the
higher level protoco or protocl version (e.g. AMQP 1.0 v. AMQP 0-10)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org