You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Toralf Lund <to...@pgs.com> on 2019/03/29 12:26:21 UTC
Force (C++) broker to drop connection and/or reset session
Another one related to issues I've mentioned in other recent posts:
I'm doing some debugging related to undesired side effects in one of our
applications after it gets a "Failed to connect (reconnect disabled)"
error while sending to or receiving from the C++ broker. In then
"manually" reopens the connection after a slight delay. As parts of the
reconnect logic, I also tend to get "'session-busy: Session detached by
peer'". This is possibly triggered by the connect itself, but it could
also come from other operations done just after connecting.
This would be a whole lot easier if I could provoke the errors into
occurring, so as to speak. I guess the "failed to connect" error means
the broker has dropped the connection, but is there a way I can force it
to do that (without restarting)? And how about the "Session detached by
peer"? It's not clear to me when exactly I get that, but I guess it's
related to sessions that existed before the connection was closed. Which
(according to information I got here earlier) Connection::open() might
try to recreate, but perhaps the operation fails? Can I somehow
manipulate the state so I can find out more about the behaviour?
Thanks.
- Toralf
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [External] Re: Force (C++) broker to drop connection and/or reset
session
Posted by Toralf Lund <to...@pgs.com>.
On 29/03/2019 15:50, Chuck Rolke wrote:
> An easy thing to try would be to turn on protocol logging. In qpidd.conf
> file or on the command line add
>
> --log-enable trace+:Protocol
>
> This would expose what led up to the session being rejected. The protocol
> logs will show if the broker is spontaneously closing the session or if
> the client(s) are violating AMQP rules in some way.
Yeah, maybe I'll try that. I'm a bit worried about the amount of log
data, though, seeing as the error doesn't occur that frequently.
>
> Dropping connections or resetting sessions from within the broker would
> be really hard. Testing has focused on having cruel clients who close
> connections and sessions at inopportune moments to test that the broker
> survives.
OK.
- Toralf
>
> -Chuck
>
>
> ----- Original Message -----
>> From: "Toralf Lund" <to...@pgs.com>
>> To: users@qpid.apache.org
>> Sent: Friday, March 29, 2019 8:26:21 AM
>> Subject: Force (C++) broker to drop connection and/or reset session
>>
>> Another one related to issues I've mentioned in other recent posts:
>>
>> I'm doing some debugging related to undesired side effects in one of our
>> applications after it gets a "Failed to connect (reconnect disabled)"
>> error while sending to or receiving from the C++ broker. In then
>> "manually" reopens the connection after a slight delay. As parts of the
>> reconnect logic, I also tend to get "'session-busy: Session detached by
>> peer'". This is possibly triggered by the connect itself, but it could
>> also come from other operations done just after connecting.
>>
>> This would be a whole lot easier if I could provoke the errors into
>> occurring, so as to speak. I guess the "failed to connect" error means
>> the broker has dropped the connection, but is there a way I can force it
>> to do that (without restarting)? And how about the "Session detached by
>> peer"? It's not clear to me when exactly I get that, but I guess it's
>> related to sessions that existed before the connection was closed. Which
>> (according to information I got here earlier) Connection::open() might
>> try to recreate, but perhaps the operation fails? Can I somehow
>> manipulate the state so I can find out more about the behaviour?
>>
>> Thanks.
>>
>> - Toralf
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Force (C++) broker to drop connection and/or reset session
Posted by Chuck Rolke <cr...@redhat.com>.
An easy thing to try would be to turn on protocol logging. In qpidd.conf
file or on the command line add
--log-enable trace+:Protocol
This would expose what led up to the session being rejected. The protocol
logs will show if the broker is spontaneously closing the session or if
the client(s) are violating AMQP rules in some way.
Dropping connections or resetting sessions from within the broker would
be really hard. Testing has focused on having cruel clients who close
connections and sessions at inopportune moments to test that the broker
survives.
-Chuck
----- Original Message -----
> From: "Toralf Lund" <to...@pgs.com>
> To: users@qpid.apache.org
> Sent: Friday, March 29, 2019 8:26:21 AM
> Subject: Force (C++) broker to drop connection and/or reset session
>
> Another one related to issues I've mentioned in other recent posts:
>
> I'm doing some debugging related to undesired side effects in one of our
> applications after it gets a "Failed to connect (reconnect disabled)"
> error while sending to or receiving from the C++ broker. In then
> "manually" reopens the connection after a slight delay. As parts of the
> reconnect logic, I also tend to get "'session-busy: Session detached by
> peer'". This is possibly triggered by the connect itself, but it could
> also come from other operations done just after connecting.
>
> This would be a whole lot easier if I could provoke the errors into
> occurring, so as to speak. I guess the "failed to connect" error means
> the broker has dropped the connection, but is there a way I can force it
> to do that (without restarting)? And how about the "Session detached by
> peer"? It's not clear to me when exactly I get that, but I guess it's
> related to sessions that existed before the connection was closed. Which
> (according to information I got here earlier) Connection::open() might
> try to recreate, but perhaps the operation fails? Can I somehow
> manipulate the state so I can find out more about the behaviour?
>
> Thanks.
>
> - Toralf
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [External] Re: Force (C++) broker to drop connection and/or reset
session
Posted by Toralf Lund <to...@pgs.com>.
On 01/04/2019 22:28, Gordon Sim wrote:
> On 29/03/2019 12:26 pm, Toralf Lund wrote:
>> Another one related to issues I've mentioned in other recent posts:
>>
>> I'm doing some debugging related to undesired side effects in one of
>> our applications after it gets a "Failed to connect (reconnect
>> disabled)" error while sending to or receiving from the C++ broker.
>> In then "manually" reopens the connection after a slight delay. As
>> parts of the reconnect logic, I also tend to get "'session-busy:
>> Session detached by peer'". This is possibly triggered by the connect
>> itself, but it could also come from other operations done just after
>> connecting.
>>
>> This would be a whole lot easier if I could provoke the errors into
>> occurring, so as to speak. I guess the "failed to connect" error
>> means the broker has dropped the connection, but is there a way I can
>> force it to do that (without restarting)? And how about the "Session
>> detached by peer"? It's not clear to me when exactly I get that, but
>> I guess it's related to sessions that existed before the connection
>> was closed. Which (according to information I got here earlier)
>> Connection::open() might try to recreate, but perhaps the operation
>> fails? Can I somehow manipulate the state so I can find out more
>> about the behaviour?
>
> You can specify the name for the session when you create it. So one
> approach might be to connect your client (the one you want to test),
> then kill -STOP that client, then start a test client that just opens
> a session with the same name (e.g. see attached), then kill -CONT the
> client which should then failover (assuming heartbeats were on) and
> hit the session-busy error.
Thanks. That helps quite I bit. The errors I get with this experiment
are consistent with what I'm seeing on the real installation - which I
guess suggests the "Session detached by peer" occurs because the session
already exists and is attached to another active connection (as seen
from the broker.) In the test case, it's quite definitely
Connection::open() itself that raises the exception. Furthermore, it
looks like it's quite all right to use the connection anyway, but the
session has to be set up again from scratch.
- Toralf
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
Re: Force (C++) broker to drop connection and/or reset session
Posted by Gordon Sim <gs...@redhat.com>.
On 29/03/2019 12:26 pm, Toralf Lund wrote:
> Another one related to issues I've mentioned in other recent posts:
>
> I'm doing some debugging related to undesired side effects in one of our
> applications after it gets a "Failed to connect (reconnect disabled)"
> error while sending to or receiving from the C++ broker. In then
> "manually" reopens the connection after a slight delay. As parts of the
> reconnect logic, I also tend to get "'session-busy: Session detached by
> peer'". This is possibly triggered by the connect itself, but it could
> also come from other operations done just after connecting.
>
> This would be a whole lot easier if I could provoke the errors into
> occurring, so as to speak. I guess the "failed to connect" error means
> the broker has dropped the connection, but is there a way I can force it
> to do that (without restarting)? And how about the "Session detached by
> peer"? It's not clear to me when exactly I get that, but I guess it's
> related to sessions that existed before the connection was closed. Which
> (according to information I got here earlier) Connection::open() might
> try to recreate, but perhaps the operation fails? Can I somehow
> manipulate the state so I can find out more about the behaviour?
You can specify the name for the session when you create it. So one
approach might be to connect your client (the one you want to test),
then kill -STOP that client, then start a test client that just opens a
session with the same name (e.g. see attached), then kill -CONT the
client which should then failover (assuming heartbeats were on) and hit
the session-busy error.