AMQ does not drop connection on stomp client ungraceful shutdown

[reachrichrulez <re...@gmail.com>]

we have an AMQ 5.3.2. running on server-1 and two perl stomp consumers on
server-2 and server-3 consuming from the same queue. I have 10 messages that
are being consumed (5 by each consumer)
when I shutdown one consumer gracefully, the other consumer gets the
messages instantly. 

Heres the issue - 
When I power off the on consumer box (case of a BCP event) the other
consumer does not get the messages untill 8 to 10 minutes after. When I
connected to the broker through jconsole. I still see the connection of the
powered-off box. When I stopped the connection (stop() button) the other
consumer started receiving the messages.

Can someone help?



--
View this message in context: http://activemq.2283324.n4.nabble.com/AMQ-does-not-drop-connection-on-stomp-client-ungraceful-shutdown-tp3639513p3639513.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: AMQ does not drop connection on stomp client ungraceful shutdown

[reachrichrulez <re...@gmail.com>]

I am doing this experiment in dev and there are no firewalls

Here is what I have concluded. 
KeepAlive comes into play when the channel is inactive. In my scenario the
broker kept sending the messages.

After ~10 minutes the connection disappeared from netstat, so broker closed
the connections too( in logs) and started delivering all messages to the
other active consumer. My theory is that it took ~10 min for tcp to confirm
(using send_buffer size and tcp retransmission timeout) that the client is
dead and cannot ACK the tcp packets.

 

--
View this message in context: http://activemq.2283324.n4.nabble.com/AMQ-does-not-drop-connection-on-stomp-client-ungraceful-shutdown-tp3639513p3663665.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: AMQ does not drop connection on stomp client ungraceful shutdown

[Ken Barber <ke...@puppetlabs.com>]

Setting tcp_keeliave_time is pretty normal practice in this case. The
client can only enable the SO_KEEPALIVE setting on a socket, but its
not responsible for the tcp_keepalive_time as you have discovered :-).

In regards to your earlier 10 minute drops do you have any firewalls
in between your clients and the STOMP connector? Some firewalls and
load-balancing devices will only keep state for a short time - which
may explain the 10 minutes perhaps. Checkpoint defaults to an hour for
example - but I've seen F5 load-balancers dropping state after 5
minutes ... its user configurable in both cases however.

ken.

On Tue, Jul 12, 2011 at 2:50 AM, reachrichrulez
<re...@gmail.com> wrote:
> Martin,
> Thanks for your response.
>
> I set the ?transport.keepAlive=true on the stomp TransportConnector url on
> the broker, but there was no effect. jconsole still showed the connection
> for a long time. It appears that this setting relies on the OS
> tcp_keepalive_time which is defaulted to 2 hours
>
> Even after the power cable was pulled off, netstat -a still showed the
> connection. Then I modifed the linux tcp settings (tcp_keepalive_time,
> tcp_keepalive_intvl and tcp_keepalive_probes) and then the connections were
> closed within a few minutes. Even jconsole reflected the same.
>
> What makes me wonder is that the default for the tcp_keepalive_time is 7200
> seconds(2 hours), but in my original test case scenario the broker
> recognized that powered off consumer in ~10 min in sent all the messages to
> the other connected consumer. Where is this ~10 min configured?
>
> Also, It appears from the source code that inactivity monitor for stomp i
> disabled.
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/AMQ-does-not-drop-connection-on-stomp-client-ungraceful-shutdown-tp3639513p3661277.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
"Join us for PuppetConf, September 22nd and 23rd in Portland, OR:
http://bit.ly/puppetconfsig"

Re: AMQ does not drop connection on stomp client ungraceful shutdown

[reachrichrulez <re...@gmail.com>]

Martin, 
Thanks for your response. 

I set the ?transport.keepAlive=true on the stomp TransportConnector url on
the broker, but there was no effect. jconsole still showed the connection
for a long time. It appears that this setting relies on the OS
tcp_keepalive_time which is defaulted to 2 hours

Even after the power cable was pulled off, netstat -a still showed the
connection. Then I modifed the linux tcp settings (tcp_keepalive_time,
tcp_keepalive_intvl and tcp_keepalive_probes) and then the connections were
closed within a few minutes. Even jconsole reflected the same. 

What makes me wonder is that the default for the tcp_keepalive_time is 7200
seconds(2 hours), but in my original test case scenario the broker
recognized that powered off consumer in ~10 min in sent all the messages to
the other connected consumer. Where is this ~10 min configured?

Also, It appears from the source code that inactivity monitor for stomp i
disabled.




--
View this message in context: http://activemq.2283324.n4.nabble.com/AMQ-does-not-drop-connection-on-stomp-client-ungraceful-shutdown-tp3639513p3661277.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: AMQ does not drop connection on stomp client ungraceful shutdown

["Martin C." <ma...@gmx.at>]

Hi,

it sounds to me that it simply takes that long for the STOMP protocol
to realize that the connection died. Maybe you can enable
tcp.keepalive on the connection via the configuration URL?

You might also have a look at the keepalive options for your transport
connection, and if you are on Linux and cannot change the options, you
may want to look into
http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/ which also
describes a LD_PRELOAD based approach to set TCP keepalive options for
sockets of foreign programs.

Best regards,
Martin

On Sat, Jul 2, 2011 at 1:10 AM, reachrichrulez <re...@gmail.com> wrote:
> we have an AMQ 5.3.2. running on server-1 and two perl stomp consumers on
> server-2 and server-3 consuming from the same queue. I have 10 messages that
> are being consumed (5 by each consumer)
> when I shutdown one consumer gracefully, the other consumer gets the
> messages instantly.
>
> Heres the issue -
> When I power off the on consumer box (case of a BCP event) the other
> consumer does not get the messages untill 8 to 10 minutes after. When I
> connected to the broker through jconsole. I still see the connection of the
> powered-off box. When I stopped the connection (stop() button) the other
> consumer started receiving the messages.
>
> Can someone help?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/AMQ-does-not-drop-connection-on-stomp-client-ungraceful-shutdown-tp3639513p3639513.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>