Slow starting activemq connection

[KaZso <zs...@gmail.com>]

Hi,

I am facing a strange performance issue with an activemq broker (v. 5.4.0). 
The setup is very simple:
- 1 broker instance
- no persistence provider
- about 200 consumers and producers using non-persistent messages. The
message size can be anything up to 1-2MB but is usually smaller.
- dedicated task runner has been disabled
- producer flow control enabled

Most message producers open and close their connections for every message
sent but the number of messages can vary from 5-10 per day to 20-30 every
minute, meaning that we do have thousands of connections being opened and
closed every day. 

This is working well enough with delivery times of up to a few hundred
milliseconds, but every know and then there is a "hiccup". Starting the jms
connection slows down to tens of seconds for no apparent reason, then
immediately picks up again. Message delivery of already established
connections are not affected, and the slowdowns seem totally random. Number
of threads is mostly constant, and the memory levels seem fine as well, with
the total garbage collection time being lower than one single slowdown.

No exceptions or warnings can be found in the logs, and the slow connections
do eventually start. The message delivery on them is fast and not
problematic at all.

Could anyone help me in finding the root cause of this issue? Could the
producer flow control delay the start of a new connection but not the flow
of messages?

Thank you,
Zsombor



--
View this message in context: http://activemq.2283324.n4.nabble.com/Slow-starting-activemq-connection-tp4661562.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Slow starting activemq connection

[KaZso <zs...@gmail.com>]

Hi Gaurav,

thank you for your reply. Xms and xmx are both set to 1G (CMS collector is
used and explicit GC disabled, nothing else), and the system usage is as
follows:

<systemUsage>
            <systemUsage>
                 <memoryUsage>
                        <memoryUsage limit="500 mb" />
                 </memoryUsage>
                 <storeUsage>
                          <storeUsage limit="1 gb" />
                 </storeUsage>
                 <tempUsage>
                          <tempUsage limit="200 mb" />
                 </tempUsage>
           </systemUsage>
 </systemUsage>

However I'm not sure this could be GC related for two reasons:

1. Producers which are reusing their jms connections do not experience any
slowness, they are completely impact free. I guess a stop the world GC would
slow them down as well.
2. According to JConsole the total GC time is very low. Seconds on ParNew
and even less on CMS. While the connection slowdowns can be tens of seconds.

We are using tcp transport, the RAM on the host running it is tens of
gigabytes, but it is of course running other applications as well. The load
average on the host is normal.

And yes the slowness seems to occur more frequently when a higher number of
large messages are being sent, but I couldn't reproduce it yet in the
development environment with our load tests.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Slow-starting-activemq-connection-tp4661562p4661706.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Slow starting activemq connection

[Gaurav Sharma <ga...@gmail.com>]

Okay, so, you have been able to somewhat correlate the slow connection establishment to load?

Are you using tcp or nio transport? And for the host running the broker, how much is the:
a) RAM 
b) Heap for the JVM
c) SystemUsage settings

I suspect your broker jvm is undergoing a full gc.

On Jan 10, 2013, at 0:09, KaZso <zs...@gmail.com> wrote:

> Thanks for the quick reply. 
> Yes, the network and the machine load were the prime suspects at first, but
> then we deployed a second broker with the same config but different ports on
> the same host, and a test application started opening / closing connections
> to both of the in parallel every couple of seconds. 
> The slowdown could only be experienced on the broker under load (the second
> broker had no consumers/producers. It wasn't doing anything except sit there
> and wait for connections, but a network glitch or a sporadic high load on
> the server should have affected it as well).
> 
> As for the second question, no particular reason except a slip up in design
> that has been with us for years.
> 
> 
> 
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Slow-starting-activemq-connection-tp4661562p4661586.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Slow starting activemq connection

[KaZso <zs...@gmail.com>]

Thanks for the quick reply. 
Yes, the network and the machine load were the prime suspects at first, but
then we deployed a second broker with the same config but different ports on
the same host, and a test application started opening / closing connections
to both of the in parallel every couple of seconds. 
The slowdown could only be experienced on the broker under load (the second
broker had no consumers/producers. It wasn't doing anything except sit there
and wait for connections, but a network glitch or a sporadic high load on
the server should have affected it as well).

As for the second question, no particular reason except a slip up in design
that has been with us for years.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Slow-starting-activemq-connection-tp4661562p4661586.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Slow starting activemq connection

[Gaurav Sharma <ga...@gmail.com>]

It could be a number of things. But first, a couple questions: 

1. how does your network look like - 2/3 level switches in the DC, flat or something else? Where are the producers deployed relative to the broker clusters? Do you have tcp keep-alive turned on, spl routing, firewalls, etc?

2. any reason for not using the pooled connection factory?

I would immediately write a long running producer->broker test and deploy on a couple machines in the 'same' network going against the 'same' broker nodes, let it run continuously overnight, capture packets via wireshark, etc and analyze the drops. Loop in your network engg. to help sample-trace the routes during your tests.

Hope this helps or maybe you already did these analyses and didn't find anything? Let us know either way.

On Jan 9, 2013, at 10:31, KaZso <zs...@gmail.com> wrote:

> Hi,
> 
> I am facing a strange performance issue with an activemq broker (v. 5.4.0). 
> The setup is very simple:
> - 1 broker instance
> - no persistence provider
> - about 200 consumers and producers using non-persistent messages. The
> message size can be anything up to 1-2MB but is usually smaller.
> - dedicated task runner has been disabled
> - producer flow control enabled
> 
> Most message producers open and close their connections for every message
> sent but the number of messages can vary from 5-10 per day to 20-30 every
> minute, meaning that we do have thousands of connections being opened and
> closed every day. 
> 
> This is working well enough with delivery times of up to a few hundred
> milliseconds, but every know and then there is a "hiccup". Starting the jms
> connection slows down to tens of seconds for no apparent reason, then
> immediately picks up again. Message delivery of already established
> connections are not affected, and the slowdowns seem totally random. Number
> of threads is mostly constant, and the memory levels seem fine as well, with
> the total garbage collection time being lower than one single slowdown.
> 
> No exceptions or warnings can be found in the logs, and the slow connections
> do eventually start. The message delivery on them is fast and not
> problematic at all.
> 
> Could anyone help me in finding the root cause of this issue? Could the
> producer flow control delay the start of a new connection but not the flow
> of messages?
> 
> Thank you,
> Zsombor
> 
> 
> 
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Slow-starting-activemq-connection-tp4661562.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.