ActiveMQ Slowdown

[Anthrope <pn...@anthrope.com>]

Hello,
     Our application is essentially a pipeline of processing engines that
communicate via JMS queues, and so far, ActiveMQ has been the JMS
implementation of our choice.  In our application, engines produce data and
dumping it onto a queue, and a number of engines read that data, process it
and dump them onto another queue that is in turn read from by other engines
downstream, and so on. We normally have one ActiveMQ instance serving one
queue, but that is configurable. The messages range in size from 300k to 1M
at the most.

Over the course of time, we've faced  different issues vis-a-vis performance
or stability but the problems have eventually been solved. Now, we're close
to going into production and are facing a serious hurdle that threatens to
delay our rollout.

The problem is this: over time, we notice that the average time it takes to
read from or write to a queue gradually increases until things become so
sluggish that we need to restart ActiveMQ. Our processing engines do not
need to be restarted because we're using the failover URL, and they
automatically reconnect. Once we restart ActiveMQ, things run fine for a
while before the sluggishness sets in. I have noticed no leaks in our own
application code, and ActiveMQ itself seems to be fine from a memory usage
standpoint.

Our configuration is as follows:

- ActiveMQ 4.1.1
- Linux  2.6.9-34.ELsmp
- Java 1.5.0_09
- Were using queues with transacted sessions
- We're using MySQL as the persistent store (Please not that we see this
problem arise with ALL store implementations, including our own homegrown
BDB store, which is why I am convinced it is not the store itself)

Does anyone have an idea as to where I might look, to solve this problem?

Thanks,
Prashanth


-- 
View this message in context: http://www.nabble.com/ActiveMQ-Slowdown-tf4038323s2354.html#a11473091
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ Slowdown

[naga007 <NS...@gmail.com>]

I see u are using 4.1.1 and we are using DMX storage(SAN) to persist data and
its very fast.
 i see that u mentioned that performance degrades overtime( is it mins or
hours or days?). What is ur throughput rate and how much memory did u
configure for MemoryUsageManager?



naga007 wrote:
> 
> 
> which version of activemq are u using and Can u post ur activemq.xml
> file?.
> 
> 
> 
> Anthrope wrote:
>> 
>> Hello,
>>      Our application is essentially a pipeline of processing engines that
>> communicate via JMS queues, and so far, ActiveMQ has been the JMS
>> implementation of our choice.  In our application, engines produce data
>> and dumping it onto a queue, and a number of engines read that data,
>> process it and dump them onto another queue that is in turn read from by
>> other engines downstream, and so on. We normally have one ActiveMQ
>> instance serving one queue, but that is configurable. The messages range
>> in size from 300k to 1M at the most.
>> 
>> Over the course of time, we've faced  different issues vis-a-vis
>> performance or stability but the problems have eventually been solved.
>> Now, we're close to going into production and are facing a serious hurdle
>> that threatens to delay our rollout.
>> 
>> The problem is this: over time, we notice that the average time it takes
>> to read from or write to a queue gradually increases until things become
>> so sluggish that we need to restart ActiveMQ. Our processing engines do
>> not need to be restarted because we're using the failover URL, and they
>> automatically reconnect. Once we restart ActiveMQ, things run fine for a
>> while before the sluggishness sets in. I have noticed no leaks in our own
>> application code, and ActiveMQ itself seems to be fine from a memory
>> usage standpoint.
>> 
>> Our configuration is as follows:
>> 
>> - ActiveMQ 4.1.1
>> - Linux  2.6.9-34.ELsmp
>> - Java 1.5.0_09
>> - Were using queues with transacted sessions
>> - We're using MySQL as the persistent store (Please not that we see this
>> problem arise with ALL store implementations, including our own homegrown
>> BDB store, which is why I am convinced it is not the store itself)
>> 
>> Does anyone have an idea as to where I might look, to solve this problem?
>> 
>> Thanks,
>> Prashanth
>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/ActiveMQ-Slowdown-tf4038323s2354.html#a11502118
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ Slowdown

[naga007 <NS...@gmail.com>]

which version of activemq are u using and Can u post ur activemq.xml file?.



Anthrope wrote:
> 
> Hello,
>      Our application is essentially a pipeline of processing engines that
> communicate via JMS queues, and so far, ActiveMQ has been the JMS
> implementation of our choice.  In our application, engines produce data
> and dumping it onto a queue, and a number of engines read that data,
> process it and dump them onto another queue that is in turn read from by
> other engines downstream, and so on. We normally have one ActiveMQ
> instance serving one queue, but that is configurable. The messages range
> in size from 300k to 1M at the most.
> 
> Over the course of time, we've faced  different issues vis-a-vis
> performance or stability but the problems have eventually been solved.
> Now, we're close to going into production and are facing a serious hurdle
> that threatens to delay our rollout.
> 
> The problem is this: over time, we notice that the average time it takes
> to read from or write to a queue gradually increases until things become
> so sluggish that we need to restart ActiveMQ. Our processing engines do
> not need to be restarted because we're using the failover URL, and they
> automatically reconnect. Once we restart ActiveMQ, things run fine for a
> while before the sluggishness sets in. I have noticed no leaks in our own
> application code, and ActiveMQ itself seems to be fine from a memory usage
> standpoint.
> 
> Our configuration is as follows:
> 
> - ActiveMQ 4.1.1
> - Linux  2.6.9-34.ELsmp
> - Java 1.5.0_09
> - Were using queues with transacted sessions
> - We're using MySQL as the persistent store (Please not that we see this
> problem arise with ALL store implementations, including our own homegrown
> BDB store, which is why I am convinced it is not the store itself)
> 
> Does anyone have an idea as to where I might look, to solve this problem?
> 
> Thanks,
> Prashanth
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/ActiveMQ-Slowdown-tf4038323s2354.html#a11501886
Sent from the ActiveMQ - User mailing list archive at Nabble.com.