You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Sandeep Chayapathi <sa...@wssource.com> on 2006/11/06 23:17:37 UTC
Activemq 4.0.2 - stops sending message after some time
Hi all,
Recently I noticed that the activemq would stop sending message to the
consumers after some time. Here is a unscientific test to duplicate this
issue:
The test involved creating around 7 durable subscriptions and ctrl+c'ing
them. Once the durable subscriptions were registered, around 8 publisher
processes were started, each process wrote a 'fortune cookie' , in an
infiniteloop, to the activemq server. In parallel, 3 subscriber
processes were started. The 3 subscribers were randomly stopped and
started, cyclically. At about 15000 messages (on each of the 8 write
processes), activemq stopped sending the message to to the subscribers
and the publisher processes hung on "syswrite" to activemq. Then
another publisher process was started, activemq actually accepted
messages, but none of the subscribers received it.
The next test involved, using mysql as the data store (instead of the
default journalled db). The same steps were followed and at about 18000
messages (on each of the 8 processes), activemq stopped sending messages
to the subscribers.
Im using activemq 4.0.2. Any suggestions/help is appreciated. Thanks
- Sandeep
Re: Activemq 4.0.2 - stops sending message after some time
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Hi Sandeep,
Could you post a memory snapshot of the broker from a tool like
jprofiler or yourkit? That would help folks help you identify what is
holding the memory.
Also if you can simplify your use case down to 1 simple unit test it
would help folks be able to reproduce your problem.
On 11/7/06, Sandeep Chayapathi <sa...@wssource.com> wrote:
> Update:
>
> I was using Stomp protocol for both the Publishers and Consumers. A
> check with jconsole showed me that there were around 20,000 threads. To
> test it further, I modified my stomp library (written in perl), to close
> connection and re-connect, every 100 messages. Activemq, consistently
> stopped responding, and the memory used was around 400MB (with another
> 500 MB of physical memory free) and the number of threads were 20,000.
> On selecting the MBean tab in jconsole, I got java.net.connection exception.
>
> For the next test, I used the ProducerTool (bundled with the activemq
> binaries). The only change was that the ProducerTool would run in a
> while(true){} loop. The idea was to see if/when activemq would break.
> Again, there were 8 Producers and 3 Consuemers (of which, one of them
> was the bundled ConsumerTool and the remaining used Stomp). Around
> 400,000th message (roughly the same number on all 8 processes), the
> jconsole showed non-heap memory used as around 400MB and the number of
> threads at 200, when activemq stopped responding. I had enabled DEBUG
> messages, but no error message were reported in the log. I was able to
> use jconsole to look into the enqueue and dequeue count.
>
> This scenario can be duplicated using the bundled ProducerTool and
> ConsumerTool. I guess the average message/second was 150 per Producer
> process. Also a live environment can easily have 8 or more producers
> (although not with such a high message/second rate), also this set of
> tests show me that the Stomp implementation is not as stable as desired.
>
> Does anyone here use activemq on a production server, if so under what
> load and what is the availability of the activemq service ? How can I
> setup a stable activemq server in a high availability manner. Thanks
>
> - Sandeep
>
> Sandeep Chayapathi wrote:
> > Hi all,
> >
> > Recently I noticed that the activemq would stop sending message to
> > the consumers after some time. Here is a unscientific test to
> > duplicate this issue:
> > The test involved creating around 7 durable subscriptions and
> > ctrl+c'ing them. Once the durable subscriptions were registered,
> > around 8 publisher processes were started, each process wrote a
> > 'fortune cookie' , in an infiniteloop, to the activemq server. In
> > parallel, 3 subscriber processes were started. The 3 subscribers were
> > randomly stopped and started, cyclically. At about 15000 messages (on
> > each of the 8 write processes), activemq stopped sending the message
> > to to the subscribers and the publisher processes hung on "syswrite"
> > to activemq. Then another publisher process was started, activemq
> > actually accepted messages, but none of the subscribers received it.
> >
> > The next test involved, using mysql as the data store (instead of the
> > default journalled db). The same steps were followed and at about
> > 18000 messages (on each of the 8 processes), activemq stopped sending
> > messages to the subscribers.
> >
> > Im using activemq 4.0.2. Any suggestions/help is appreciated. Thanks
> >
> > - Sandeep
> >
>
>
--
Regards,
Hiram
Blog: http://hiramchirino.com
Re: Activemq 4.0.2 - stops sending message after some time
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Sandeep, the memory profile is indeed that needed bits. Could you
upload it to a publicly accessible website and post a link?
On 11/8/06, Sandeep Chayapathi <sa...@wssource.com> wrote:
> Here is the profile data from YourKit profiler (cpu and telemetry). I
> cannot attach the memory profile, as the zip file is quite large (70M).
> In case you need that file, mail me back. Later I will mail a simplified
> test case. Thanks.
>
> - Sandeep
>
>
--
Regards,
Hiram
Blog: http://hiramchirino.com
Re: Activemq 4.0.2 - stops sending message after some time
Posted by Sandeep Chayapathi <sa...@wssource.com>.
I have uploaded the memory profile on a file sharing site, the link is:
http://s25.quicksharing.com/v/8490663/your_kit_memory_profile.bzip2.html
(the direct link is:
http://s25.quicksharing.com/d/8490663/1163090148/your_kit_memory_profile.bzip2)
Note: the bzip2 file is around 60 MB in size and when unzipped, it
expands to about 260 MB.
- Sandeep
Sandeep Chayapathi wrote:
> Here is the profile data from YourKit profiler (cpu and telemetry). I
> cannot attach the memory profile, as the zip file is quite large
> (70M). In case you need that file, mail me back. Later I will mail a
> simplified test case. Thanks.
>
> - Sandeep
Re: Activemq 4.0.2 - stops sending message after some time
Posted by Sandeep Chayapathi <sa...@wssource.com>.
Here is the profile data from YourKit profiler (cpu and telemetry). I
cannot attach the memory profile, as the zip file is quite large (70M).
In case you need that file, mail me back. Later I will mail a simplified
test case. Thanks.
- Sandeep
Re: Activemq 4.0.2 - stops sending message after some time
Posted by Sandeep Chayapathi <sa...@wssource.com>.
Here is the profile data from YourKit profiler (cpu and telemetry). I
cannot attach the memory profile, as the zip file is quite large (70M).
In case you need that file, mail me back. Later I will mail a simplified
test case. Thanks.
- Sandeep
Re: Activemq 4.0.2 - stops sending message after some time
Posted by Sandeep Chayapathi <sa...@wssource.com>.
Update:
I was using Stomp protocol for both the Publishers and Consumers. A
check with jconsole showed me that there were around 20,000 threads. To
test it further, I modified my stomp library (written in perl), to close
connection and re-connect, every 100 messages. Activemq, consistently
stopped responding, and the memory used was around 400MB (with another
500 MB of physical memory free) and the number of threads were 20,000.
On selecting the MBean tab in jconsole, I got java.net.connection exception.
For the next test, I used the ProducerTool (bundled with the activemq
binaries). The only change was that the ProducerTool would run in a
while(true){} loop. The idea was to see if/when activemq would break.
Again, there were 8 Producers and 3 Consuemers (of which, one of them
was the bundled ConsumerTool and the remaining used Stomp). Around
400,000th message (roughly the same number on all 8 processes), the
jconsole showed non-heap memory used as around 400MB and the number of
threads at 200, when activemq stopped responding. I had enabled DEBUG
messages, but no error message were reported in the log. I was able to
use jconsole to look into the enqueue and dequeue count.
This scenario can be duplicated using the bundled ProducerTool and
ConsumerTool. I guess the average message/second was 150 per Producer
process. Also a live environment can easily have 8 or more producers
(although not with such a high message/second rate), also this set of
tests show me that the Stomp implementation is not as stable as desired.
Does anyone here use activemq on a production server, if so under what
load and what is the availability of the activemq service ? How can I
setup a stable activemq server in a high availability manner. Thanks
- Sandeep
Sandeep Chayapathi wrote:
> Hi all,
>
> Recently I noticed that the activemq would stop sending message to
> the consumers after some time. Here is a unscientific test to
> duplicate this issue:
> The test involved creating around 7 durable subscriptions and
> ctrl+c'ing them. Once the durable subscriptions were registered,
> around 8 publisher processes were started, each process wrote a
> 'fortune cookie' , in an infiniteloop, to the activemq server. In
> parallel, 3 subscriber processes were started. The 3 subscribers were
> randomly stopped and started, cyclically. At about 15000 messages (on
> each of the 8 write processes), activemq stopped sending the message
> to to the subscribers and the publisher processes hung on "syswrite"
> to activemq. Then another publisher process was started, activemq
> actually accepted messages, but none of the subscribers received it.
>
> The next test involved, using mysql as the data store (instead of the
> default journalled db). The same steps were followed and at about
> 18000 messages (on each of the 8 processes), activemq stopped sending
> messages to the subscribers.
>
> Im using activemq 4.0.2. Any suggestions/help is appreciated. Thanks
>
> - Sandeep
>
Re: Activemq 4.0.2 - stops sending message after some time
Posted by Sandeep Chayapathi <sa...@wssource.com>.
Update:
I was using Stomp protocol for both the Publishers and Consumers. A
check with jconsole showed me that there were around 20,000 threads. To
test it further, I modified my stomp library (written in perl), to close
connection and re-connect, every 100 messages. Activemq, consistently
stopped responding, and the memory used was around 400MB (with another
500 MB of physical memory free) and the number of threads were 20,000.
On selecting the MBean tab in jconsole, I got java.net.connection exception.
For the next test, I used the ProducerTool (bundled with the activemq
binaries). The only change was that the ProducerTool would run in a
while(true){} loop. The idea was to see if/when activemq would break.
Again, there were 8 Producers and 3 Consuemers (of which, one of them
was the bundled ConsumerTool and the remaining used Stomp). Around
400,000th message (roughly the same number on all 8 processes), the
jconsole showed non-heap memory used as around 400MB and the number of
threads at 200, when activemq stopped responding. I had enabled DEBUG
messages, but no error message were reported in the log. I was able to
use jconsole to look into the enqueue and dequeue count.
This scenario can be duplicated using the bundled ProducerTool and
ConsumerTool. I guess the average message/second was 150 per Producer
process. Also a live environment can easily have 8 or more producers
(although not with such a high message/second rate), also this set of
tests show me that the Stomp implementation is not as stable as desired.
Does anyone here use activemq on a production server, if so under what
load and what is the availability of the activemq service ? How can I
setup a stable activemq server in a high availability manner. Thanks
- Sandeep
Sandeep Chayapathi wrote:
> Hi all,
>
> Recently I noticed that the activemq would stop sending message to
> the consumers after some time. Here is a unscientific test to
> duplicate this issue:
> The test involved creating around 7 durable subscriptions and
> ctrl+c'ing them. Once the durable subscriptions were registered,
> around 8 publisher processes were started, each process wrote a
> 'fortune cookie' , in an infiniteloop, to the activemq server. In
> parallel, 3 subscriber processes were started. The 3 subscribers were
> randomly stopped and started, cyclically. At about 15000 messages (on
> each of the 8 write processes), activemq stopped sending the message
> to to the subscribers and the publisher processes hung on "syswrite"
> to activemq. Then another publisher process was started, activemq
> actually accepted messages, but none of the subscribers received it.
>
> The next test involved, using mysql as the data store (instead of the
> default journalled db). The same steps were followed and at about
> 18000 messages (on each of the 8 processes), activemq stopped sending
> messages to the subscribers.
>
> Im using activemq 4.0.2. Any suggestions/help is appreciated. Thanks
>
> - Sandeep
>