You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Igor Bogicevic <ca...@gmail.com> on 2006/07/01 00:48:43 UTC
Re: problems with ActiveMQ with large number of producers/consumers
I also forgot to answer on this thread... this was not the problem
related to the lack of the memory, it's related to hanging
connections, i've set up the broker with the large amount of memory
(>2GB) in both broker configuration and via jconsole and problem was
defintely in the connections which were not cleaned up - for unknown
reason to me. it was easy to block queue with large number of
un-properly closed sessions and relatively hard with properly closed
connections in a shutdown hook.
thanks,
igor
On 6/30/06, skarthik <ka...@necam.com> wrote:
>
> Hi Hiram,
>
> I was wondering what is the relationship between the Broker memory limit set
> in the broker XML and the JVM heap size specified in when launching the
> broker. For example should the heap size be some extra percentage of the
> configured broker memory? I guess the broker memory can nenver be greater
> than the -Xmx value.
>
> thanks,
> karthik
> --
> View this message in context: http://www.nabble.com/problems-with-ActiveMQ-with-large-number-of-producers-consumers-tf1823629.html#a5126960
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
Re: problems with ActiveMQ with large number of producers/consumers
Posted by Igor Bogicevic <ca...@gmail.com>.
Thanks Hiram,
I'll definitely check it out!
Thanks,
Igor
On 7/3/06, Hiram Chirino <hi...@hiramchirino.com> wrote:
> I Igor,
>
> I just recently fixed a issue where a consumer could be created and not
> properly cleaned up if an async exception occured. See
> http://issues.apache.org/activemq/browse/AMQ-724. I have a feeling that
> this could have been the cause of the hung consumers.
>
> Could you retest against a subsequent 4.1-SNAPSHOT or 4.0.2-SNAPSHOT?
>
> Thanks!
>
> On 7/3/06, Igor Bogicevic <ca...@gmail.com> wrote:
> >
> > Hi James,
> >
> > well, it doesn't depend on the prefetch value (I have tried with 1000
> > and 2000 prefetch messages, and without prefetching at all), and what
> > happens is this:
> > 1) create about 60 producers and 60 consumers (I have attached the
> > small and ugly test I used - basically I've forked about 60 processes
> > on each side, which means they had separate sessions and connections)
> > 2) if you cut out the shutdown hook in this test, which handles
> > cleanup of connections and sessions, and you kill large amount of
> > producers/consumers this queue will be blocked in the terms that
> > neither producers or consumers will be able to either send or receive
> > messages on this queue - sometimes this happens for a period of a
> > 20-30s and very often for good (>10h, that's the longest period i've
> > left it before i've restarted broker).
> > 3) in case there is shutdown hook with properly closing sessions and
> > connections, this is very hard to repeat, and in case of 4.0.0 i've
> > managed to block the queue with repetitve starting/killing large
> > number of producers and consumers, but i've never managed to reproduce
> > this with 4.0.1
> > 4) i could see a large number of hanging connections (jmx console
> > showed much more connections than there actually were)
> > 4) after restarting broker all works like a charm
> >
> > Thanks,
> > Igor
> >
> > On 7/3/06, James Strachan <ja...@gmail.com> wrote:
> > > By 'blocking the queue' do you just mean messages were not dispatched
> > > to consumers? If so how many messages were on the queue, how many
> > > connections & consumers did the broker think were open & what were the
> > > prefetch values? If you are having trouble of dead connections not
> > > being closed down agressively enough then your issue could just be an
> > > effect of messages being placed into prefetch buffers for clients that
> > > have not been detected as failed yet? Do you have any kind of JUnit
> > > test case that can replicate your issue?
> > >
> > > On 7/3/06, Igor Bogicevic <ca...@gmail.com> wrote:
> > > > I am using activemq 4.0.1 (I've did tests on the both 4.0.0 and 4.0.1)
> > > > and I have attached activemq.xml (basically, almost out of the box
> > > > configuration). I am using persistent queues, and I have tried with
> > > > prefetching and transactions, but that didn't really change the
> > > > behavior.
> > > >
> > > > Thanks,
> > > > Igor
> > > >
> > > > On 7/3/06, James Strachan <ja...@gmail.com> wrote:
> > > > > On 6/30/06, Igor Bogicevic <ca...@gmail.com> wrote:
> > > > > > I also forgot to answer on this thread... this was not the problem
> > > > > > related to the lack of the memory, it's related to hanging
> > > > > > connections, i've set up the broker with the large amount of
> > memory
> > > > > > (>2GB) in both broker configuration and via jconsole and problem
> > was
> > > > > > defintely in the connections which were not cleaned up - for
> > unknown
> > > > > > reason to me. it was easy to block queue with large number of
> > > > > > un-properly closed sessions and relatively hard with properly
> > closed
> > > > > > connections in a shutdown hook.
> > > > >
> > > > > Using what version and what quality of service? e.g. are we talking
> > > > > 4.0.1 and non-persistent queues? What was your XML config file?
> > > > >
> > > > > --
> > > > >
> > > > > James
> > > > > -------
> > > > > http://radio.weblogs.com/0112098/
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > James
> > > -------
> > > http://radio.weblogs.com/0112098/
> > >
> >
> >
> >
>
>
> --
> Regards,
> Hiram
>
> Blog: http://hiramchirino.com
>
>
Re: problems with ActiveMQ with large number of producers/consumers
Posted by Hiram Chirino <hi...@hiramchirino.com>.
I Igor,
I just recently fixed a issue where a consumer could be created and not
properly cleaned up if an async exception occured. See
http://issues.apache.org/activemq/browse/AMQ-724. I have a feeling that
this could have been the cause of the hung consumers.
Could you retest against a subsequent 4.1-SNAPSHOT or 4.0.2-SNAPSHOT?
Thanks!
On 7/3/06, Igor Bogicevic <ca...@gmail.com> wrote:
>
> Hi James,
>
> well, it doesn't depend on the prefetch value (I have tried with 1000
> and 2000 prefetch messages, and without prefetching at all), and what
> happens is this:
> 1) create about 60 producers and 60 consumers (I have attached the
> small and ugly test I used - basically I've forked about 60 processes
> on each side, which means they had separate sessions and connections)
> 2) if you cut out the shutdown hook in this test, which handles
> cleanup of connections and sessions, and you kill large amount of
> producers/consumers this queue will be blocked in the terms that
> neither producers or consumers will be able to either send or receive
> messages on this queue - sometimes this happens for a period of a
> 20-30s and very often for good (>10h, that's the longest period i've
> left it before i've restarted broker).
> 3) in case there is shutdown hook with properly closing sessions and
> connections, this is very hard to repeat, and in case of 4.0.0 i've
> managed to block the queue with repetitve starting/killing large
> number of producers and consumers, but i've never managed to reproduce
> this with 4.0.1
> 4) i could see a large number of hanging connections (jmx console
> showed much more connections than there actually were)
> 4) after restarting broker all works like a charm
>
> Thanks,
> Igor
>
> On 7/3/06, James Strachan <ja...@gmail.com> wrote:
> > By 'blocking the queue' do you just mean messages were not dispatched
> > to consumers? If so how many messages were on the queue, how many
> > connections & consumers did the broker think were open & what were the
> > prefetch values? If you are having trouble of dead connections not
> > being closed down agressively enough then your issue could just be an
> > effect of messages being placed into prefetch buffers for clients that
> > have not been detected as failed yet? Do you have any kind of JUnit
> > test case that can replicate your issue?
> >
> > On 7/3/06, Igor Bogicevic <ca...@gmail.com> wrote:
> > > I am using activemq 4.0.1 (I've did tests on the both 4.0.0 and 4.0.1)
> > > and I have attached activemq.xml (basically, almost out of the box
> > > configuration). I am using persistent queues, and I have tried with
> > > prefetching and transactions, but that didn't really change the
> > > behavior.
> > >
> > > Thanks,
> > > Igor
> > >
> > > On 7/3/06, James Strachan <ja...@gmail.com> wrote:
> > > > On 6/30/06, Igor Bogicevic <ca...@gmail.com> wrote:
> > > > > I also forgot to answer on this thread... this was not the problem
> > > > > related to the lack of the memory, it's related to hanging
> > > > > connections, i've set up the broker with the large amount of
> memory
> > > > > (>2GB) in both broker configuration and via jconsole and problem
> was
> > > > > defintely in the connections which were not cleaned up - for
> unknown
> > > > > reason to me. it was easy to block queue with large number of
> > > > > un-properly closed sessions and relatively hard with properly
> closed
> > > > > connections in a shutdown hook.
> > > >
> > > > Using what version and what quality of service? e.g. are we talking
> > > > 4.0.1 and non-persistent queues? What was your XML config file?
> > > >
> > > > --
> > > >
> > > > James
> > > > -------
> > > > http://radio.weblogs.com/0112098/
> > > >
> > >
> > >
> > >
> >
> >
> > --
> >
> > James
> > -------
> > http://radio.weblogs.com/0112098/
> >
>
>
>
--
Regards,
Hiram
Blog: http://hiramchirino.com
Re: problems with ActiveMQ with large number of producers/consumers
Posted by Igor Bogicevic <ca...@gmail.com>.
Hi James,
well, it doesn't depend on the prefetch value (I have tried with 1000
and 2000 prefetch messages, and without prefetching at all), and what
happens is this:
1) create about 60 producers and 60 consumers (I have attached the
small and ugly test I used - basically I've forked about 60 processes
on each side, which means they had separate sessions and connections)
2) if you cut out the shutdown hook in this test, which handles
cleanup of connections and sessions, and you kill large amount of
producers/consumers this queue will be blocked in the terms that
neither producers or consumers will be able to either send or receive
messages on this queue - sometimes this happens for a period of a
20-30s and very often for good (>10h, that's the longest period i've
left it before i've restarted broker).
3) in case there is shutdown hook with properly closing sessions and
connections, this is very hard to repeat, and in case of 4.0.0 i've
managed to block the queue with repetitve starting/killing large
number of producers and consumers, but i've never managed to reproduce
this with 4.0.1
4) i could see a large number of hanging connections (jmx console
showed much more connections than there actually were)
4) after restarting broker all works like a charm
Thanks,
Igor
On 7/3/06, James Strachan <ja...@gmail.com> wrote:
> By 'blocking the queue' do you just mean messages were not dispatched
> to consumers? If so how many messages were on the queue, how many
> connections & consumers did the broker think were open & what were the
> prefetch values? If you are having trouble of dead connections not
> being closed down agressively enough then your issue could just be an
> effect of messages being placed into prefetch buffers for clients that
> have not been detected as failed yet? Do you have any kind of JUnit
> test case that can replicate your issue?
>
> On 7/3/06, Igor Bogicevic <ca...@gmail.com> wrote:
> > I am using activemq 4.0.1 (I've did tests on the both 4.0.0 and 4.0.1)
> > and I have attached activemq.xml (basically, almost out of the box
> > configuration). I am using persistent queues, and I have tried with
> > prefetching and transactions, but that didn't really change the
> > behavior.
> >
> > Thanks,
> > Igor
> >
> > On 7/3/06, James Strachan <ja...@gmail.com> wrote:
> > > On 6/30/06, Igor Bogicevic <ca...@gmail.com> wrote:
> > > > I also forgot to answer on this thread... this was not the problem
> > > > related to the lack of the memory, it's related to hanging
> > > > connections, i've set up the broker with the large amount of memory
> > > > (>2GB) in both broker configuration and via jconsole and problem was
> > > > defintely in the connections which were not cleaned up - for unknown
> > > > reason to me. it was easy to block queue with large number of
> > > > un-properly closed sessions and relatively hard with properly closed
> > > > connections in a shutdown hook.
> > >
> > > Using what version and what quality of service? e.g. are we talking
> > > 4.0.1 and non-persistent queues? What was your XML config file?
> > >
> > > --
> > >
> > > James
> > > -------
> > > http://radio.weblogs.com/0112098/
> > >
> >
> >
> >
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
Re: problems with ActiveMQ with large number of producers/consumers
Posted by James Strachan <ja...@gmail.com>.
By 'blocking the queue' do you just mean messages were not dispatched
to consumers? If so how many messages were on the queue, how many
connections & consumers did the broker think were open & what were the
prefetch values? If you are having trouble of dead connections not
being closed down agressively enough then your issue could just be an
effect of messages being placed into prefetch buffers for clients that
have not been detected as failed yet? Do you have any kind of JUnit
test case that can replicate your issue?
On 7/3/06, Igor Bogicevic <ca...@gmail.com> wrote:
> I am using activemq 4.0.1 (I've did tests on the both 4.0.0 and 4.0.1)
> and I have attached activemq.xml (basically, almost out of the box
> configuration). I am using persistent queues, and I have tried with
> prefetching and transactions, but that didn't really change the
> behavior.
>
> Thanks,
> Igor
>
> On 7/3/06, James Strachan <ja...@gmail.com> wrote:
> > On 6/30/06, Igor Bogicevic <ca...@gmail.com> wrote:
> > > I also forgot to answer on this thread... this was not the problem
> > > related to the lack of the memory, it's related to hanging
> > > connections, i've set up the broker with the large amount of memory
> > > (>2GB) in both broker configuration and via jconsole and problem was
> > > defintely in the connections which were not cleaned up - for unknown
> > > reason to me. it was easy to block queue with large number of
> > > un-properly closed sessions and relatively hard with properly closed
> > > connections in a shutdown hook.
> >
> > Using what version and what quality of service? e.g. are we talking
> > 4.0.1 and non-persistent queues? What was your XML config file?
> >
> > --
> >
> > James
> > -------
> > http://radio.weblogs.com/0112098/
> >
>
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: problems with ActiveMQ with large number of producers/consumers
Posted by Igor Bogicevic <ca...@gmail.com>.
I am using activemq 4.0.1 (I've did tests on the both 4.0.0 and 4.0.1)
and I have attached activemq.xml (basically, almost out of the box
configuration). I am using persistent queues, and I have tried with
prefetching and transactions, but that didn't really change the
behavior.
Thanks,
Igor
On 7/3/06, James Strachan <ja...@gmail.com> wrote:
> On 6/30/06, Igor Bogicevic <ca...@gmail.com> wrote:
> > I also forgot to answer on this thread... this was not the problem
> > related to the lack of the memory, it's related to hanging
> > connections, i've set up the broker with the large amount of memory
> > (>2GB) in both broker configuration and via jconsole and problem was
> > defintely in the connections which were not cleaned up - for unknown
> > reason to me. it was easy to block queue with large number of
> > un-properly closed sessions and relatively hard with properly closed
> > connections in a shutdown hook.
>
> Using what version and what quality of service? e.g. are we talking
> 4.0.1 and non-persistent queues? What was your XML config file?
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
Re: problems with ActiveMQ with large number of producers/consumers
Posted by James Strachan <ja...@gmail.com>.
On 6/30/06, Igor Bogicevic <ca...@gmail.com> wrote:
> I also forgot to answer on this thread... this was not the problem
> related to the lack of the memory, it's related to hanging
> connections, i've set up the broker with the large amount of memory
> (>2GB) in both broker configuration and via jconsole and problem was
> defintely in the connections which were not cleaned up - for unknown
> reason to me. it was easy to block queue with large number of
> un-properly closed sessions and relatively hard with properly closed
> connections in a shutdown hook.
Using what version and what quality of service? e.g. are we talking
4.0.1 and non-persistent queues? What was your XML config file?
--
James
-------
http://radio.weblogs.com/0112098/