Problems with prefetch and TemporaryQueues

[Zemus <si...@tieto.com>]

Hi,

I have a problem where TemporaryQueues are left (with 0 consumers) after the
applications creating them have finished.

This scenario occurs for both ActiveMQ 4.1.1 and 5.3.0 on my P4 2.6 GHz, 3
GB RAM, Ubuntu 9.10, Sun JDK, default settings for ActiveMQ.

I've tried to recreate a simpler version of the scenario. Basically I have a
server that creates a Topic where clients that wants to subscribe may post a
message (with a TemporaryQueue as the JMS reply) in order to be added. The
server then has a thread that sends messages to all the TemporaryQueues. (I
know a simple Topic could push the updates here, but in the real system the
clients can receive information only intended for one single client.)

In my scenario the server is faster at generating messages than the clients
are in processing them so the queues start to build up. Pretty soon (if
clients are added and removed once a minute or so) it will hit a limit and
the processing of messages in the broker seems to come to a halt. I think it
is when the prefetch limit is reached.
The real problem here is that if a client disconnects successfully (might
hang in the broker communication) it is possible for the server to
"resurrect" its TemporaryQueue, probably by sending prefetchbuffered
messages to it. This way the TemporaryQueue is left on the broker with no
consumer (the client disconnected) with its queued up messages counting
towards the prefetch limit.

By running JConsole and purging the queue I can get the broker to wake up,
but if many messages are queued up the consumerless TempQueue will queue
those messages which might again cause the prefetch limit to be reached and
halt the broker.
Also, I've seen old TemporaryQueues that were removed minutes ago come back
and start queueing buffered messages when I start purging other consumerless
TempQueues.


In short I think there are two problems here.
First, the detection of the client disconnection is not instant on the
server. This makes it possible to send messages to the TemporaryQueue
without getting an exception after it should have been removed.

The second is that the TemporaryQueue is able to resurrect after its
creator/consumer has left. In other words it has outlived its connection.


I attach a simple client and server that are able to reproduce this reliably
(at least on my computer).
Just run the server and maybe two clients. Restart the clients perhaps 1-2
times per minute and the issue will show pretty quickly. At least on my P4,
maybe a computer with multiple cores will behave different.

Regards
Zemus

http://old.nabble.com/file/p27130529/TestClient.java TestClient.java 
http://old.nabble.com/file/p27130529/ClientShutdownHook.java
ClientShutdownHook.java 

http://old.nabble.com/file/p27130529/TestServer.java TestServer.java 
http://old.nabble.com/file/p27130529/ServerShutdownHook.java
ServerShutdownHook.java 
-- 
View this message in context: http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27130529.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problems with prefetch and TemporaryQueues

[Zemus <si...@tieto.com>]

I have now registered this as a Jira issue along with a JUnit test case:
http://issues.apache.org/activemq/browse/AMQ-2571

I added another observation.
"When the TempQueue is auto(re)created it is not removed when the server
closes its connection as one (or at least I) would expect. Also, the server
gets an InvalidDestinationException if I, at server side, try to connect a
consumer to the TempQueue. I guess this means that although the server is
the initiator for the auto(re)creation it does not become the owner of the
TempQueue. But who is then the owner? The broker itself?"

Best Regards
Simon
-- 
View this message in context: http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27184578.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problems with prefetch and TemporaryQueues

[Gary Tully <ga...@gmail.com>]

on second thoughts, the network of brokers support, which bridges temp
destinations by default, that needs (or builds on) auto create of temp
destinations. So the option to turn off auto creation for temp destinations
could be exposed as a configuration option on the broker but it can't be the
default behavior.

2010/1/14 Gary Tully <ga...@gmail.com>

> Great that you have a solution that works.
> I think you have a good point here. I cannot think of a reason to have auto
> create for temp queues save that it negates the need to call
> session.createTempX which is not well behaved w.r.t JMS semantics. Can you
> open a jira issue to track this?
>
> As to the Junit test case, really all you need to do is follow some of the
> existing test cases in activemq-core/src/test - look for something with
> *Temp*Test in the file name and you should get close to what you need.
>
> 2010/1/14 Zemus <si...@tieto.com>
>
>
>> Thanks for the answer, Gary.
>>
>> I rebuilt ActiveMQ with
>> setAutoCreateDestinations(false);
>> in the constructor of TempQueueRegion.
>> It seems to solve the problem. I put some logging into
>> AbstractRegion.lookup
>> to verify this.
>>
>> Is there any reason to have autocreating enabled by default for
>> TempQueues?
>> Since the creator of the queue is the only that can consume from it, that
>> would mean that the sender creating the queue has the only connection that
>> can read from it. Or am I missing something?
>>
>> Perhaps the default value should be changed to false if the common usage
>> of
>> TempQueue benefits from it?
>>
>> As for a JUnit test case I will have to ask my employers if I can spend
>> time
>> on that. Are there any ActiveMQ specific guidelines for how they should be
>> constructed?
>>
>> Best Regards
>> Zemus
>>
>>
>>
>> Gary Tully wrote:
>> >
>> > this sounds like a reasonable theory.
>> > An ActiveMQConnection by default registers interest in the advisory
>> > messages
>> > for temp destination removal so it does try and track temp queues as you
>> > suggest.
>> > So if advisory support is enabled for the broker the window for
>> recreation
>> > of a temp queue should be small.
>> >
>> > However I think there could be a window between temp queue deletion and
>> > the
>> > advisory message dispatch such that the replying connection (what you
>> deem
>> > server) could consider the queue valid when it is not, resulting in a
>> new
>> > temp queue auto creation.
>> >
>> > Can you make a junit test case from what you have and attach it to a new
>> > jira issue.
>> >
>> > I see two solutions, have the broker optionally not auto create temp
>> > destinations or have it send out the advisory messages for temp
>> > destination
>> > deletion before the deletion happens. (this will still depend on the
>> > consumer in a connection reacting to the advisory though which will
>> still
>> > leave a window when a connection can think the temp queue is valid - it
>> > will
>> > help but may not be sufficient.)
>> >
>> > Exposing an option to not auto create temp destinations may be the
>> > simplest
>> > solution. Such an option is currently on the
>> >
>> org.apache.activemq.broker.region.AbstractRegion.setAutoCreateDestinations(boolean)
>> > if you control the broker start you may want to explore disabling this
>> > through code. Not sure if all that you need is there in the api at the
>> > moment, but it could provide a workaround that is worth investigating a
>> > bit.
>> >
>> > 2010/1/13 Zemus <si...@tieto.com>
>> >
>> >>
>> >>
>> >> Zemus wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > I have a problem where TemporaryQueues are left (with 0 consumers)
>> >> after
>> >> > the applications creating them have finished.
>> >> >
>> >> > This scenario occurs for both ActiveMQ 4.1.1 and 5.3.0 on my P4 2.6
>> >> GHz,
>> >> 3
>> >> > GB RAM, Ubuntu 9.10, Sun JDK, default settings for ActiveMQ.
>> >> >
>> >> > I've tried to recreate a simpler version of the scenario. Basically I
>> >> have
>> >> > a server that creates a Topic where clients that wants to subscribe
>> may
>> >> > post a message (with a TemporaryQueue as the JMS reply) in order to
>> be
>> >> > added. The server then has a thread that sends messages to all the
>> >> > TemporaryQueues. (I know a simple Topic could push the updates here,
>> >> but
>> >> > in the real system the clients can receive information only intended
>> >> for
>> >> > one single client.)
>> >> >
>> >> > In my scenario the server is faster at generating messages than the
>> >> > clients are in processing them so the queues start to build up.
>> Pretty
>> >> > soon (if clients are added and removed once a minute or so) it will
>> hit
>> >> a
>> >> > limit and the processing of messages in the broker seems to come to a
>> >> > halt. I think it is when the prefetch limit is reached.
>> >> > The real problem here is that if a client disconnects successfully
>> >> (might
>> >> > hang in the broker communication) it is possible for the server to
>> >> > "resurrect" its TemporaryQueue, probably by sending prefetchbuffered
>> >> > messages to it. This way the TemporaryQueue is left on the broker
>> with
>> >> no
>> >> > consumer (the client disconnected) with its queued up messages
>> counting
>> >> > towards the prefetch limit.
>> >> >
>> >> > By running JConsole and purging the queue I can get the broker to
>> wake
>> >> up,
>> >> > but if many messages are queued up the consumerless TempQueue will
>> >> queue
>> >> > those messages which might again cause the prefetch limit to be
>> reached
>> >> > and halt the broker.
>> >> > Also, I've seen old TemporaryQueues that were removed minutes ago
>> come
>> >> > back and start queueing buffered messages when I start purging other
>> >> > consumerless TempQueues.
>> >> >
>> >> >
>> >> > In short I think there are two problems here.
>> >> > First, the detection of the client disconnection is not instant on
>> the
>> >> > server. This makes it possible to send messages to the TemporaryQueue
>> >> > without getting an exception after it should have been removed.
>> >> >
>> >> > The second is that the TemporaryQueue is able to resurrect after its
>> >> > creator/consumer has left. In other words it has outlived its
>> >> connection.
>> >> >
>> >> >
>> >> > I attach a simple client and server that are able to reproduce this
>> >> > reliably (at least on my computer).
>> >> > Just run the server and maybe two clients. Restart the clients
>> perhaps
>> >> 1-2
>> >> > times per minute and the issue will show pretty quickly. At least on
>> my
>> >> > P4, maybe a computer with multiple cores will behave different.
>> >> >
>> >> > Regards
>> >> > Zemus
>> >> >
>> >> >  http://old.nabble.com/file/p27130529/TestClient.javaTestClient.java
>> >> >  http://old.nabble.com/file/p27130529/ClientShutdownHook.java
>> >> > ClientShutdownHook.java
>> >> >
>> >> >  http://old.nabble.com/file/p27130529/TestServer.javaTestServer.java
>> >> >  http://old.nabble.com/file/p27130529/ServerShutdownHook.java
>> >> > ServerShutdownHook.java
>> >> >
>> >>
>> >>
>> >> After more investigation I've changed my mind about the nature of the
>> >> problem. The prefetch limit is not involved.
>> >>
>> >> My new theory goes like this:
>> >>
>> >> 1. The temporary queue is removed correctly when the client exists and
>> >> closes its connection.
>> >>
>> >> 2. The server might receive an exception when it tries to send to the
>> >> removed temporary queue. BUT sometimes it doesn't get that exception
>> >> (perhaps because the broker was soo busy that it didn't have time to
>> >> notify
>> >> the activemq part of the server in time) and instead the broker creates
>> a
>> >> new queue with the same name, although without any consumer, when the
>> >> server
>> >> sends the next message to the stored destination.
>> >>
>> >> 3. The server knows nothing about this and keeps sending messages that
>> >> pile
>> >> up in the broker, which in turn, eventually runs out of memory (at
>> least
>> >> jconsole reports MemoryPercentageUsed 100 for the broker object).
>> >>
>> >>
>> >> If this sounds reasonable, my questions are instead:
>> >>
>> >> * How can I prevent the broker from creating a new temporary queue
>> >> because
>> >> of the server's late message?
>> >>
>> >> * Would listening to advisory support for the temp queue at the server
>> >> side
>> >> notify me in time to stop the sending of any more messages when the
>> temp
>> >> queue is removed?
>> >>
>> >> * Or is there a better way that would stop the server from sending a
>> >> message
>> >> to a removed temp queue?
>> >>
>> >> Regards
>> >> Zemus
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27147013.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source Integration
>> > http://fusesource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27162131.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>



-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Re: Problems with prefetch and TemporaryQueues

[Gary Tully <ga...@gmail.com>]

Great that you have a solution that works.
I think you have a good point here. I cannot think of a reason to have auto
create for temp queues save that it negates the need to call
session.createTempX which is not well behaved w.r.t JMS semantics. Can you
open a jira issue to track this?

As to the Junit test case, really all you need to do is follow some of the
existing test cases in activemq-core/src/test - look for something with
*Temp*Test in the file name and you should get close to what you need.

2010/1/14 Zemus <si...@tieto.com>

>
> Thanks for the answer, Gary.
>
> I rebuilt ActiveMQ with
> setAutoCreateDestinations(false);
> in the constructor of TempQueueRegion.
> It seems to solve the problem. I put some logging into
> AbstractRegion.lookup
> to verify this.
>
> Is there any reason to have autocreating enabled by default for TempQueues?
> Since the creator of the queue is the only that can consume from it, that
> would mean that the sender creating the queue has the only connection that
> can read from it. Or am I missing something?
>
> Perhaps the default value should be changed to false if the common usage of
> TempQueue benefits from it?
>
> As for a JUnit test case I will have to ask my employers if I can spend
> time
> on that. Are there any ActiveMQ specific guidelines for how they should be
> constructed?
>
> Best Regards
> Zemus
>
>
>
> Gary Tully wrote:
> >
> > this sounds like a reasonable theory.
> > An ActiveMQConnection by default registers interest in the advisory
> > messages
> > for temp destination removal so it does try and track temp queues as you
> > suggest.
> > So if advisory support is enabled for the broker the window for
> recreation
> > of a temp queue should be small.
> >
> > However I think there could be a window between temp queue deletion and
> > the
> > advisory message dispatch such that the replying connection (what you
> deem
> > server) could consider the queue valid when it is not, resulting in a new
> > temp queue auto creation.
> >
> > Can you make a junit test case from what you have and attach it to a new
> > jira issue.
> >
> > I see two solutions, have the broker optionally not auto create temp
> > destinations or have it send out the advisory messages for temp
> > destination
> > deletion before the deletion happens. (this will still depend on the
> > consumer in a connection reacting to the advisory though which will still
> > leave a window when a connection can think the temp queue is valid - it
> > will
> > help but may not be sufficient.)
> >
> > Exposing an option to not auto create temp destinations may be the
> > simplest
> > solution. Such an option is currently on the
> >
> org.apache.activemq.broker.region.AbstractRegion.setAutoCreateDestinations(boolean)
> > if you control the broker start you may want to explore disabling this
> > through code. Not sure if all that you need is there in the api at the
> > moment, but it could provide a workaround that is worth investigating a
> > bit.
> >
> > 2010/1/13 Zemus <si...@tieto.com>
> >
> >>
> >>
> >> Zemus wrote:
> >> >
> >> > Hi,
> >> >
> >> > I have a problem where TemporaryQueues are left (with 0 consumers)
> >> after
> >> > the applications creating them have finished.
> >> >
> >> > This scenario occurs for both ActiveMQ 4.1.1 and 5.3.0 on my P4 2.6
> >> GHz,
> >> 3
> >> > GB RAM, Ubuntu 9.10, Sun JDK, default settings for ActiveMQ.
> >> >
> >> > I've tried to recreate a simpler version of the scenario. Basically I
> >> have
> >> > a server that creates a Topic where clients that wants to subscribe
> may
> >> > post a message (with a TemporaryQueue as the JMS reply) in order to be
> >> > added. The server then has a thread that sends messages to all the
> >> > TemporaryQueues. (I know a simple Topic could push the updates here,
> >> but
> >> > in the real system the clients can receive information only intended
> >> for
> >> > one single client.)
> >> >
> >> > In my scenario the server is faster at generating messages than the
> >> > clients are in processing them so the queues start to build up. Pretty
> >> > soon (if clients are added and removed once a minute or so) it will
> hit
> >> a
> >> > limit and the processing of messages in the broker seems to come to a
> >> > halt. I think it is when the prefetch limit is reached.
> >> > The real problem here is that if a client disconnects successfully
> >> (might
> >> > hang in the broker communication) it is possible for the server to
> >> > "resurrect" its TemporaryQueue, probably by sending prefetchbuffered
> >> > messages to it. This way the TemporaryQueue is left on the broker with
> >> no
> >> > consumer (the client disconnected) with its queued up messages
> counting
> >> > towards the prefetch limit.
> >> >
> >> > By running JConsole and purging the queue I can get the broker to wake
> >> up,
> >> > but if many messages are queued up the consumerless TempQueue will
> >> queue
> >> > those messages which might again cause the prefetch limit to be
> reached
> >> > and halt the broker.
> >> > Also, I've seen old TemporaryQueues that were removed minutes ago come
> >> > back and start queueing buffered messages when I start purging other
> >> > consumerless TempQueues.
> >> >
> >> >
> >> > In short I think there are two problems here.
> >> > First, the detection of the client disconnection is not instant on the
> >> > server. This makes it possible to send messages to the TemporaryQueue
> >> > without getting an exception after it should have been removed.
> >> >
> >> > The second is that the TemporaryQueue is able to resurrect after its
> >> > creator/consumer has left. In other words it has outlived its
> >> connection.
> >> >
> >> >
> >> > I attach a simple client and server that are able to reproduce this
> >> > reliably (at least on my computer).
> >> > Just run the server and maybe two clients. Restart the clients perhaps
> >> 1-2
> >> > times per minute and the issue will show pretty quickly. At least on
> my
> >> > P4, maybe a computer with multiple cores will behave different.
> >> >
> >> > Regards
> >> > Zemus
> >> >
> >> >  http://old.nabble.com/file/p27130529/TestClient.java TestClient.java
> >> >  http://old.nabble.com/file/p27130529/ClientShutdownHook.java
> >> > ClientShutdownHook.java
> >> >
> >> >  http://old.nabble.com/file/p27130529/TestServer.java TestServer.java
> >> >  http://old.nabble.com/file/p27130529/ServerShutdownHook.java
> >> > ServerShutdownHook.java
> >> >
> >>
> >>
> >> After more investigation I've changed my mind about the nature of the
> >> problem. The prefetch limit is not involved.
> >>
> >> My new theory goes like this:
> >>
> >> 1. The temporary queue is removed correctly when the client exists and
> >> closes its connection.
> >>
> >> 2. The server might receive an exception when it tries to send to the
> >> removed temporary queue. BUT sometimes it doesn't get that exception
> >> (perhaps because the broker was soo busy that it didn't have time to
> >> notify
> >> the activemq part of the server in time) and instead the broker creates
> a
> >> new queue with the same name, although without any consumer, when the
> >> server
> >> sends the next message to the stored destination.
> >>
> >> 3. The server knows nothing about this and keeps sending messages that
> >> pile
> >> up in the broker, which in turn, eventually runs out of memory (at least
> >> jconsole reports MemoryPercentageUsed 100 for the broker object).
> >>
> >>
> >> If this sounds reasonable, my questions are instead:
> >>
> >> * How can I prevent the broker from creating a new temporary queue
> >> because
> >> of the server's late message?
> >>
> >> * Would listening to advisory support for the temp queue at the server
> >> side
> >> notify me in time to stop the sending of any more messages when the temp
> >> queue is removed?
> >>
> >> * Or is there a better way that would stop the server from sending a
> >> message
> >> to a removed temp queue?
> >>
> >> Regards
> >> Zemus
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27147013.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27162131.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Re: Problems with prefetch and TemporaryQueues

[Zemus <si...@tieto.com>]

Thanks for the answer, Gary.

I rebuilt ActiveMQ with
setAutoCreateDestinations(false);
in the constructor of TempQueueRegion.
It seems to solve the problem. I put some logging into AbstractRegion.lookup
to verify this.

Is there any reason to have autocreating enabled by default for TempQueues?
Since the creator of the queue is the only that can consume from it, that
would mean that the sender creating the queue has the only connection that
can read from it. Or am I missing something?

Perhaps the default value should be changed to false if the common usage of
TempQueue benefits from it?

As for a JUnit test case I will have to ask my employers if I can spend time
on that. Are there any ActiveMQ specific guidelines for how they should be
constructed?

Best Regards
Zemus



Gary Tully wrote:
> 
> this sounds like a reasonable theory.
> An ActiveMQConnection by default registers interest in the advisory
> messages
> for temp destination removal so it does try and track temp queues as you
> suggest.
> So if advisory support is enabled for the broker the window for recreation
> of a temp queue should be small.
> 
> However I think there could be a window between temp queue deletion and
> the
> advisory message dispatch such that the replying connection (what you deem
> server) could consider the queue valid when it is not, resulting in a new
> temp queue auto creation.
> 
> Can you make a junit test case from what you have and attach it to a new
> jira issue.
> 
> I see two solutions, have the broker optionally not auto create temp
> destinations or have it send out the advisory messages for temp
> destination
> deletion before the deletion happens. (this will still depend on the
> consumer in a connection reacting to the advisory though which will still
> leave a window when a connection can think the temp queue is valid - it
> will
> help but may not be sufficient.)
> 
> Exposing an option to not auto create temp destinations may be the
> simplest
> solution. Such an option is currently on the
> org.apache.activemq.broker.region.AbstractRegion.setAutoCreateDestinations(boolean)
> if you control the broker start you may want to explore disabling this
> through code. Not sure if all that you need is there in the api at the
> moment, but it could provide a workaround that is worth investigating a
> bit.
> 
> 2010/1/13 Zemus <si...@tieto.com>
> 
>>
>>
>> Zemus wrote:
>> >
>> > Hi,
>> >
>> > I have a problem where TemporaryQueues are left (with 0 consumers)
>> after
>> > the applications creating them have finished.
>> >
>> > This scenario occurs for both ActiveMQ 4.1.1 and 5.3.0 on my P4 2.6
>> GHz,
>> 3
>> > GB RAM, Ubuntu 9.10, Sun JDK, default settings for ActiveMQ.
>> >
>> > I've tried to recreate a simpler version of the scenario. Basically I
>> have
>> > a server that creates a Topic where clients that wants to subscribe may
>> > post a message (with a TemporaryQueue as the JMS reply) in order to be
>> > added. The server then has a thread that sends messages to all the
>> > TemporaryQueues. (I know a simple Topic could push the updates here,
>> but
>> > in the real system the clients can receive information only intended
>> for
>> > one single client.)
>> >
>> > In my scenario the server is faster at generating messages than the
>> > clients are in processing them so the queues start to build up. Pretty
>> > soon (if clients are added and removed once a minute or so) it will hit
>> a
>> > limit and the processing of messages in the broker seems to come to a
>> > halt. I think it is when the prefetch limit is reached.
>> > The real problem here is that if a client disconnects successfully
>> (might
>> > hang in the broker communication) it is possible for the server to
>> > "resurrect" its TemporaryQueue, probably by sending prefetchbuffered
>> > messages to it. This way the TemporaryQueue is left on the broker with
>> no
>> > consumer (the client disconnected) with its queued up messages counting
>> > towards the prefetch limit.
>> >
>> > By running JConsole and purging the queue I can get the broker to wake
>> up,
>> > but if many messages are queued up the consumerless TempQueue will
>> queue
>> > those messages which might again cause the prefetch limit to be reached
>> > and halt the broker.
>> > Also, I've seen old TemporaryQueues that were removed minutes ago come
>> > back and start queueing buffered messages when I start purging other
>> > consumerless TempQueues.
>> >
>> >
>> > In short I think there are two problems here.
>> > First, the detection of the client disconnection is not instant on the
>> > server. This makes it possible to send messages to the TemporaryQueue
>> > without getting an exception after it should have been removed.
>> >
>> > The second is that the TemporaryQueue is able to resurrect after its
>> > creator/consumer has left. In other words it has outlived its
>> connection.
>> >
>> >
>> > I attach a simple client and server that are able to reproduce this
>> > reliably (at least on my computer).
>> > Just run the server and maybe two clients. Restart the clients perhaps
>> 1-2
>> > times per minute and the issue will show pretty quickly. At least on my
>> > P4, maybe a computer with multiple cores will behave different.
>> >
>> > Regards
>> > Zemus
>> >
>> >  http://old.nabble.com/file/p27130529/TestClient.java TestClient.java
>> >  http://old.nabble.com/file/p27130529/ClientShutdownHook.java
>> > ClientShutdownHook.java
>> >
>> >  http://old.nabble.com/file/p27130529/TestServer.java TestServer.java
>> >  http://old.nabble.com/file/p27130529/ServerShutdownHook.java
>> > ServerShutdownHook.java
>> >
>>
>>
>> After more investigation I've changed my mind about the nature of the
>> problem. The prefetch limit is not involved.
>>
>> My new theory goes like this:
>>
>> 1. The temporary queue is removed correctly when the client exists and
>> closes its connection.
>>
>> 2. The server might receive an exception when it tries to send to the
>> removed temporary queue. BUT sometimes it doesn't get that exception
>> (perhaps because the broker was soo busy that it didn't have time to
>> notify
>> the activemq part of the server in time) and instead the broker creates a
>> new queue with the same name, although without any consumer, when the
>> server
>> sends the next message to the stored destination.
>>
>> 3. The server knows nothing about this and keeps sending messages that
>> pile
>> up in the broker, which in turn, eventually runs out of memory (at least
>> jconsole reports MemoryPercentageUsed 100 for the broker object).
>>
>>
>> If this sounds reasonable, my questions are instead:
>>
>> * How can I prevent the broker from creating a new temporary queue
>> because
>> of the server's late message?
>>
>> * Would listening to advisory support for the temp queue at the server
>> side
>> notify me in time to stop the sending of any more messages when the temp
>> queue is removed?
>>
>> * Or is there a better way that would stop the server from sending a
>> message
>> to a removed temp queue?
>>
>> Regards
>> Zemus
>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27147013.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> -- 
> http://blog.garytully.com
> 
> Open Source Integration
> http://fusesource.com
> 
> 

-- 
View this message in context: http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27162131.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problems with prefetch and TemporaryQueues

[Gary Tully <ga...@gmail.com>]

this sounds like a reasonable theory.
An ActiveMQConnection by default registers interest in the advisory messages
for temp destination removal so it does try and track temp queues as you
suggest.
So if advisory support is enabled for the broker the window for recreation
of a temp queue should be small.

However I think there could be a window between temp queue deletion and the
advisory message dispatch such that the replying connection (what you deem
server) could consider the queue valid when it is not, resulting in a new
temp queue auto creation.

Can you make a junit test case from what you have and attach it to a new
jira issue.

I see two solutions, have the broker optionally not auto create temp
destinations or have it send out the advisory messages for temp destination
deletion before the deletion happens. (this will still depend on the
consumer in a connection reacting to the advisory though which will still
leave a window when a connection can think the temp queue is valid - it will
help but may not be sufficient.)

Exposing an option to not auto create temp destinations may be the simplest
solution. Such an option is currently on the
org.apache.activemq.broker.region.AbstractRegion.setAutoCreateDestinations(boolean)
if you control the broker start you may want to explore disabling this
through code. Not sure if all that you need is there in the api at the
moment, but it could provide a workaround that is worth investigating a bit.

2010/1/13 Zemus <si...@tieto.com>

>
>
> Zemus wrote:
> >
> > Hi,
> >
> > I have a problem where TemporaryQueues are left (with 0 consumers) after
> > the applications creating them have finished.
> >
> > This scenario occurs for both ActiveMQ 4.1.1 and 5.3.0 on my P4 2.6 GHz,
> 3
> > GB RAM, Ubuntu 9.10, Sun JDK, default settings for ActiveMQ.
> >
> > I've tried to recreate a simpler version of the scenario. Basically I
> have
> > a server that creates a Topic where clients that wants to subscribe may
> > post a message (with a TemporaryQueue as the JMS reply) in order to be
> > added. The server then has a thread that sends messages to all the
> > TemporaryQueues. (I know a simple Topic could push the updates here, but
> > in the real system the clients can receive information only intended for
> > one single client.)
> >
> > In my scenario the server is faster at generating messages than the
> > clients are in processing them so the queues start to build up. Pretty
> > soon (if clients are added and removed once a minute or so) it will hit a
> > limit and the processing of messages in the broker seems to come to a
> > halt. I think it is when the prefetch limit is reached.
> > The real problem here is that if a client disconnects successfully (might
> > hang in the broker communication) it is possible for the server to
> > "resurrect" its TemporaryQueue, probably by sending prefetchbuffered
> > messages to it. This way the TemporaryQueue is left on the broker with no
> > consumer (the client disconnected) with its queued up messages counting
> > towards the prefetch limit.
> >
> > By running JConsole and purging the queue I can get the broker to wake
> up,
> > but if many messages are queued up the consumerless TempQueue will queue
> > those messages which might again cause the prefetch limit to be reached
> > and halt the broker.
> > Also, I've seen old TemporaryQueues that were removed minutes ago come
> > back and start queueing buffered messages when I start purging other
> > consumerless TempQueues.
> >
> >
> > In short I think there are two problems here.
> > First, the detection of the client disconnection is not instant on the
> > server. This makes it possible to send messages to the TemporaryQueue
> > without getting an exception after it should have been removed.
> >
> > The second is that the TemporaryQueue is able to resurrect after its
> > creator/consumer has left. In other words it has outlived its connection.
> >
> >
> > I attach a simple client and server that are able to reproduce this
> > reliably (at least on my computer).
> > Just run the server and maybe two clients. Restart the clients perhaps
> 1-2
> > times per minute and the issue will show pretty quickly. At least on my
> > P4, maybe a computer with multiple cores will behave different.
> >
> > Regards
> > Zemus
> >
> >  http://old.nabble.com/file/p27130529/TestClient.java TestClient.java
> >  http://old.nabble.com/file/p27130529/ClientShutdownHook.java
> > ClientShutdownHook.java
> >
> >  http://old.nabble.com/file/p27130529/TestServer.java TestServer.java
> >  http://old.nabble.com/file/p27130529/ServerShutdownHook.java
> > ServerShutdownHook.java
> >
>
>
> After more investigation I've changed my mind about the nature of the
> problem. The prefetch limit is not involved.
>
> My new theory goes like this:
>
> 1. The temporary queue is removed correctly when the client exists and
> closes its connection.
>
> 2. The server might receive an exception when it tries to send to the
> removed temporary queue. BUT sometimes it doesn't get that exception
> (perhaps because the broker was soo busy that it didn't have time to notify
> the activemq part of the server in time) and instead the broker creates a
> new queue with the same name, although without any consumer, when the
> server
> sends the next message to the stored destination.
>
> 3. The server knows nothing about this and keeps sending messages that pile
> up in the broker, which in turn, eventually runs out of memory (at least
> jconsole reports MemoryPercentageUsed 100 for the broker object).
>
>
> If this sounds reasonable, my questions are instead:
>
> * How can I prevent the broker from creating a new temporary queue because
> of the server's late message?
>
> * Would listening to advisory support for the temp queue at the server side
> notify me in time to stop the sending of any more messages when the temp
> queue is removed?
>
> * Or is there a better way that would stop the server from sending a
> message
> to a removed temp queue?
>
> Regards
> Zemus
>
>
> --
> View this message in context:
> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27147013.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Re: Problems with prefetch and TemporaryQueues

[Zemus <si...@tieto.com>]

Zemus wrote:
> 
> Hi,
> 
> I have a problem where TemporaryQueues are left (with 0 consumers) after
> the applications creating them have finished.
> 
> This scenario occurs for both ActiveMQ 4.1.1 and 5.3.0 on my P4 2.6 GHz, 3
> GB RAM, Ubuntu 9.10, Sun JDK, default settings for ActiveMQ.
> 
> I've tried to recreate a simpler version of the scenario. Basically I have
> a server that creates a Topic where clients that wants to subscribe may
> post a message (with a TemporaryQueue as the JMS reply) in order to be
> added. The server then has a thread that sends messages to all the
> TemporaryQueues. (I know a simple Topic could push the updates here, but
> in the real system the clients can receive information only intended for
> one single client.)
> 
> In my scenario the server is faster at generating messages than the
> clients are in processing them so the queues start to build up. Pretty
> soon (if clients are added and removed once a minute or so) it will hit a
> limit and the processing of messages in the broker seems to come to a
> halt. I think it is when the prefetch limit is reached.
> The real problem here is that if a client disconnects successfully (might
> hang in the broker communication) it is possible for the server to
> "resurrect" its TemporaryQueue, probably by sending prefetchbuffered
> messages to it. This way the TemporaryQueue is left on the broker with no
> consumer (the client disconnected) with its queued up messages counting
> towards the prefetch limit.
> 
> By running JConsole and purging the queue I can get the broker to wake up,
> but if many messages are queued up the consumerless TempQueue will queue
> those messages which might again cause the prefetch limit to be reached
> and halt the broker.
> Also, I've seen old TemporaryQueues that were removed minutes ago come
> back and start queueing buffered messages when I start purging other
> consumerless TempQueues.
> 
> 
> In short I think there are two problems here.
> First, the detection of the client disconnection is not instant on the
> server. This makes it possible to send messages to the TemporaryQueue
> without getting an exception after it should have been removed.
> 
> The second is that the TemporaryQueue is able to resurrect after its
> creator/consumer has left. In other words it has outlived its connection.
> 
> 
> I attach a simple client and server that are able to reproduce this
> reliably (at least on my computer).
> Just run the server and maybe two clients. Restart the clients perhaps 1-2
> times per minute and the issue will show pretty quickly. At least on my
> P4, maybe a computer with multiple cores will behave different.
> 
> Regards
> Zemus
> 
>  http://old.nabble.com/file/p27130529/TestClient.java TestClient.java 
>  http://old.nabble.com/file/p27130529/ClientShutdownHook.java
> ClientShutdownHook.java 
> 
>  http://old.nabble.com/file/p27130529/TestServer.java TestServer.java 
>  http://old.nabble.com/file/p27130529/ServerShutdownHook.java
> ServerShutdownHook.java 
> 


After more investigation I've changed my mind about the nature of the
problem. The prefetch limit is not involved.

My new theory goes like this:

1. The temporary queue is removed correctly when the client exists and
closes its connection.

2. The server might receive an exception when it tries to send to the
removed temporary queue. BUT sometimes it doesn't get that exception
(perhaps because the broker was soo busy that it didn't have time to notify
the activemq part of the server in time) and instead the broker creates a
new queue with the same name, although without any consumer, when the server
sends the next message to the stored destination.

3. The server knows nothing about this and keeps sending messages that pile
up in the broker, which in turn, eventually runs out of memory (at least
jconsole reports MemoryPercentageUsed 100 for the broker object).


If this sounds reasonable, my questions are instead:

* How can I prevent the broker from creating a new temporary queue because
of the server's late message?

* Would listening to advisory support for the temp queue at the server side
notify me in time to stop the sending of any more messages when the temp
queue is removed?

* Or is there a better way that would stop the server from sending a message
to a removed temp queue?

Regards
Zemus


-- 
View this message in context: http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27147013.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.