You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by kwark <pe...@gmail.com> on 2007/11/21 15:20:41 UTC
VirtualTopic and persistency problem
I have a problem with VirtualTopics and persistency.
Consider the following setup:
I have a producer which produces messages to virtualtopic:
VirtualTopic.mytopic
I have a manager with an embedded broker and two statically defined queues:
Consumer.module1.VirtualTopic.mytopic and
Consumer.module2.VirtualTopic.mytopic
I have two consumers, one for each queue defined above
Now everything works fine as expected as long as the broker stays online:
producer produces messages and even when one of the consumers goes down, it
receives all produced messages as soon as it comes back online. Cool!
Consider now the following scenario:
1) I start the manager with embedded broker, I start the consumer for
module1 and the producer.
2) I let this run for a while and then I first stop the consumer and then
the producer. At this moment consumer for module1 has received all the
produced messages, but consumer for module2 was never started and did not
receive any of the produced messages. This can also be seen from inspection
of the queues through JMX: queue Consumer.module2.VirtualTopic.mytopic still
has x messages in its queue.
3) Now I stop and restart the manager (broker). During restart I see a
Warning message that the referenceStore is not valid and AMQ is recovering.
It also logs that it has recovered x operations from redo log.
4) I also start the consumer for module2, however it never receives the
messages produced before the broker restarted.
Is this known behaviour? Did I forget to do something special?
Some more information. This was tested with both AMQ 4.1 and latest 5.0
snapshot. None of them produced the desired result. I use JMS transactions
and producer is set to produce persistent messages.
I did some debugging when the broker starts and the recovered messages from
the redo log are messages of the VirtualTopic.mytopic topic. These messages
are recovered and AMQ tries to add them to the reference store for that
topic, which is a KahaTopicReferenceStore. When I look at the implementation
of addMessageReference I see that it checks the subscriberCount of the topic
and only adds it to the store when this is > 0. If I check the
VirtualTopic.mytopic with JMX I can see that this is 0, so it never gets
added back to the topics reference store.
Can anyone provide some more insight on how to get persistency working
correctly with VirtualTopics?
--
View this message in context: http://www.nabble.com/VirtualTopic-and-persistency-problem-tf4850349s2354.html#a13877860
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: VirtualTopic and persistency problem
Posted by kwark <pe...@gmail.com>.
James,
I use the following to create the destinations when starting the broker:
public void startBroker() throws Exception {
BrokerService broker = new BrokerService();
ActiveMQQueue d1 = (ActiveMQQueue)
ActiveMQDestination.createDestination("Consumer.module1.VirtualTopic.mytopic",
ActiveMQDestination.QUEUE_TYPE);
ActiveMQQueue d2 = (ActiveMQQueue)
ActiveMQDestination.createDestination("Consumer.module2.VirtualTopic.mytopic",
ActiveMQDestination.QUEUE_TYPE);
broker.setDestinations(new ActiveMQDestination[] { d1, d2 });
broker.addConnector("tcp://localhost:61616");
broker.start();
}
I've just tried it with the static xml configuration like you suggested and
a standalone ActiveMQ broker, instead of my embedded broker (as above).
When I start the broker, I can see through JMX console that queues and the
VirtualTopic have been created. This is before I start any of the consumers
or my producer.
Unfortunately I get the similar behaviour as before and so no persistency
for VirtualTopics :-(
I get exactly the same behavior when I stop the broker by killing the
process.
When I stop the broker by Ctrl-C in the window, I don't get the "Recovered x
messages from redo log" logging.
Some more info from JMX:
When I send f.e. 20 messages from my Producer, while only consumer for
module1 is started, I see the following through JMX.
For VirtualTopic.mytopic: EnqueueCount = 20, DequeueCount = 0,
DispatchCount=0, ConsumerCount=0 and QueueSize=0
For Consumer.module1.VirtualTopic.mytopic: EnqueueCount = 20, DequeueCount =
20, DispatchCount=20, ConsumerCount=1 and QueueSize=0
For not-started Consumer.module2.VirtualTopic.mytopic EnqueueCount=20,
DequeueCount=0, DispatchCount=0, ConsumerCount=0 and QueueSize=20
It looks like only the Topic is actually persistent and not the queues. When
the broker restarts it recovers the messages for the topic, but because the
consumerCount = 0 (no durable subscriptions) it does not add the messages to
the reference store and they are effectively lost.
Any thoughts?
James.Strachan wrote:
>
> BTW are you positive that the consumer queues are actually created in
> each of the test cases you're doing? i.e. when you start/stop the
> broker are they created up front before you send any messages? If not
> you might wanna use this...
>
> http://activemq.apache.org/configure-startup-destinations.html
>
> On 21/11/2007, kwark <pe...@gmail.com> wrote:
>>
>> I have a problem with VirtualTopics and persistency.
>>
>> Consider the following setup:
>>
>> I have a producer which produces messages to virtualtopic:
>> VirtualTopic.mytopic
>> I have a manager with an embedded broker and two statically defined
>> queues:
>> Consumer.module1.VirtualTopic.mytopic and
>> Consumer.module2.VirtualTopic.mytopic
>> I have two consumers, one for each queue defined above
>>
>> Now everything works fine as expected as long as the broker stays online:
>> producer produces messages and even when one of the consumers goes down,
>> it
>> receives all produced messages as soon as it comes back online. Cool!
>>
>> Consider now the following scenario:
>> 1) I start the manager with embedded broker, I start the consumer for
>> module1 and the producer.
>> 2) I let this run for a while and then I first stop the consumer and then
>> the producer. At this moment consumer for module1 has received all the
>> produced messages, but consumer for module2 was never started and did not
>> receive any of the produced messages. This can also be seen from
>> inspection
>> of the queues through JMX: queue Consumer.module2.VirtualTopic.mytopic
>> still
>> has x messages in its queue.
>> 3) Now I stop and restart the manager (broker). During restart I see a
>> Warning message that the referenceStore is not valid and AMQ is
>> recovering.
>> It also logs that it has recovered x operations from redo log.
>> 4) I also start the consumer for module2, however it never receives the
>> messages produced before the broker restarted.
>>
>> Is this known behaviour? Did I forget to do something special?
>>
>> Some more information. This was tested with both AMQ 4.1 and latest 5.0
>> snapshot. None of them produced the desired result. I use JMS
>> transactions
>> and producer is set to produce persistent messages.
>>
>> I did some debugging when the broker starts and the recovered messages
>> from
>> the redo log are messages of the VirtualTopic.mytopic topic. These
>> messages
>> are recovered and AMQ tries to add them to the reference store for that
>> topic, which is a KahaTopicReferenceStore. When I look at the
>> implementation
>> of addMessageReference I see that it checks the subscriberCount of the
>> topic
>> and only adds it to the store when this is > 0. If I check the
>> VirtualTopic.mytopic with JMX I can see that this is 0, so it never gets
>> added back to the topics reference store.
>>
>> Can anyone provide some more insight on how to get persistency working
>> correctly with VirtualTopics?
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/VirtualTopic-and-persistency-problem-tf4850349s2354.html#a13877860
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://open.iona.com
>
>
--
View this message in context: http://www.nabble.com/VirtualTopic-and-persistency-problem-tf4850349s2354.html#a13879851
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: VirtualTopic and persistency problem
Posted by James Strachan <ja...@gmail.com>.
BTW are you positive that the consumer queues are actually created in
each of the test cases you're doing? i.e. when you start/stop the
broker are they created up front before you send any messages? If not
you might wanna use this...
http://activemq.apache.org/configure-startup-destinations.html
On 21/11/2007, kwark <pe...@gmail.com> wrote:
>
> I have a problem with VirtualTopics and persistency.
>
> Consider the following setup:
>
> I have a producer which produces messages to virtualtopic:
> VirtualTopic.mytopic
> I have a manager with an embedded broker and two statically defined queues:
> Consumer.module1.VirtualTopic.mytopic and
> Consumer.module2.VirtualTopic.mytopic
> I have two consumers, one for each queue defined above
>
> Now everything works fine as expected as long as the broker stays online:
> producer produces messages and even when one of the consumers goes down, it
> receives all produced messages as soon as it comes back online. Cool!
>
> Consider now the following scenario:
> 1) I start the manager with embedded broker, I start the consumer for
> module1 and the producer.
> 2) I let this run for a while and then I first stop the consumer and then
> the producer. At this moment consumer for module1 has received all the
> produced messages, but consumer for module2 was never started and did not
> receive any of the produced messages. This can also be seen from inspection
> of the queues through JMX: queue Consumer.module2.VirtualTopic.mytopic still
> has x messages in its queue.
> 3) Now I stop and restart the manager (broker). During restart I see a
> Warning message that the referenceStore is not valid and AMQ is recovering.
> It also logs that it has recovered x operations from redo log.
> 4) I also start the consumer for module2, however it never receives the
> messages produced before the broker restarted.
>
> Is this known behaviour? Did I forget to do something special?
>
> Some more information. This was tested with both AMQ 4.1 and latest 5.0
> snapshot. None of them produced the desired result. I use JMS transactions
> and producer is set to produce persistent messages.
>
> I did some debugging when the broker starts and the recovered messages from
> the redo log are messages of the VirtualTopic.mytopic topic. These messages
> are recovered and AMQ tries to add them to the reference store for that
> topic, which is a KahaTopicReferenceStore. When I look at the implementation
> of addMessageReference I see that it checks the subscriberCount of the topic
> and only adds it to the store when this is > 0. If I check the
> VirtualTopic.mytopic with JMX I can see that this is 0, so it never gets
> added back to the topics reference store.
>
> Can anyone provide some more insight on how to get persistency working
> correctly with VirtualTopics?
>
>
>
>
>
>
>
> --
> View this message in context: http://www.nabble.com/VirtualTopic-and-persistency-problem-tf4850349s2354.html#a13877860
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://macstrac.blogspot.com/
Open Source Integration
http://open.iona.com