Problems with Virtual Topics

[Shyam Santhanam <ss...@vmware.com>]

Hello ActiveMQ experts,

I'm facing some difficulty using virtual topics under a network of brokers
configuration.

Background: We have a network connector that uses a custom DiscoveryAgent to
discover other brokers in the network and setup the network connector. Every
node has a single broker.

We have one or more virtual topics e.g VirtualTopic.X, each with a consumer
queue (say Consumer.A.VirtualTopic.X), and one consumer for this queue per
network node (broker).  We send persistent messages to the VirtualTopic. We
use KahaDB as the persistence store.

The consumers use a transacted session in AUTO_ACKNOWLEDGE mode. We perform
an explicit commit/rollback depending on whether our messages were
successfully handled or some transient condition prevents them from being
handled.

This all seems to work under a single broker/node. When we test this in a
real network of brokers, we start seeing a number of problems:

1. Under some circumstances, every consumer on every node receives the same
message, even though we are consuming from a queue. The distributed queue
behaves as though it is really a set of independent queues.

2. We see messages being retained indefinitely sometimes and consumers
getting delivered already consumed, older messages on nodes when they are
restarted.

Are we missing some configuration here? Are there any conditions under which
what I just described would happen, and how do we prevent it? I've copied
our broker configuration below. The "disc" protocol in the network connector
URI pertains to our custom discovery agent.

Appreciate any insight you can provide.

Thanks,
Shyam

   <broker
        xmlns="http://activemq.apache.org/schema/core"
        id="activeMQBroker"
        brokerName="${broker.name}"
        dataDirectory="${broker.data.directory}"
        useJmx="true"
        useShutdownHook="true"
        keepDurableSubsActive="false">

        <destinationPolicy>
            <policyMap>
                <policyEntries>
                    <policyEntry queue=">" memoryLimit="5mb"/>
                    <policyEntry topic=">" memoryLimit="5mb">
                        <subscriptionRecoveryPolicy>
                            <lastImageSubscriptionRecoveryPolicy/>
                        </subscriptionRecoveryPolicy>
                    </policyEntry>
                </policyEntries>
            </policyMap>

        </destinationPolicy>

        <managementContext>
            <managementContext createConnector="false" />
        </managementContext>

        <networkConnectors>
            <networkConnector name="default-nc"
uri="disc://(tcp://${broker.name}:${jms.openwire.port})">
                <excludedDestinations>
                    <queue physicalName="Consumer.*.VirtualTopic.>" />
                </excludedDestinations>
            </networkConnector>
        </networkConnectors>

        <persistenceAdapter>
            <KahaDB
                directory="..."
                journalMaxFileLength="32mb"/>
        </persistenceAdapter>

        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="${jms.memoryUsage}" />
                </memoryUsage>

                <storeUsage>
                    <storeUsage limit="${jms.diskStoreUsage}"
name="broker.storage.${broker.name}" />
                </storeUsage>

                <tempUsage>
                    <tempUsage limit="${jms.tempUsage}" />
                </tempUsage>
            </systemUsage>
        </systemUsage>

        <transportConnectors>
            <transportConnector
                name="openwire"
                uri="tcp://${ip.primary}:${jms.openwire.port}"
               
discoveryUri="disc://(tcp://${broker.name}:${jms.openwire.port})" />
            <transportConnector
                name="stomp"
                uri="stomp://${ip.primary}:${jms.stomp.port}" />
        </transportConnectors>
    </broker>

-- 
View this message in context: http://old.nabble.com/Problems-with-Virtual-Topics-tp28100311p28100311.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problems with Virtual Topics

[Shyam Santhanam <ss...@vmware.com>]

James - great thanks for your replies, they were very helpful.


James Casey-2 wrote:
> 
> use suppressDuplicateQueueSubscriptions="true" on the network connectors I
> normally only see one additional consumer - what is your topology ?
> 

Thanks, I'm definitely gonna try this out. We have a fully-connected
topology, i.e every broker is connected to every other. 


James Casey-2 wrote:
> 
> Could it be this : <https://issues.apache.org/activemq/browse/AMQ-2586> ?
> 

Yes, perhaps it's just the statistic that is off. Makes more sense now.
Thank you.


James Casey-2 wrote:
> 
>> When I try to browse the contents of
>> the topic, I often find that there are some residual messages (usually
>> far
>> fewer messages than indicated by the QueueSize statistic, but > 0
>> nonetheless). Again, this doesn't gel with my understanding. I thought
>> that
>> the topics themselves are "virtual" which should imply that they don't
>> have
>> a persistent cache of messages anywhere? Are the virtual topics
>> themselves
>> also persisted? If so, when do messages in the topic get evacuated?
>>
>> You can  use the topic as a normal topic - perhaps this is the effect you
> see ?  when you subscribe then any  messages received will be sent to your
> topic consumer.
> 

Yes this would explain it. Since we never consume from these topics
directly, how do we prevent messages from backing up indefinitely in these
topics? We'd prefer not to have a TTL set on our messages, since this is
very critical non-transient data.

-- 
View this message in context: http://old.nabble.com/Problems-with-Virtual-Topics-tp28100311p28111206.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problems with Virtual Topics

[James Casey <ja...@gmail.com>]

On 1 April 2010 16:04, Shyam Santhanam <ss...@vmware.com> wrote:

>
> There are a few more observations I've made while trying to debug this.
> We've
> noticed that sometimes the messages received from the consumer queues have
> JMSDestination set to the VirtualTopic name, and at other times they have
> the consumer queue name. As shown earlier, I do have the
> excludedDestinations clause necessary to exclude forwarding to the queues.
> Is this still to be expected?
>
>
Hi shyam,

we've seen this too - it depends on where the messages were sent to.

Assume you have 2 brokers A and B.

Connect your virtual topic consumer to A.

Now if a message is sent to A, you get the topic name as destination.

If a message is sent to B, you get the queue name as destination.


> Also, looking in JMX at the VirtualTopic's topic MBeans, I've noticed that
> the topics themselves have consumers in addition to the queue. Are these
> consumers created internally by the broker to enable forwarding, or could
> this indicate a problem? Further, there are far many more consumers for the
> topics. For example, when testing with 2 nodes: Although I have only one
> consumer queue attached to the topic, and one consumer  to this topic on
> each node, I notice that the VirtualTopic itself shows 6 consumers. Where
> are these consumers coming from?
>
> Normally  I see some extra consumers from the network of brokers.  since we
use suppressDuplicateQueueSubscriptions="true" on the network connectors I
normally only see one additional consumer - what is your topology ?


Another observation when looking at the MBeans: the QueueSize of the topics
> appears to increase steadily over time.


Could it be this : <https://issues.apache.org/activemq/browse/AMQ-2586> ?


> When I try to browse the contents of
> the topic, I often find that there are some residual messages (usually far
> fewer messages than indicated by the QueueSize statistic, but > 0
> nonetheless). Again, this doesn't gel with my understanding. I thought that
> the topics themselves are "virtual" which should imply that they don't have
> a persistent cache of messages anywhere? Are the virtual topics themselves
> also persisted? If so, when do messages in the topic get evacuated?
>
> You can  use the topic as a normal topic - perhaps this is the effect you
see ?  when you subscribe then any  messages received will be sent to your
topic consumer.


> If someone familiar with this area could answer these questions, it could
> really help us make some important decisions about how, and whether, to use
> virtual topics to implement our requirements. Has someone used them
> effectively in a network-of-brokers scenario? If so, can you share the
> specifics of your configuration?
>
>
> Well we're in the middle of moving from durable consumer to virtual topics,
so I wouldn't say we've
got a final config yet.  We're basically using the config as mentioned on
the apache web pages.

James.

--
> View this message in context:
> http://old.nabble.com/Problems-with-Virtual-Topics-tp28100311p28107798.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

Re: Problems with Virtual Topics

[Shyam Santhanam <ss...@vmware.com>]

There are a few more observations I've made while trying to debug this. We've
noticed that sometimes the messages received from the consumer queues have
JMSDestination set to the VirtualTopic name, and at other times they have
the consumer queue name. As shown earlier, I do have the
excludedDestinations clause necessary to exclude forwarding to the queues.
Is this still to be expected?

Also, looking in JMX at the VirtualTopic's topic MBeans, I've noticed that
the topics themselves have consumers in addition to the queue. Are these
consumers created internally by the broker to enable forwarding, or could
this indicate a problem? Further, there are far many more consumers for the
topics. For example, when testing with 2 nodes: Although I have only one
consumer queue attached to the topic, and one consumer  to this topic on
each node, I notice that the VirtualTopic itself shows 6 consumers. Where
are these consumers coming from?

Another observation when looking at the MBeans: the QueueSize of the topics
appears to increase steadily over time. When I try to browse the contents of
the topic, I often find that there are some residual messages (usually far
fewer messages than indicated by the QueueSize statistic, but > 0
nonetheless). Again, this doesn't gel with my understanding. I thought that
the topics themselves are "virtual" which should imply that they don't have
a persistent cache of messages anywhere? Are the virtual topics themselves
also persisted? If so, when do messages in the topic get evacuated?

If someone familiar with this area could answer these questions, it could
really help us make some important decisions about how, and whether, to use
virtual topics to implement our requirements. Has someone used them
effectively in a network-of-brokers scenario? If so, can you share the
specifics of your configuration?


-- 
View this message in context: http://old.nabble.com/Problems-with-Virtual-Topics-tp28100311p28107798.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.