You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Joe Smith <jo...@yahoo.com> on 2011/06/08 23:31:43 UTC
Not able to load-balance messages in a cluster using PrefetchPolicy
Hi,
We are using 5.4.2 running on Linux w/Java 1.6. The cluster has 3 primary brokers
and corresponding slaves (residing on separate hosts). We have 3 msg consumer processes (using Springframework w/3 concurrent
threads each), connected to each broker.
Occasionally a consumer process gets a message that has long running
time due to interaction with another external system. So a msg could take up to 20 min to process (waiting for the
other system to complete their task).
To better load balance the msgs - to prevent msgs from being
stuck behind an earlier/long-running msg, we use the
prefetchPolicy.queuePrefetch=1 configuration on the ConnectionFactory.
The prefetch works - where newer msgs get distributed to
other consumer threads that are free to work in a single consumer process. It looks like the msgs are dispatched
to the 3 consumer processes (instead of 9 consumer threads) in a round-robin
fashion in the cluster. Since we
have 3 independent consumer processes (spring/jvm), we have seen multiple msgs
get dispatched to one consumer process and waiting, while the other 2 consumer
processes connected to the other 2 brokers are idle.
Is there a correct configuration that will distribute the
msgs from one broker to another without being stuck behind a long-running
consumer thread? We can work
around the issue by increasing the number of consumer threads within each
consumer process, but thought there could be a better, more efficient solution.
We have used the following config params:
- ConnectionFactory
- prefetchPolicy
- dispatchAsync
= false
- useAsyncSend
= false
- alwaysSessionAsync
= false
- springframework DefaultMessageListenerContainer
- receiveTimeout
= 0
- broker config xml
- conduitSubscriptions="false"
- we
added ?jms.prefetchPolicy.all=1 to the NetworkConnector's uri attribute, we
also tried with just .queuePrefetch instead of .all.
- we've
tried different strict ordering dispatch policy - made no difference
Things we've noticed.
- In
springframework we can set the value for prefetchPolicy to either 1 or 0, but
the behavior seems to be the same.
However, we could not use value 0 when we use our pure Java client as
MessageListener.
- Msgs
are dispatched to the 3 brokers in a more or less round-robin fashion. We have notice occasionally that each
consumer process still gets a batch of msgs. Once it gets a batch, it does not release any queued msgs to
other brokers/consumers that are free.
There are many configuration parameters. We may have missed some critical ones.
Thanks for your help.
Attached is the springframework config. It's one file but I duplicated the
beans for ConnectionFactory, listener, containers, etc to simulate the 3
independent processes. The number
of thread are all set to one for testing purposes.
The activemq.xml is pretty much the default.
I used a Java client to send a msg: the payload is just a number. The msg listener reads the number and sleeps for that many seconds. This way I can enqueue a msg with large number followed by msgs w/smaller numbers.
A sample run shows one thread is processing multiple msgs while consumers in the other containers are idle.
Thread-2 [1] sleep 2
Thread-2 [1] wake 2
Thread-4 [1] sleep 1
Thread-3 [1] sleep 2
Thread-2 [2] sleep 3
Thread-4 [1] wake 1
Thread-4 [2] sleep 4
Thread-3 [1] wake 2
Thread-2 [2] wake 3
Thread-4 [2] wake 4
Thread-4 [3] sleep 5
Thread-4 [3] wake 5
Thread-4 [4] sleep 6
Thread-4 [4] wake 6
Thread-4 [5] sleep 7
Thread-4 [5] wake 7
Thread-4 [6] sleep 8
Thread-4 [6] wake 8
Thread-4 [7] sleep 9
Thread-4 [7] wake 9
Thread-4 [8] sleep 10
Thread-4 [8] wake 10
enter q to exit>
Thread-3 [2] sleep 10
Thread-2 [3] sleep 20
Thread-4 [9] sleep 10
Thread-3 [2] wake 10
Thread-3 [3] sleep 5
Thread-4 [9] wake 10
Thread-3 [3] wake 5
Thread-3 [4] sleep 5
Thread-3 [4] wake 5
Thread-3 [5] sleep 5
Thread-2 [3] wake 20 <-- t2 is wake and idle
Thread-3 [5] wake 5 <-- t3 process batch of msgs
Thread-3 [6] sleep 5
Thread-3 [6] wake 5
Thread-3 [7] sleep 5
Thread-3 [7] wake 5
Thread-3 [8] sleep 5
Thread-3 [8] wake 5
Thread-3 [9] sleep 5
Thread-3 [9] wake 5
Thread-3 [10] sleep 5
Thread-3 [10] wake 5
Thread-3 [11] sleep 5
Thread-3 [11] wake 5
Thread-3 [12] sleep 5
Thread-3 [12] wake 5
Thread-3 [13] sleep 5
Thread-3 [13] wake 5
Thread-3 [14] sleep 5
Thread-3 [14] wake 5
Re: Could not start connection when using jms.prefetchPolicy on failover protocol url
Posted by Joe Smith <jo...@yahoo.com>.
Hi Gary,
Thanks. I tried the decreaseNetworkConsumerPriority - still the msgs were being routed to the other 2 brokers. I think the combination of a busy local consume with a low prefetchPolicy may be forcing the messages to the other brokers. I've also tried the TTL and round-robin dispatch policy (to force an even distribution). Producer sends to virutal topic, consumers reads from Consumer.* queue. But we still see the same behavior. It could be that the ack from the remote consumer to track the actual size of the remote prefetch buffer is not in sync. Sorry - don't know enough about the code to pin it down.
Thanks, again.
________________________________
From: Gary Tully <ga...@gmail.com>
To: users@activemq.apache.org; Joe Smith <jo...@yahoo.com>
Sent: Friday, September 16, 2011 11:38 AM
Subject: Re: Could not start connection when using jms.prefetchPolicy on failover protocol url
the prefetch on a network connector does not know about the load on
the remote broker. A network connector is just another consumer, it
simply turns a message dispatch into a send to the remote broker. So
the only way to restrict the movement of messages across the network
would be producer flow control based on a memory limit on the remote
broker.
You can make network connectors have a lower priority,
decreaseNetworkConsumerPriority=true but if all local consumers are
busy, the lower priority network connector will get messages.
Re: Could not start connection when using jms.prefetchPolicy on
failover protocol url
Posted by Gary Tully <ga...@gmail.com>.
the prefetch on a network connector does not know about the load on
the remote broker. A network connector is just another consumer, it
simply turns a message dispatch into a send to the remote broker. So
the only way to restrict the movement of messages across the network
would be producer flow control based on a memory limit on the remote
broker.
You can make network connectors have a lower priority,
decreaseNetworkConsumerPriority=true but if all local consumers are
busy, the lower priority network connector will get messages.
On 16 September 2011 15:58, Joe Smith <jo...@yahoo.com> wrote:
> Hi Gary,
>
> Yes, we saw in the debug logs that remote consumer and subscription are added to the local broker. We noticed the prefetch count was 1000, instead of 1 on the consumer's uri. We added the prefetchSize to the networkConnector, but still the same result. When we use consumers that consume messages quickly - the cluster load balanced evenly. However, when we made the processing from 1 to 20 secs (via a parameter in each message), the broker where the producer is connected and when the consumer on that broker was busy already, the broker seems to directed the message to the other brokers - even if the consumers on the other brokers were busy/processing as well.
>
> Really appreciate you and all the folks who are helping. Hope some of this feedback will help.
>
> Thanks
>
>
> ________________________________
> From: Gary Tully <ga...@gmail.com>
> To: users@activemq.apache.org; Joe Smith <jo...@yahoo.com>
> Sent: Friday, September 16, 2011 5:56 AM
> Subject: Re: Could not start connection when using jms.prefetchPolicy on failover protocol url
>
> note that there is a perfetch configuration option on a networkConnector.
--
http://fusesource.com
http://blog.garytully.com
Re: Could not start connection when using jms.prefetchPolicy on failover protocol url
Posted by Joe Smith <jo...@yahoo.com>.
Hi Gary,
Yes, we saw in the debug logs that remote consumer and subscription are added to the local broker. We noticed the prefetch count was 1000, instead of 1 on the consumer's uri. We added the prefetchSize to the networkConnector, but still the same result. When we use consumers that consume messages quickly - the cluster load balanced evenly. However, when we made the processing from 1 to 20 secs (via a parameter in each message), the broker where the producer is connected and when the consumer on that broker was busy already, the broker seems to directed the message to the other brokers - even if the consumers on the other brokers were busy/processing as well.
Really appreciate you and all the folks who are helping. Hope some of this feedback will help.
Thanks
________________________________
From: Gary Tully <ga...@gmail.com>
To: users@activemq.apache.org; Joe Smith <jo...@yahoo.com>
Sent: Friday, September 16, 2011 5:56 AM
Subject: Re: Could not start connection when using jms.prefetchPolicy on failover protocol url
note that there is a perfetch configuration option on a networkConnector.
Re: Could not start connection when using jms.prefetchPolicy on
failover protocol url
Posted by Gary Tully <ga...@gmail.com>.
note that there is a perfetch configuration option on a networkConnector.
On 15 September 2011 22:48, Joe Smith <jo...@yahoo.com> wrote:
> Hi Johan,
>
>
> Thanks for the info - placing the prefetch on the failover worked. I put the prefetch on the tcp: based on the page at http://activemq.apache.org/what-is-the-prefetch-limit-for.html.
>
>
> The failover page dose not reference prefetch option: http://activemq.apache.org/failover-transport-reference.html.
> Thanks,
> again.
>
>
> Unfortunately the behavior seem to be the same when we have 3 brokers in a cluster with 1 producer and 4 consumers (each consumer connected to a separate broker - except the 4th which connected to the same broker as consumer 1). When producer sends msgs, most of the msgs got distributed to consumers to 2 of the 3 brokers.
>
>
> The msgs took different amount of time to process - even with prefetch - I see one broker processed 1 msg and sits idle, where messages were queued up on the other 3 consumers. I thought the prefetch is supposed to distribute the message to who ever is free to process (empty prefetch buffer), but the behavior was showing msgs gets queued up behind each consumer regardless - so some sit idle while another has msgs queued up.
>
>
> The distribution was round-robin 1 to each consumer - favoriting consumers on 2 brokers out of 3. In multiple runs, consumer on broker 2 always get 1msg out of 30 sent. Broker 2 happened to where the producer is connected. The pattern seems to suggest if local the consumer on the broker where the producer is connected, the broker forwards msgs to the other brokers - regardless if consumers other brokers are busy as well. Prefetch does not seem to work across cluster. I tried with both sync and async receive, I placed prefetchSize=1 on networkConnector - same behavior.
>
>
> Thanks
>
>
> ________________________________
> From: Johan Edstrom <se...@gmail.com>
> To: "users@activemq.apache.org" <us...@activemq.apache.org>
> Sent: Thursday, September 8, 2011 8:50 PM
> Subject: Re: Could not start connection when using jms.prefetchPolicy on failover protocol url
>
> You need to have the options outside the parentheses, there should be samples on the failover uri page.
>
--
http://fusesource.com
http://blog.garytully.com
Re: Could not start connection when using jms.prefetchPolicy on failover protocol url
Posted by Joe Smith <jo...@yahoo.com>.
Hi Johan,
Thanks for the info - placing the prefetch on the failover worked. I put the prefetch on the tcp: based on the page at http://activemq.apache.org/what-is-the-prefetch-limit-for.html.
The failover page dose not reference prefetch option: http://activemq.apache.org/failover-transport-reference.html.
Thanks,
again.
Unfortunately the behavior seem to be the same when we have 3 brokers in a cluster with 1 producer and 4 consumers (each consumer connected to a separate broker - except the 4th which connected to the same broker as consumer 1). When producer sends msgs, most of the msgs got distributed to consumers to 2 of the 3 brokers.
The msgs took different amount of time to process - even with prefetch - I see one broker processed 1 msg and sits idle, where messages were queued up on the other 3 consumers. I thought the prefetch is supposed to distribute the message to who ever is free to process (empty prefetch buffer), but the behavior was showing msgs gets queued up behind each consumer regardless - so some sit idle while another has msgs queued up.
The distribution was round-robin 1 to each consumer - favoriting consumers on 2 brokers out of 3. In multiple runs, consumer on broker 2 always get 1msg out of 30 sent. Broker 2 happened to where the producer is connected. The pattern seems to suggest if local the consumer on the broker where the producer is connected, the broker forwards msgs to the other brokers - regardless if consumers other brokers are busy as well. Prefetch does not seem to work across cluster. I tried with both sync and async receive, I placed prefetchSize=1 on networkConnector - same behavior.
Thanks
________________________________
From: Johan Edstrom <se...@gmail.com>
To: "users@activemq.apache.org" <us...@activemq.apache.org>
Sent: Thursday, September 8, 2011 8:50 PM
Subject: Re: Could not start connection when using jms.prefetchPolicy on failover protocol url
You need to have the options outside the parentheses, there should be samples on the failover uri page.
Re: Could not start connection when using jms.prefetchPolicy on failover protocol url
Posted by Johan Edstrom <se...@gmail.com>.
You need to have the options outside the parentheses, there should be samples on the failover uri page.
On Sep 8, 2011, at 17:58, joe smith <wa...@yahoo.com> wrote:
> Could not start connection when using the following URL pattern:
>
> failover:(tcp://localhost:61616?jms.prefetchPolicy.queuePrefetch=1,tcp://localhost:62616?jms.prefetchPolicy.queuePrefetch=1)
>
> When prefetchPolicy is removed, connection started normally.
>
> Broker version: 5.4.2
> Java client library
> JVM 1.6
>
> We do not access to the client code and can only pass in configs via url.
>
> Thanks for any help.
Could not start connection when using jms.prefetchPolicy on failover protocol url
Posted by joe smith <wa...@yahoo.com>.
Could not start connection when using the following URL pattern:
failover:(tcp://localhost:61616?jms.prefetchPolicy.queuePrefetch=1,tcp://localhost:62616?jms.prefetchPolicy.queuePrefetch=1)
When prefetchPolicy is removed, connection started normally.
Broker version: 5.4.2
Java client library
JVM 1.6
We do not access to the client code and can only pass in configs via url.
Thanks for any help.
Re: Not able to load-balance messages in a cluster using PrefetchPolicy
Posted by joe smith <wa...@yahoo.com>.
Hi Martin,
We use springframework for the listener and the listener container - we did not set the listener/container to be exclusive - not sure if you can do so in spring. However, I don't know if that is a default setting in spring - will look into it.
We don't use message group either - so any consumer listening on the queue can take any msg in any order.
Thanks for the suggestion.
--- On Thu, 6/9/11, Martin C. <ma...@gmx.at> wrote:
From: Martin C. <ma...@gmx.at>
Subject: Re: Not able to load-balance messages in a cluster using PrefetchPolicy
To: users@activemq.apache.org
Date: Thursday, June 9, 2011, 11:41 AM
Hi,
just as further input: have you checked you didn't declare an
exclusive consumer and you didn't use message groups?
Best regards,
Martin
Re: Not able to load-balance messages in a cluster using PrefetchPolicy
Posted by "Martin C." <ma...@gmx.at>.
Hi,
just as further input: have you checked you didn't declare an
exclusive consumer and you didn't use message groups?
Best regards,
Martin
On Thu, Jun 9, 2011 at 4:38 PM, joe smith <wa...@yahoo.com> wrote:
> Hi Edan,
>
> Yes. with the springframework's PrefetchPolicy bean, I've used both values. Unfortunately, the msgs were still queued within a listener container bounded a single broker. The msgs were not re-distributed to the other 2 listen containers (which were idle) bounded to the other 2 brokers.
>
> Thanks
>
>
> --- On Wed, 6/8/11, Idzerda, Edan <Ed...@PremierInc.com> wrote:
>
> From: Idzerda, Edan <Ed...@PremierInc.com>
> Subject: RE: Not able to load-balance messages in a cluster using PrefetchPolicy
> To: "users@activemq.apache.org" <us...@activemq.apache.org>
> Date: Wednesday, June 8, 2011, 5:49 PM
>
> Have you tried prefetch=0 ? I was doing some similar testing a while ago and as I recall, prefetch=0 was the setting that behaved best for my round-robin consumer setup.
>
>
RE: Not able to load-balance messages in a cluster using PrefetchPolicy
Posted by joe smith <wa...@yahoo.com>.
Hi Edan,
Yes. with the springframework's PrefetchPolicy bean, I've used both values. Unfortunately, the msgs were still queued within a listener container bounded a single broker. The msgs were not re-distributed to the other 2 listen containers (which were idle) bounded to the other 2 brokers.
Thanks
--- On Wed, 6/8/11, Idzerda, Edan <Ed...@PremierInc.com> wrote:
From: Idzerda, Edan <Ed...@PremierInc.com>
Subject: RE: Not able to load-balance messages in a cluster using PrefetchPolicy
To: "users@activemq.apache.org" <us...@activemq.apache.org>
Date: Wednesday, June 8, 2011, 5:49 PM
Have you tried prefetch=0 ? I was doing some similar testing a while ago and as I recall, prefetch=0 was the setting that behaved best for my round-robin consumer setup.
RE: Not able to load-balance messages in a cluster using
PrefetchPolicy
Posted by "Idzerda, Edan" <Ed...@PremierInc.com>.
Have you tried prefetch=0 ? I was doing some similar testing a while ago and as I recall, prefetch=0 was the setting that behaved best for my round-robin consumer setup.
From: Joe Smith [mailto:joesmithcomm@yahoo.com]
Sent: Wednesday, June 08, 2011 5:32 PM
To: users@activemq.apache.org
Subject: Not able to load-balance messages in a cluster using PrefetchPolicy
Hi,
We are using 5.4.2 running on Linux w/Java 1.6. The cluster has 3 primary brokers and corresponding slaves (residing on separate hosts). We have 3 msg consumer processes (using Springframework w/3 concurrent threads each), connected to each broker. Occasionally a consumer process gets a message that has long running time due to interaction with another external system. So a msg could take up to 20 min to process (waiting for the other system to complete their task).
To better load balance the msgs - to prevent msgs from being stuck behind an earlier/long-running msg, we use the prefetchPolicy.queuePrefetch=1 configuration on the ConnectionFactory.
The prefetch works - where newer msgs get distributed to other consumer threads that are free to work in a single consumer process. It looks like the msgs are dispatched to the 3 consumer processes (instead of 9 consumer threads) in a round-robin fashion in the cluster. Since we have 3 independent consumer processes (spring/jvm), we have seen multiple msgs get dispatched to one consumer process and waiting, while the other 2 consumer processes connected to the other 2 brokers are idle.
Is there a correct configuration that will distribute the msgs from one broker to another without being stuck behind a long-running consumer thread? We can work around the issue by increasing the number of consumer threads within each consumer process, but thought there could be a better, more efficient solution.
We have used the following config params:
- ConnectionFactory
- prefetchPolicy
- dispatchAsync = false
- useAsyncSend = false
- alwaysSessionAsync = false
- springframework DefaultMessageListenerContainer
- receiveTimeout = 0
- broker config xml
- conduitSubscriptions="false"
- we added ?jms.prefetchPolicy.all=1 to the NetworkConnector's uri attribute, we also tried with just .queuePrefetch instead of .all.
- we've tried different strict ordering dispatch policy - made no difference
Things we've noticed.
- In springframework we can set the value for prefetchPolicy to either 1 or 0, but the behavior seems to be the same. However, we could not use value 0 when we use our pure Java client as MessageListener.
- Msgs are dispatched to the 3 brokers in a more or less round-robin fashion. We have notice occasionally that each consumer process still gets a batch of msgs. Once it gets a batch, it does not release any queued msgs to other brokers/consumers that are free.
There are many configuration parameters. We may have missed some critical ones.
Thanks for your help.
Attached is the springframework config. It's one file but I duplicated the beans for ConnectionFactory, listener, containers, etc to simulate the 3 independent processes. The number of thread are all set to one for testing purposes.
The activemq.xml is pretty much the default.
-----------------------------------------
***Note:The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader of
this message is not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient,
you are hereby notified that any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify the Sender
immediately by replying to the message and deleting it from your
computer. Thank you. Premier Inc.