You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Eric Van Dewoestine <er...@gmail.com> on 2009/10/07 19:09:58 UTC
Issue creating a distributed queue using store and forward.
ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
I'm running into an issue with the store and forward feature of
activemq, which I using in an attempt to create a highly available
distributed queue. I'm trying to figure out if the issue is a
misconfiguration on my part, expected behavior of activemq, or a bug.
The summary of the problem is that given 2 brokers, B1 and B2, which
each have one consumer, C1 and C2, which are subscribed to the same
queue. If I stop a consumer on one of the brokers, the pending
messages from that broker are not always forwarded to the other broker
which still has a consumer, leading to those messages getting
indefinitely stuck.
The steps I use to reproduce this scenario are as follows (Note: all
producing and consuming is performed over the stomp transport):
Couple notes about the consumers:
- they have a prefetchSize of 40
- the processing of messages can take some time, so for the purposes
of this exercise, I've created a simple consumer that sleeps for 10
seconds before sending the message ack (using client-individual ack
mode)
1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
yet running.
2. produce a few thousand messages to B1
Note: B1 now has a few thousand pending messages and B2 has 0.
3. start consumer C2 (listing for messages from B2)
Note: messages are are successfully received and begin processing
(monitoring the brokers shows pending messages decreasing). Now B2
has all the pending messages and B1 has 0.
4. start consumer C1 (listing for messages from B1)
Note: no messages are received, which is another issue I have since
B2 now has thousands of pending messages which C1 could help
process, but instead sits idle while C2 is forced to handle all the
messages.
5. stop consumer C2
Note: now I have thousands of messages sitting on B2 and 0 on B1
where a C1 is alive and ready to handle them. So at this point,
despite having a consumer running, thousands of messages are stuck
in the queue.
6. stop consumer C1
Note: now I have no consumers. Stopping and restarting C1 has no
effect on the pending messages sitting on B1's queue.
7. stop both brokers
8. start B1, then start B2
9. start C1
Note: now all messages have migrated from B2 to B1 and C1 is again
processing messages.
So after step 5, the only way to recover from the stuck messages is to
restart the brokers.
Below is my current connector config which I have on both brokers.
I've tried playing with the various properties of the connector, but
it seems as though no matter what I try the above scenario continues
to occur.
<networkConnector
name="default-nc"
uri="multicast://default?group=${broker.group}"
dynamicOnly="true"
networkTTL="25"
suppressDuplicateQueueSubscriptions="true"/>
So, is this an activemq bug? Am I mis-using activemq? Is there some
other way to achieve a highly available distributed queue?
Any help in this regard is greatly appreciated.
--
eric
Re: Issue creating a distributed queue using store and forward.
Posted by Eric Van Dewoestine <er...@gmail.com>.
Ah, that attribute isn't documented on the activemq site ;)
I tried that one out, but unfortunately I don't see any change in
behavior. I verified that the prefetchSize is indeed set to 1 by
examining the network connector attributes for both brokers via jmx.
Thank you so much for your help so far, I really do appreciate it.
I'm going to start digging through the code to see if I can determine
why these messages refuse to budge. I'll probably start by seeing if
Joe's assumption about messages not visiting a broker more than once
is the cause.
If you or anyone else has any other tips or suggestions, I'd love to
hear them.
On Thu, Oct 08, 2009 at 05:42:48PM +0100, Rob Davies wrote:
> Hi Eric,
>
> sorry for being so quick in my reply - in the broker xml config set
> the prefetchSize property on the network - e.g.
>
> <networkConnectors>
> <networkConnector
> uri="static://(tcp://localhost:61617)"
> name="Connection to 61617"
> dynamicOnly="true"
> prefetchSize="1"/>
>
> </networkConnectors>
> On 8 Oct 2009, at 17:16, Eric Van Dewoestine wrote:
>
> > I guess I'm not sure what setting you are referring to. I tried
> > setting a couple different prefetch policies on the tcp connector like
> > so:
> >
> > 1st attempt:
> > <transportConnector
> > uri="tcp://${broker.host}:0?jms.prefetchPolicy.queuePrefetch=1"
> > discoveryUri="multicast://default?group=${broker.group}"/>
> >
> > 2nd attempt:
> > <transportConnector
> > uri="tcp://${broker.host}:0?jms.prefetchPolicy.all=1"
> > discoveryUri="multicast://default?group=${broker.group}"/>
> >
> > But neither of them seemed to change the behavior. Are you referring
> > to another setting? Can you provide an example of the setting you are
> > referring to?
> >
> > On Thu, Oct 08, 2009 at 04:22:11PM +0100, Rob Davies wrote:
> >> do you set the prefetchSize property on the network itself ? Setting
> >> it on the consumers wouldn't make any difference
> >> On 8 Oct 2009, at 15:39, Eric Van Dewoestine wrote:
> >>
> >>> I tried setting the prefetchSize to 1 for both consumers, but
> >>> unfortunately, that didn't resolve the issue. Messages still end up
> >>> stuck on B2 when C2 is stopped.
> >>>
> >>> On Thu, Oct 08, 2009 at 09:33:06AM +0100, Rob Davies wrote:
> >>>> set the prefetchSize=1 on your network - to avoid orphaned messages
> >>>> on
> >>>> B2
> >>>>
> >>>> On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:
> >>>>
> >>>>> Hmm, that seems like a flawed approach if the broker it visited
> >>>>> has
> >>>>> changed state and is now eligible to process those messages.
> >>>>> How is
> >>>>> this type of fail over intended to be handled with activemq? Is
> >>>>> there
> >>>>> an alternate solution that I'm missing?
> >>>>>
> >>>>> On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
> >>>>>>
> >>>>>> I think that by design a message will not be forwarded to a
> >>>>>> broker
> >>>>>> that it
> >>>>>> has already visited.
> >>>>>>
> >>>>>> Joe
> >>>>>> http://www.ttmsolutions.com
> >>>>>>
> >>>>>>
> >>>>>> Eric Van wrote:
> >>>>>>>
> >>>>>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots
> >>>>>>> listing)
> >>>>>>>
> >>>>>>> I'm running into an issue with the store and forward feature of
> >>>>>>> activemq, which I using in an attempt to create a highly
> >>>>>>> available
> >>>>>>> distributed queue. I'm trying to figure out if the issue is a
> >>>>>>> misconfiguration on my part, expected behavior of activemq, or a
> >>>>>>> bug.
> >>>>>>>
> >>>>>>> The summary of the problem is that given 2 brokers, B1 and B2,
> >>>>>>> which
> >>>>>>> each have one consumer, C1 and C2, which are subscribed to the
> >>>>>>> same
> >>>>>>> queue. If I stop a consumer on one of the brokers, the pending
> >>>>>>> messages from that broker are not always forwarded to the other
> >>>>>>> broker
> >>>>>>> which still has a consumer, leading to those messages getting
> >>>>>>> indefinitely stuck.
> >>>>>>>
> >>>>>>> The steps I use to reproduce this scenario are as follows (Note:
> >>>>>>> all
> >>>>>>> producing and consuming is performed over the stomp transport):
> >>>>>>>
> >>>>>>> Couple notes about the consumers:
> >>>>>>> - they have a prefetchSize of 40
> >>>>>>> - the processing of messages can take some time, so for the
> >>>>>>> purposes
> >>>>>>> of this exercise, I've created a simple consumer that sleeps for
> >>>>>>> 10
> >>>>>>> seconds before sending the message ack (using client-individual
> >>>>>>> ack
> >>>>>>> mode)
> >>>>>>>
> >>>>>>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are
> >>>>>>> not
> >>>>>>> yet running.
> >>>>>>> 2. produce a few thousand messages to B1
> >>>>>>> Note: B1 now has a few thousand pending messages and B2 has 0.
> >>>>>>> 3. start consumer C2 (listing for messages from B2)
> >>>>>>> Note: messages are are successfully received and begin
> >>>>>>> processing
> >>>>>>> (monitoring the brokers shows pending messages decreasing). Now
> >>>>>>> B2
> >>>>>>> has all the pending messages and B1 has 0.
> >>>>>>> 4. start consumer C1 (listing for messages from B1)
> >>>>>>> Note: no messages are received, which is another issue I have
> >>>>>>> since
> >>>>>>> B2 now has thousands of pending messages which C1 could help
> >>>>>>> process, but instead sits idle while C2 is forced to handle all
> >>>>>>> the
> >>>>>>> messages.
> >>>>>>> 5. stop consumer C2
> >>>>>>> Note: now I have thousands of messages sitting on B2 and 0 on B1
> >>>>>>> where a C1 is alive and ready to handle them. So at this point,
> >>>>>>> despite having a consumer running, thousands of messages are
> >>>>>>> stuck
> >>>>>>> in the queue.
> >>>>>>> 6. stop consumer C1
> >>>>>>> Note: now I have no consumers. Stopping and restarting C1 has
> >>>>>>> no
> >>>>>>> effect on the pending messages sitting on B1's queue.
> >>>>>>> 7. stop both brokers
> >>>>>>> 8. start B1, then start B2
> >>>>>>> 9. start C1
> >>>>>>> Note: now all messages have migrated from B2 to B1 and C1 is
> >>>>>>> again
> >>>>>>> processing messages.
> >>>>>>>
> >>>>>>> So after step 5, the only way to recover from the stuck messages
> >>>>>>> is to
> >>>>>>> restart the brokers.
> >>>>>>>
> >>>>>>> Below is my current connector config which I have on both
> >>>>>>> brokers.
> >>>>>>> I've tried playing with the various properties of the connector,
> >>>>>>> but
> >>>>>>> it seems as though no matter what I try the above scenario
> >>>>>>> continues
> >>>>>>> to occur.
> >>>>>>>
> >>>>>>> <networkConnector
> >>>>>>> name="default-nc"
> >>>>>>> uri="multicast://default?group=${broker.group}"
> >>>>>>> dynamicOnly="true"
> >>>>>>> networkTTL="25"
> >>>>>>> suppressDuplicateQueueSubscriptions="true"/>
> >>>>>>>
> >>>>>>>
> >>>>>>> So, is this an activemq bug? Am I mis-using activemq? Is there
> >>>>>>> some
> >>>>>>> other way to achieve a highly available distributed queue?
> >>>>>>>
> >>>>>>> Any help in this regard is greatly appreciated.
> >>>>>>>
> >>>>>>> --
> >>>>>>> eric
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>> --
> >>>>> eric
> >>>>
> >>>> Rob Davies
> >>>> http://twitter.com/rajdavies
> >>>> I work here: http://fusesource.com
> >>>> My Blog: http://rajdavies.blogspot.com/
> >>>> I'm writing this: http://www.manning.com/snyder/
> >>>
> >>> --
> >>> eric
> >>
> >> Rob Davies
> >> http://twitter.com/rajdavies
> >> I work here: http://fusesource.com
> >> My Blog: http://rajdavies.blogspot.com/
> >> I'm writing this: http://www.manning.com/snyder/
> >>
> >>
> >>
> >>
> >>
> >
> > --
> > eric
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
--
eric
Re: Issue creating a distributed queue using store and forward.
Posted by Rob Davies <ra...@gmail.com>.
Hi Eric,
sorry for being so quick in my reply - in the broker xml config set
the prefetchSize property on the network - e.g.
<networkConnectors>
<networkConnector
uri="static://(tcp://localhost:61617)"
name="Connection to 61617"
dynamicOnly="true"
prefetchSize="1"/>
</networkConnectors>
On 8 Oct 2009, at 17:16, Eric Van Dewoestine wrote:
> I guess I'm not sure what setting you are referring to. I tried
> setting a couple different prefetch policies on the tcp connector like
> so:
>
> 1st attempt:
> <transportConnector
> uri="tcp://${broker.host}:0?jms.prefetchPolicy.queuePrefetch=1"
> discoveryUri="multicast://default?group=${broker.group}"/>
>
> 2nd attempt:
> <transportConnector
> uri="tcp://${broker.host}:0?jms.prefetchPolicy.all=1"
> discoveryUri="multicast://default?group=${broker.group}"/>
>
> But neither of them seemed to change the behavior. Are you referring
> to another setting? Can you provide an example of the setting you are
> referring to?
>
> On Thu, Oct 08, 2009 at 04:22:11PM +0100, Rob Davies wrote:
>> do you set the prefetchSize property on the network itself ? Setting
>> it on the consumers wouldn't make any difference
>> On 8 Oct 2009, at 15:39, Eric Van Dewoestine wrote:
>>
>>> I tried setting the prefetchSize to 1 for both consumers, but
>>> unfortunately, that didn't resolve the issue. Messages still end up
>>> stuck on B2 when C2 is stopped.
>>>
>>> On Thu, Oct 08, 2009 at 09:33:06AM +0100, Rob Davies wrote:
>>>> set the prefetchSize=1 on your network - to avoid orphaned messages
>>>> on
>>>> B2
>>>>
>>>> On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:
>>>>
>>>>> Hmm, that seems like a flawed approach if the broker it visited
>>>>> has
>>>>> changed state and is now eligible to process those messages.
>>>>> How is
>>>>> this type of fail over intended to be handled with activemq? Is
>>>>> there
>>>>> an alternate solution that I'm missing?
>>>>>
>>>>> On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
>>>>>>
>>>>>> I think that by design a message will not be forwarded to a
>>>>>> broker
>>>>>> that it
>>>>>> has already visited.
>>>>>>
>>>>>> Joe
>>>>>> http://www.ttmsolutions.com
>>>>>>
>>>>>>
>>>>>> Eric Van wrote:
>>>>>>>
>>>>>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots
>>>>>>> listing)
>>>>>>>
>>>>>>> I'm running into an issue with the store and forward feature of
>>>>>>> activemq, which I using in an attempt to create a highly
>>>>>>> available
>>>>>>> distributed queue. I'm trying to figure out if the issue is a
>>>>>>> misconfiguration on my part, expected behavior of activemq, or a
>>>>>>> bug.
>>>>>>>
>>>>>>> The summary of the problem is that given 2 brokers, B1 and B2,
>>>>>>> which
>>>>>>> each have one consumer, C1 and C2, which are subscribed to the
>>>>>>> same
>>>>>>> queue. If I stop a consumer on one of the brokers, the pending
>>>>>>> messages from that broker are not always forwarded to the other
>>>>>>> broker
>>>>>>> which still has a consumer, leading to those messages getting
>>>>>>> indefinitely stuck.
>>>>>>>
>>>>>>> The steps I use to reproduce this scenario are as follows (Note:
>>>>>>> all
>>>>>>> producing and consuming is performed over the stomp transport):
>>>>>>>
>>>>>>> Couple notes about the consumers:
>>>>>>> - they have a prefetchSize of 40
>>>>>>> - the processing of messages can take some time, so for the
>>>>>>> purposes
>>>>>>> of this exercise, I've created a simple consumer that sleeps for
>>>>>>> 10
>>>>>>> seconds before sending the message ack (using client-individual
>>>>>>> ack
>>>>>>> mode)
>>>>>>>
>>>>>>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are
>>>>>>> not
>>>>>>> yet running.
>>>>>>> 2. produce a few thousand messages to B1
>>>>>>> Note: B1 now has a few thousand pending messages and B2 has 0.
>>>>>>> 3. start consumer C2 (listing for messages from B2)
>>>>>>> Note: messages are are successfully received and begin
>>>>>>> processing
>>>>>>> (monitoring the brokers shows pending messages decreasing). Now
>>>>>>> B2
>>>>>>> has all the pending messages and B1 has 0.
>>>>>>> 4. start consumer C1 (listing for messages from B1)
>>>>>>> Note: no messages are received, which is another issue I have
>>>>>>> since
>>>>>>> B2 now has thousands of pending messages which C1 could help
>>>>>>> process, but instead sits idle while C2 is forced to handle all
>>>>>>> the
>>>>>>> messages.
>>>>>>> 5. stop consumer C2
>>>>>>> Note: now I have thousands of messages sitting on B2 and 0 on B1
>>>>>>> where a C1 is alive and ready to handle them. So at this point,
>>>>>>> despite having a consumer running, thousands of messages are
>>>>>>> stuck
>>>>>>> in the queue.
>>>>>>> 6. stop consumer C1
>>>>>>> Note: now I have no consumers. Stopping and restarting C1 has
>>>>>>> no
>>>>>>> effect on the pending messages sitting on B1's queue.
>>>>>>> 7. stop both brokers
>>>>>>> 8. start B1, then start B2
>>>>>>> 9. start C1
>>>>>>> Note: now all messages have migrated from B2 to B1 and C1 is
>>>>>>> again
>>>>>>> processing messages.
>>>>>>>
>>>>>>> So after step 5, the only way to recover from the stuck messages
>>>>>>> is to
>>>>>>> restart the brokers.
>>>>>>>
>>>>>>> Below is my current connector config which I have on both
>>>>>>> brokers.
>>>>>>> I've tried playing with the various properties of the connector,
>>>>>>> but
>>>>>>> it seems as though no matter what I try the above scenario
>>>>>>> continues
>>>>>>> to occur.
>>>>>>>
>>>>>>> <networkConnector
>>>>>>> name="default-nc"
>>>>>>> uri="multicast://default?group=${broker.group}"
>>>>>>> dynamicOnly="true"
>>>>>>> networkTTL="25"
>>>>>>> suppressDuplicateQueueSubscriptions="true"/>
>>>>>>>
>>>>>>>
>>>>>>> So, is this an activemq bug? Am I mis-using activemq? Is there
>>>>>>> some
>>>>>>> other way to achieve a highly available distributed queue?
>>>>>>>
>>>>>>> Any help in this regard is greatly appreciated.
>>>>>>>
>>>>>>> --
>>>>>>> eric
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> eric
>>>>
>>>> Rob Davies
>>>> http://twitter.com/rajdavies
>>>> I work here: http://fusesource.com
>>>> My Blog: http://rajdavies.blogspot.com/
>>>> I'm writing this: http://www.manning.com/snyder/
>>>
>>> --
>>> eric
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>
> --
> eric
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: Issue creating a distributed queue using store and forward.
Posted by Eric Van Dewoestine <er...@gmail.com>.
I guess I'm not sure what setting you are referring to. I tried
setting a couple different prefetch policies on the tcp connector like
so:
1st attempt:
<transportConnector
uri="tcp://${broker.host}:0?jms.prefetchPolicy.queuePrefetch=1"
discoveryUri="multicast://default?group=${broker.group}"/>
2nd attempt:
<transportConnector
uri="tcp://${broker.host}:0?jms.prefetchPolicy.all=1"
discoveryUri="multicast://default?group=${broker.group}"/>
But neither of them seemed to change the behavior. Are you referring
to another setting? Can you provide an example of the setting you are
referring to?
On Thu, Oct 08, 2009 at 04:22:11PM +0100, Rob Davies wrote:
> do you set the prefetchSize property on the network itself ? Setting
> it on the consumers wouldn't make any difference
> On 8 Oct 2009, at 15:39, Eric Van Dewoestine wrote:
>
> > I tried setting the prefetchSize to 1 for both consumers, but
> > unfortunately, that didn't resolve the issue. Messages still end up
> > stuck on B2 when C2 is stopped.
> >
> > On Thu, Oct 08, 2009 at 09:33:06AM +0100, Rob Davies wrote:
> >> set the prefetchSize=1 on your network - to avoid orphaned messages
> >> on
> >> B2
> >>
> >> On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:
> >>
> >>> Hmm, that seems like a flawed approach if the broker it visited has
> >>> changed state and is now eligible to process those messages. How is
> >>> this type of fail over intended to be handled with activemq? Is
> >>> there
> >>> an alternate solution that I'm missing?
> >>>
> >>> On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
> >>>>
> >>>> I think that by design a message will not be forwarded to a broker
> >>>> that it
> >>>> has already visited.
> >>>>
> >>>> Joe
> >>>> http://www.ttmsolutions.com
> >>>>
> >>>>
> >>>> Eric Van wrote:
> >>>>>
> >>>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots
> >>>>> listing)
> >>>>>
> >>>>> I'm running into an issue with the store and forward feature of
> >>>>> activemq, which I using in an attempt to create a highly available
> >>>>> distributed queue. I'm trying to figure out if the issue is a
> >>>>> misconfiguration on my part, expected behavior of activemq, or a
> >>>>> bug.
> >>>>>
> >>>>> The summary of the problem is that given 2 brokers, B1 and B2,
> >>>>> which
> >>>>> each have one consumer, C1 and C2, which are subscribed to the
> >>>>> same
> >>>>> queue. If I stop a consumer on one of the brokers, the pending
> >>>>> messages from that broker are not always forwarded to the other
> >>>>> broker
> >>>>> which still has a consumer, leading to those messages getting
> >>>>> indefinitely stuck.
> >>>>>
> >>>>> The steps I use to reproduce this scenario are as follows (Note:
> >>>>> all
> >>>>> producing and consuming is performed over the stomp transport):
> >>>>>
> >>>>> Couple notes about the consumers:
> >>>>> - they have a prefetchSize of 40
> >>>>> - the processing of messages can take some time, so for the
> >>>>> purposes
> >>>>> of this exercise, I've created a simple consumer that sleeps for
> >>>>> 10
> >>>>> seconds before sending the message ack (using client-individual
> >>>>> ack
> >>>>> mode)
> >>>>>
> >>>>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are
> >>>>> not
> >>>>> yet running.
> >>>>> 2. produce a few thousand messages to B1
> >>>>> Note: B1 now has a few thousand pending messages and B2 has 0.
> >>>>> 3. start consumer C2 (listing for messages from B2)
> >>>>> Note: messages are are successfully received and begin processing
> >>>>> (monitoring the brokers shows pending messages decreasing). Now
> >>>>> B2
> >>>>> has all the pending messages and B1 has 0.
> >>>>> 4. start consumer C1 (listing for messages from B1)
> >>>>> Note: no messages are received, which is another issue I have
> >>>>> since
> >>>>> B2 now has thousands of pending messages which C1 could help
> >>>>> process, but instead sits idle while C2 is forced to handle all
> >>>>> the
> >>>>> messages.
> >>>>> 5. stop consumer C2
> >>>>> Note: now I have thousands of messages sitting on B2 and 0 on B1
> >>>>> where a C1 is alive and ready to handle them. So at this point,
> >>>>> despite having a consumer running, thousands of messages are
> >>>>> stuck
> >>>>> in the queue.
> >>>>> 6. stop consumer C1
> >>>>> Note: now I have no consumers. Stopping and restarting C1 has no
> >>>>> effect on the pending messages sitting on B1's queue.
> >>>>> 7. stop both brokers
> >>>>> 8. start B1, then start B2
> >>>>> 9. start C1
> >>>>> Note: now all messages have migrated from B2 to B1 and C1 is
> >>>>> again
> >>>>> processing messages.
> >>>>>
> >>>>> So after step 5, the only way to recover from the stuck messages
> >>>>> is to
> >>>>> restart the brokers.
> >>>>>
> >>>>> Below is my current connector config which I have on both brokers.
> >>>>> I've tried playing with the various properties of the connector,
> >>>>> but
> >>>>> it seems as though no matter what I try the above scenario
> >>>>> continues
> >>>>> to occur.
> >>>>>
> >>>>> <networkConnector
> >>>>> name="default-nc"
> >>>>> uri="multicast://default?group=${broker.group}"
> >>>>> dynamicOnly="true"
> >>>>> networkTTL="25"
> >>>>> suppressDuplicateQueueSubscriptions="true"/>
> >>>>>
> >>>>>
> >>>>> So, is this an activemq bug? Am I mis-using activemq? Is there
> >>>>> some
> >>>>> other way to achieve a highly available distributed queue?
> >>>>>
> >>>>> Any help in this regard is greatly appreciated.
> >>>>>
> >>>>> --
> >>>>> eric
> >>>>>
> >>>>>
> >>>
> >>> --
> >>> eric
> >>
> >> Rob Davies
> >> http://twitter.com/rajdavies
> >> I work here: http://fusesource.com
> >> My Blog: http://rajdavies.blogspot.com/
> >> I'm writing this: http://www.manning.com/snyder/
> >
> > --
> > eric
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
--
eric
Re: Issue creating a distributed queue using store and forward.
Posted by Rob Davies <ra...@gmail.com>.
do you set the prefetchSize property on the network itself ? Setting
it on the consumers wouldn't make any difference
On 8 Oct 2009, at 15:39, Eric Van Dewoestine wrote:
> I tried setting the prefetchSize to 1 for both consumers, but
> unfortunately, that didn't resolve the issue. Messages still end up
> stuck on B2 when C2 is stopped.
>
> On Thu, Oct 08, 2009 at 09:33:06AM +0100, Rob Davies wrote:
>> set the prefetchSize=1 on your network - to avoid orphaned messages
>> on
>> B2
>>
>> On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:
>>
>>> Hmm, that seems like a flawed approach if the broker it visited has
>>> changed state and is now eligible to process those messages. How is
>>> this type of fail over intended to be handled with activemq? Is
>>> there
>>> an alternate solution that I'm missing?
>>>
>>> On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
>>>>
>>>> I think that by design a message will not be forwarded to a broker
>>>> that it
>>>> has already visited.
>>>>
>>>> Joe
>>>> http://www.ttmsolutions.com
>>>>
>>>>
>>>> Eric Van wrote:
>>>>>
>>>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots
>>>>> listing)
>>>>>
>>>>> I'm running into an issue with the store and forward feature of
>>>>> activemq, which I using in an attempt to create a highly available
>>>>> distributed queue. I'm trying to figure out if the issue is a
>>>>> misconfiguration on my part, expected behavior of activemq, or a
>>>>> bug.
>>>>>
>>>>> The summary of the problem is that given 2 brokers, B1 and B2,
>>>>> which
>>>>> each have one consumer, C1 and C2, which are subscribed to the
>>>>> same
>>>>> queue. If I stop a consumer on one of the brokers, the pending
>>>>> messages from that broker are not always forwarded to the other
>>>>> broker
>>>>> which still has a consumer, leading to those messages getting
>>>>> indefinitely stuck.
>>>>>
>>>>> The steps I use to reproduce this scenario are as follows (Note:
>>>>> all
>>>>> producing and consuming is performed over the stomp transport):
>>>>>
>>>>> Couple notes about the consumers:
>>>>> - they have a prefetchSize of 40
>>>>> - the processing of messages can take some time, so for the
>>>>> purposes
>>>>> of this exercise, I've created a simple consumer that sleeps for
>>>>> 10
>>>>> seconds before sending the message ack (using client-individual
>>>>> ack
>>>>> mode)
>>>>>
>>>>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are
>>>>> not
>>>>> yet running.
>>>>> 2. produce a few thousand messages to B1
>>>>> Note: B1 now has a few thousand pending messages and B2 has 0.
>>>>> 3. start consumer C2 (listing for messages from B2)
>>>>> Note: messages are are successfully received and begin processing
>>>>> (monitoring the brokers shows pending messages decreasing). Now
>>>>> B2
>>>>> has all the pending messages and B1 has 0.
>>>>> 4. start consumer C1 (listing for messages from B1)
>>>>> Note: no messages are received, which is another issue I have
>>>>> since
>>>>> B2 now has thousands of pending messages which C1 could help
>>>>> process, but instead sits idle while C2 is forced to handle all
>>>>> the
>>>>> messages.
>>>>> 5. stop consumer C2
>>>>> Note: now I have thousands of messages sitting on B2 and 0 on B1
>>>>> where a C1 is alive and ready to handle them. So at this point,
>>>>> despite having a consumer running, thousands of messages are
>>>>> stuck
>>>>> in the queue.
>>>>> 6. stop consumer C1
>>>>> Note: now I have no consumers. Stopping and restarting C1 has no
>>>>> effect on the pending messages sitting on B1's queue.
>>>>> 7. stop both brokers
>>>>> 8. start B1, then start B2
>>>>> 9. start C1
>>>>> Note: now all messages have migrated from B2 to B1 and C1 is
>>>>> again
>>>>> processing messages.
>>>>>
>>>>> So after step 5, the only way to recover from the stuck messages
>>>>> is to
>>>>> restart the brokers.
>>>>>
>>>>> Below is my current connector config which I have on both brokers.
>>>>> I've tried playing with the various properties of the connector,
>>>>> but
>>>>> it seems as though no matter what I try the above scenario
>>>>> continues
>>>>> to occur.
>>>>>
>>>>> <networkConnector
>>>>> name="default-nc"
>>>>> uri="multicast://default?group=${broker.group}"
>>>>> dynamicOnly="true"
>>>>> networkTTL="25"
>>>>> suppressDuplicateQueueSubscriptions="true"/>
>>>>>
>>>>>
>>>>> So, is this an activemq bug? Am I mis-using activemq? Is there
>>>>> some
>>>>> other way to achieve a highly available distributed queue?
>>>>>
>>>>> Any help in this regard is greatly appreciated.
>>>>>
>>>>> --
>>>>> eric
>>>>>
>>>>>
>>>
>>> --
>>> eric
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>
> --
> eric
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: Issue creating a distributed queue using store and forward.
Posted by Eric Van Dewoestine <er...@gmail.com>.
I tried setting the prefetchSize to 1 for both consumers, but
unfortunately, that didn't resolve the issue. Messages still end up
stuck on B2 when C2 is stopped.
On Thu, Oct 08, 2009 at 09:33:06AM +0100, Rob Davies wrote:
> set the prefetchSize=1 on your network - to avoid orphaned messages on
> B2
>
> On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:
>
> > Hmm, that seems like a flawed approach if the broker it visited has
> > changed state and is now eligible to process those messages. How is
> > this type of fail over intended to be handled with activemq? Is there
> > an alternate solution that I'm missing?
> >
> > On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
> >>
> >> I think that by design a message will not be forwarded to a broker
> >> that it
> >> has already visited.
> >>
> >> Joe
> >> http://www.ttmsolutions.com
> >>
> >>
> >> Eric Van wrote:
> >>>
> >>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
> >>>
> >>> I'm running into an issue with the store and forward feature of
> >>> activemq, which I using in an attempt to create a highly available
> >>> distributed queue. I'm trying to figure out if the issue is a
> >>> misconfiguration on my part, expected behavior of activemq, or a
> >>> bug.
> >>>
> >>> The summary of the problem is that given 2 brokers, B1 and B2, which
> >>> each have one consumer, C1 and C2, which are subscribed to the same
> >>> queue. If I stop a consumer on one of the brokers, the pending
> >>> messages from that broker are not always forwarded to the other
> >>> broker
> >>> which still has a consumer, leading to those messages getting
> >>> indefinitely stuck.
> >>>
> >>> The steps I use to reproduce this scenario are as follows (Note: all
> >>> producing and consuming is performed over the stomp transport):
> >>>
> >>> Couple notes about the consumers:
> >>> - they have a prefetchSize of 40
> >>> - the processing of messages can take some time, so for the purposes
> >>> of this exercise, I've created a simple consumer that sleeps for 10
> >>> seconds before sending the message ack (using client-individual ack
> >>> mode)
> >>>
> >>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
> >>> yet running.
> >>> 2. produce a few thousand messages to B1
> >>> Note: B1 now has a few thousand pending messages and B2 has 0.
> >>> 3. start consumer C2 (listing for messages from B2)
> >>> Note: messages are are successfully received and begin processing
> >>> (monitoring the brokers shows pending messages decreasing). Now B2
> >>> has all the pending messages and B1 has 0.
> >>> 4. start consumer C1 (listing for messages from B1)
> >>> Note: no messages are received, which is another issue I have
> >>> since
> >>> B2 now has thousands of pending messages which C1 could help
> >>> process, but instead sits idle while C2 is forced to handle all
> >>> the
> >>> messages.
> >>> 5. stop consumer C2
> >>> Note: now I have thousands of messages sitting on B2 and 0 on B1
> >>> where a C1 is alive and ready to handle them. So at this point,
> >>> despite having a consumer running, thousands of messages are stuck
> >>> in the queue.
> >>> 6. stop consumer C1
> >>> Note: now I have no consumers. Stopping and restarting C1 has no
> >>> effect on the pending messages sitting on B1's queue.
> >>> 7. stop both brokers
> >>> 8. start B1, then start B2
> >>> 9. start C1
> >>> Note: now all messages have migrated from B2 to B1 and C1 is again
> >>> processing messages.
> >>>
> >>> So after step 5, the only way to recover from the stuck messages
> >>> is to
> >>> restart the brokers.
> >>>
> >>> Below is my current connector config which I have on both brokers.
> >>> I've tried playing with the various properties of the connector, but
> >>> it seems as though no matter what I try the above scenario continues
> >>> to occur.
> >>>
> >>> <networkConnector
> >>> name="default-nc"
> >>> uri="multicast://default?group=${broker.group}"
> >>> dynamicOnly="true"
> >>> networkTTL="25"
> >>> suppressDuplicateQueueSubscriptions="true"/>
> >>>
> >>>
> >>> So, is this an activemq bug? Am I mis-using activemq? Is there some
> >>> other way to achieve a highly available distributed queue?
> >>>
> >>> Any help in this regard is greatly appreciated.
> >>>
> >>> --
> >>> eric
> >>>
> >>>
> >
> > --
> > eric
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
--
eric
Re: Issue creating a distributed queue using store and forward.
Posted by Rob Davies <ra...@gmail.com>.
set the prefetchSize=1 on your network - to avoid orphaned messages on
B2
On 7 Oct 2009, at 20:06, Eric Van Dewoestine wrote:
> Hmm, that seems like a flawed approach if the broker it visited has
> changed state and is now eligible to process those messages. How is
> this type of fail over intended to be handled with activemq? Is there
> an alternate solution that I'm missing?
>
> On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
>>
>> I think that by design a message will not be forwarded to a broker
>> that it
>> has already visited.
>>
>> Joe
>> http://www.ttmsolutions.com
>>
>>
>> Eric Van wrote:
>>>
>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
>>>
>>> I'm running into an issue with the store and forward feature of
>>> activemq, which I using in an attempt to create a highly available
>>> distributed queue. I'm trying to figure out if the issue is a
>>> misconfiguration on my part, expected behavior of activemq, or a
>>> bug.
>>>
>>> The summary of the problem is that given 2 brokers, B1 and B2, which
>>> each have one consumer, C1 and C2, which are subscribed to the same
>>> queue. If I stop a consumer on one of the brokers, the pending
>>> messages from that broker are not always forwarded to the other
>>> broker
>>> which still has a consumer, leading to those messages getting
>>> indefinitely stuck.
>>>
>>> The steps I use to reproduce this scenario are as follows (Note: all
>>> producing and consuming is performed over the stomp transport):
>>>
>>> Couple notes about the consumers:
>>> - they have a prefetchSize of 40
>>> - the processing of messages can take some time, so for the purposes
>>> of this exercise, I've created a simple consumer that sleeps for 10
>>> seconds before sending the message ack (using client-individual ack
>>> mode)
>>>
>>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
>>> yet running.
>>> 2. produce a few thousand messages to B1
>>> Note: B1 now has a few thousand pending messages and B2 has 0.
>>> 3. start consumer C2 (listing for messages from B2)
>>> Note: messages are are successfully received and begin processing
>>> (monitoring the brokers shows pending messages decreasing). Now B2
>>> has all the pending messages and B1 has 0.
>>> 4. start consumer C1 (listing for messages from B1)
>>> Note: no messages are received, which is another issue I have
>>> since
>>> B2 now has thousands of pending messages which C1 could help
>>> process, but instead sits idle while C2 is forced to handle all
>>> the
>>> messages.
>>> 5. stop consumer C2
>>> Note: now I have thousands of messages sitting on B2 and 0 on B1
>>> where a C1 is alive and ready to handle them. So at this point,
>>> despite having a consumer running, thousands of messages are stuck
>>> in the queue.
>>> 6. stop consumer C1
>>> Note: now I have no consumers. Stopping and restarting C1 has no
>>> effect on the pending messages sitting on B1's queue.
>>> 7. stop both brokers
>>> 8. start B1, then start B2
>>> 9. start C1
>>> Note: now all messages have migrated from B2 to B1 and C1 is again
>>> processing messages.
>>>
>>> So after step 5, the only way to recover from the stuck messages
>>> is to
>>> restart the brokers.
>>>
>>> Below is my current connector config which I have on both brokers.
>>> I've tried playing with the various properties of the connector, but
>>> it seems as though no matter what I try the above scenario continues
>>> to occur.
>>>
>>> <networkConnector
>>> name="default-nc"
>>> uri="multicast://default?group=${broker.group}"
>>> dynamicOnly="true"
>>> networkTTL="25"
>>> suppressDuplicateQueueSubscriptions="true"/>
>>>
>>>
>>> So, is this an activemq bug? Am I mis-using activemq? Is there some
>>> other way to achieve a highly available distributed queue?
>>>
>>> Any help in this regard is greatly appreciated.
>>>
>>> --
>>> eric
>>>
>>>
>
> --
> eric
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: Issue creating a distributed queue using store and forward.
Posted by Eric Van Dewoestine <er...@gmail.com>.
Hmm, that seems like a flawed approach if the broker it visited has
changed state and is now eligible to process those messages. How is
this type of fail over intended to be handled with activemq? Is there
an alternate solution that I'm missing?
On Wed, Oct 07, 2009 at 11:43:49AM -0700, Joe Fernandez wrote:
>
> I think that by design a message will not be forwarded to a broker that it
> has already visited.
>
> Joe
> http://www.ttmsolutions.com
>
>
> Eric Van wrote:
> >
> > ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
> >
> > I'm running into an issue with the store and forward feature of
> > activemq, which I using in an attempt to create a highly available
> > distributed queue. I'm trying to figure out if the issue is a
> > misconfiguration on my part, expected behavior of activemq, or a bug.
> >
> > The summary of the problem is that given 2 brokers, B1 and B2, which
> > each have one consumer, C1 and C2, which are subscribed to the same
> > queue. If I stop a consumer on one of the brokers, the pending
> > messages from that broker are not always forwarded to the other broker
> > which still has a consumer, leading to those messages getting
> > indefinitely stuck.
> >
> > The steps I use to reproduce this scenario are as follows (Note: all
> > producing and consuming is performed over the stomp transport):
> >
> > Couple notes about the consumers:
> > - they have a prefetchSize of 40
> > - the processing of messages can take some time, so for the purposes
> > of this exercise, I've created a simple consumer that sleeps for 10
> > seconds before sending the message ack (using client-individual ack
> > mode)
> >
> > 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
> > yet running.
> > 2. produce a few thousand messages to B1
> > Note: B1 now has a few thousand pending messages and B2 has 0.
> > 3. start consumer C2 (listing for messages from B2)
> > Note: messages are are successfully received and begin processing
> > (monitoring the brokers shows pending messages decreasing). Now B2
> > has all the pending messages and B1 has 0.
> > 4. start consumer C1 (listing for messages from B1)
> > Note: no messages are received, which is another issue I have since
> > B2 now has thousands of pending messages which C1 could help
> > process, but instead sits idle while C2 is forced to handle all the
> > messages.
> > 5. stop consumer C2
> > Note: now I have thousands of messages sitting on B2 and 0 on B1
> > where a C1 is alive and ready to handle them. So at this point,
> > despite having a consumer running, thousands of messages are stuck
> > in the queue.
> > 6. stop consumer C1
> > Note: now I have no consumers. Stopping and restarting C1 has no
> > effect on the pending messages sitting on B1's queue.
> > 7. stop both brokers
> > 8. start B1, then start B2
> > 9. start C1
> > Note: now all messages have migrated from B2 to B1 and C1 is again
> > processing messages.
> >
> > So after step 5, the only way to recover from the stuck messages is to
> > restart the brokers.
> >
> > Below is my current connector config which I have on both brokers.
> > I've tried playing with the various properties of the connector, but
> > it seems as though no matter what I try the above scenario continues
> > to occur.
> >
> > <networkConnector
> > name="default-nc"
> > uri="multicast://default?group=${broker.group}"
> > dynamicOnly="true"
> > networkTTL="25"
> > suppressDuplicateQueueSubscriptions="true"/>
> >
> >
> > So, is this an activemq bug? Am I mis-using activemq? Is there some
> > other way to achieve a highly available distributed queue?
> >
> > Any help in this regard is greatly appreciated.
> >
> > --
> > eric
> >
> >
--
eric
Re: Issue creating a distributed queue using store and forward.
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
I think that by design a message will not be forwarded to a broker that it
has already visited.
Joe
http://www.ttmsolutions.com
Eric Van wrote:
>
> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
>
> I'm running into an issue with the store and forward feature of
> activemq, which I using in an attempt to create a highly available
> distributed queue. I'm trying to figure out if the issue is a
> misconfiguration on my part, expected behavior of activemq, or a bug.
>
> The summary of the problem is that given 2 brokers, B1 and B2, which
> each have one consumer, C1 and C2, which are subscribed to the same
> queue. If I stop a consumer on one of the brokers, the pending
> messages from that broker are not always forwarded to the other broker
> which still has a consumer, leading to those messages getting
> indefinitely stuck.
>
> The steps I use to reproduce this scenario are as follows (Note: all
> producing and consuming is performed over the stomp transport):
>
> Couple notes about the consumers:
> - they have a prefetchSize of 40
> - the processing of messages can take some time, so for the purposes
> of this exercise, I've created a simple consumer that sleeps for 10
> seconds before sending the message ack (using client-individual ack
> mode)
>
> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
> yet running.
> 2. produce a few thousand messages to B1
> Note: B1 now has a few thousand pending messages and B2 has 0.
> 3. start consumer C2 (listing for messages from B2)
> Note: messages are are successfully received and begin processing
> (monitoring the brokers shows pending messages decreasing). Now B2
> has all the pending messages and B1 has 0.
> 4. start consumer C1 (listing for messages from B1)
> Note: no messages are received, which is another issue I have since
> B2 now has thousands of pending messages which C1 could help
> process, but instead sits idle while C2 is forced to handle all the
> messages.
> 5. stop consumer C2
> Note: now I have thousands of messages sitting on B2 and 0 on B1
> where a C1 is alive and ready to handle them. So at this point,
> despite having a consumer running, thousands of messages are stuck
> in the queue.
> 6. stop consumer C1
> Note: now I have no consumers. Stopping and restarting C1 has no
> effect on the pending messages sitting on B1's queue.
> 7. stop both brokers
> 8. start B1, then start B2
> 9. start C1
> Note: now all messages have migrated from B2 to B1 and C1 is again
> processing messages.
>
> So after step 5, the only way to recover from the stuck messages is to
> restart the brokers.
>
> Below is my current connector config which I have on both brokers.
> I've tried playing with the various properties of the connector, but
> it seems as though no matter what I try the above scenario continues
> to occur.
>
> <networkConnector
> name="default-nc"
> uri="multicast://default?group=${broker.group}"
> dynamicOnly="true"
> networkTTL="25"
> suppressDuplicateQueueSubscriptions="true"/>
>
>
> So, is this an activemq bug? Am I mis-using activemq? Is there some
> other way to achieve a highly available distributed queue?
>
> Any help in this regard is greatly appreciated.
>
> --
> eric
>
>
--
View this message in context: http://www.nabble.com/Issue-creating-a-distributed-queue-using-store-and-forward.-tp25790672p25791796.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Issue creating a distributed queue using store and forward.
Posted by Eric Van Dewoestine <er...@gmail.com>.
Thanks Gary, that is extremely helpful. Are there plans to address
this potential deficiency? You mentioned some improvement ideas, but
does a ticket exist yet to address them?
I'll probably take your ideas and try to at least put something
together that works for us, but since I'm very new to the activemq
code base I'm not so sure that what I end up with will be upstream
worthy.
On Fri, Oct 09, 2009 at 02:10:23PM +0100, Gary Tully wrote:
> Eric,
> just in case you have not seen it, you may find some of the detail in
> the the following old thread relevant to your investigation.
>
> http://www.nabble.com/Serious-dispatch-issue-tt23990060.html#a23996544
>
>
> 2009/10/8 Eric Van Dewoestine <er...@gmail.com>:
> > Thanks for the link. Did you try applying that patch? If so did it
> > resolve your issue?
> >
> > I'll give the patch a shot, but the note in the ticket regarding
> > failures for large message counts and Joe's warnings make me a little
> > apprehensive. I'll update my findings as soon as I get a chance to
> > test it out.
> >
> > On Thu, Oct 8, 2009 at 10:17 AM, farshad <te...@hotmail.com> wrote:
> >>
> >> Hi Eric,
> >>
> >> I think you and I have run into the same problem. Please check below.
> >>
> >> Farshad
> >>
> >> http://www.nabble.com/Question-about-Queue-destinations-in-network-of-brokers-td25776018.html
> >>
> >>
> >> Eric Van wrote:
> >>>
> >>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
> >>>
> >>> I'm running into an issue with the store and forward feature of
> >>> activemq, which I using in an attempt to create a highly available
> >>> distributed queue. I'm trying to figure out if the issue is a
> >>> misconfiguration on my part, expected behavior of activemq, or a bug.
> >>>
> >>> The summary of the problem is that given 2 brokers, B1 and B2, which
> >>> each have one consumer, C1 and C2, which are subscribed to the same
> >>> queue. If I stop a consumer on one of the brokers, the pending
> >>> messages from that broker are not always forwarded to the other broker
> >>> which still has a consumer, leading to those messages getting
> >>> indefinitely stuck.
> >>>
> >>> The steps I use to reproduce this scenario are as follows (Note: all
> >>> producing and consuming is performed over the stomp transport):
> >>>
> >>> Couple notes about the consumers:
> >>> - they have a prefetchSize of 40
> >>> - the processing of messages can take some time, so for the purposes
> >>> of this exercise, I've created a simple consumer that sleeps for 10
> >>> seconds before sending the message ack (using client-individual ack
> >>> mode)
> >>>
> >>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
> >>> yet running.
> >>> 2. produce a few thousand messages to B1
> >>> Note: B1 now has a few thousand pending messages and B2 has 0.
> >>> 3. start consumer C2 (listing for messages from B2)
> >>> Note: messages are are successfully received and begin processing
> >>> (monitoring the brokers shows pending messages decreasing). Now B2
> >>> has all the pending messages and B1 has 0.
> >>> 4. start consumer C1 (listing for messages from B1)
> >>> Note: no messages are received, which is another issue I have since
> >>> B2 now has thousands of pending messages which C1 could help
> >>> process, but instead sits idle while C2 is forced to handle all the
> >>> messages.
> >>> 5. stop consumer C2
> >>> Note: now I have thousands of messages sitting on B2 and 0 on B1
> >>> where a C1 is alive and ready to handle them. So at this point,
> >>> despite having a consumer running, thousands of messages are stuck
> >>> in the queue.
> >>> 6. stop consumer C1
> >>> Note: now I have no consumers. Stopping and restarting C1 has no
> >>> effect on the pending messages sitting on B1's queue.
> >>> 7. stop both brokers
> >>> 8. start B1, then start B2
> >>> 9. start C1
> >>> Note: now all messages have migrated from B2 to B1 and C1 is again
> >>> processing messages.
> >>>
> >>> So after step 5, the only way to recover from the stuck messages is to
> >>> restart the brokers.
> >>>
> >>> Below is my current connector config which I have on both brokers.
> >>> I've tried playing with the various properties of the connector, but
> >>> it seems as though no matter what I try the above scenario continues
> >>> to occur.
> >>>
> >>> <networkConnector
> >>> name="default-nc"
> >>> uri="multicast://default?group=${broker.group}"
> >>> dynamicOnly="true"
> >>> networkTTL="25"
> >>> suppressDuplicateQueueSubscriptions="true"/>
> >>>
> >>>
> >>> So, is this an activemq bug? Am I mis-using activemq? Is there some
> >>> other way to achieve a highly available distributed queue?
> >>>
> >>> Any help in this regard is greatly appreciated.
> >>>
> >>> --
> >>> eric
> >>>
> >>>
> >>
> >> --
> >> View this message in context: http://www.nabble.com/Issue-creating-a-distributed-queue-using-store-and-forward.-tp25790672p25803310.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> > --
> > eric
> >
>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
--
eric
Re: Issue creating a distributed queue using store and forward.
Posted by farshad <te...@hotmail.com>.
I did not try the patch as it seemed like it would make matters worse. I had
to move on and ended up implementing something else for now. I will keep an
eye on 5.3 which will hopefully resolve this issue.
-Farshad
Gary Tully wrote:
>
> Eric,
> just in case you have not seen it, you may find some of the detail in
> the the following old thread relevant to your investigation.
>
> http://www.nabble.com/Serious-dispatch-issue-tt23990060.html#a23996544
>
>
> 2009/10/8 Eric Van Dewoestine <er...@gmail.com>:
>> Thanks for the link. Did you try applying that patch? If so did it
>> resolve your issue?
>>
>> I'll give the patch a shot, but the note in the ticket regarding
>> failures for large message counts and Joe's warnings make me a little
>> apprehensive. I'll update my findings as soon as I get a chance to
>> test it out.
>>
>> On Thu, Oct 8, 2009 at 10:17 AM, farshad <te...@hotmail.com> wrote:
>>>
>>> Hi Eric,
>>>
>>> I think you and I have run into the same problem. Please check below.
>>>
>>> Farshad
>>>
>>> http://www.nabble.com/Question-about-Queue-destinations-in-network-of-brokers-td25776018.html
>>>
>>>
>>> Eric Van wrote:
>>>>
>>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
>>>>
>>>> I'm running into an issue with the store and forward feature of
>>>> activemq, which I using in an attempt to create a highly available
>>>> distributed queue. I'm trying to figure out if the issue is a
>>>> misconfiguration on my part, expected behavior of activemq, or a bug.
>>>>
>>>> The summary of the problem is that given 2 brokers, B1 and B2, which
>>>> each have one consumer, C1 and C2, which are subscribed to the same
>>>> queue. If I stop a consumer on one of the brokers, the pending
>>>> messages from that broker are not always forwarded to the other broker
>>>> which still has a consumer, leading to those messages getting
>>>> indefinitely stuck.
>>>>
>>>> The steps I use to reproduce this scenario are as follows (Note: all
>>>> producing and consuming is performed over the stomp transport):
>>>>
>>>> Couple notes about the consumers:
>>>> - they have a prefetchSize of 40
>>>> - the processing of messages can take some time, so for the purposes
>>>> of this exercise, I've created a simple consumer that sleeps for 10
>>>> seconds before sending the message ack (using client-individual ack
>>>> mode)
>>>>
>>>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
>>>> yet running.
>>>> 2. produce a few thousand messages to B1
>>>> Note: B1 now has a few thousand pending messages and B2 has 0.
>>>> 3. start consumer C2 (listing for messages from B2)
>>>> Note: messages are are successfully received and begin processing
>>>> (monitoring the brokers shows pending messages decreasing). Now B2
>>>> has all the pending messages and B1 has 0.
>>>> 4. start consumer C1 (listing for messages from B1)
>>>> Note: no messages are received, which is another issue I have since
>>>> B2 now has thousands of pending messages which C1 could help
>>>> process, but instead sits idle while C2 is forced to handle all the
>>>> messages.
>>>> 5. stop consumer C2
>>>> Note: now I have thousands of messages sitting on B2 and 0 on B1
>>>> where a C1 is alive and ready to handle them. So at this point,
>>>> despite having a consumer running, thousands of messages are stuck
>>>> in the queue.
>>>> 6. stop consumer C1
>>>> Note: now I have no consumers. Stopping and restarting C1 has no
>>>> effect on the pending messages sitting on B1's queue.
>>>> 7. stop both brokers
>>>> 8. start B1, then start B2
>>>> 9. start C1
>>>> Note: now all messages have migrated from B2 to B1 and C1 is again
>>>> processing messages.
>>>>
>>>> So after step 5, the only way to recover from the stuck messages is to
>>>> restart the brokers.
>>>>
>>>> Below is my current connector config which I have on both brokers.
>>>> I've tried playing with the various properties of the connector, but
>>>> it seems as though no matter what I try the above scenario continues
>>>> to occur.
>>>>
>>>> <networkConnector
>>>> name="default-nc"
>>>> uri="multicast://default?group=${broker.group}"
>>>> dynamicOnly="true"
>>>> networkTTL="25"
>>>> suppressDuplicateQueueSubscriptions="true"/>
>>>>
>>>>
>>>> So, is this an activemq bug? Am I mis-using activemq? Is there some
>>>> other way to achieve a highly available distributed queue?
>>>>
>>>> Any help in this regard is greatly appreciated.
>>>>
>>>> --
>>>> eric
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Issue-creating-a-distributed-queue-using-store-and-forward.-tp25790672p25803310.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>> --
>> eric
>>
>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://www.nabble.com/Issue-creating-a-distributed-queue-using-store-and-forward.-tp25790672p25865066.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Issue creating a distributed queue using store and forward.
Posted by Gary Tully <ga...@gmail.com>.
Eric,
just in case you have not seen it, you may find some of the detail in
the the following old thread relevant to your investigation.
http://www.nabble.com/Serious-dispatch-issue-tt23990060.html#a23996544
2009/10/8 Eric Van Dewoestine <er...@gmail.com>:
> Thanks for the link. Did you try applying that patch? If so did it
> resolve your issue?
>
> I'll give the patch a shot, but the note in the ticket regarding
> failures for large message counts and Joe's warnings make me a little
> apprehensive. I'll update my findings as soon as I get a chance to
> test it out.
>
> On Thu, Oct 8, 2009 at 10:17 AM, farshad <te...@hotmail.com> wrote:
>>
>> Hi Eric,
>>
>> I think you and I have run into the same problem. Please check below.
>>
>> Farshad
>>
>> http://www.nabble.com/Question-about-Queue-destinations-in-network-of-brokers-td25776018.html
>>
>>
>> Eric Van wrote:
>>>
>>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
>>>
>>> I'm running into an issue with the store and forward feature of
>>> activemq, which I using in an attempt to create a highly available
>>> distributed queue. I'm trying to figure out if the issue is a
>>> misconfiguration on my part, expected behavior of activemq, or a bug.
>>>
>>> The summary of the problem is that given 2 brokers, B1 and B2, which
>>> each have one consumer, C1 and C2, which are subscribed to the same
>>> queue. If I stop a consumer on one of the brokers, the pending
>>> messages from that broker are not always forwarded to the other broker
>>> which still has a consumer, leading to those messages getting
>>> indefinitely stuck.
>>>
>>> The steps I use to reproduce this scenario are as follows (Note: all
>>> producing and consuming is performed over the stomp transport):
>>>
>>> Couple notes about the consumers:
>>> - they have a prefetchSize of 40
>>> - the processing of messages can take some time, so for the purposes
>>> of this exercise, I've created a simple consumer that sleeps for 10
>>> seconds before sending the message ack (using client-individual ack
>>> mode)
>>>
>>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
>>> yet running.
>>> 2. produce a few thousand messages to B1
>>> Note: B1 now has a few thousand pending messages and B2 has 0.
>>> 3. start consumer C2 (listing for messages from B2)
>>> Note: messages are are successfully received and begin processing
>>> (monitoring the brokers shows pending messages decreasing). Now B2
>>> has all the pending messages and B1 has 0.
>>> 4. start consumer C1 (listing for messages from B1)
>>> Note: no messages are received, which is another issue I have since
>>> B2 now has thousands of pending messages which C1 could help
>>> process, but instead sits idle while C2 is forced to handle all the
>>> messages.
>>> 5. stop consumer C2
>>> Note: now I have thousands of messages sitting on B2 and 0 on B1
>>> where a C1 is alive and ready to handle them. So at this point,
>>> despite having a consumer running, thousands of messages are stuck
>>> in the queue.
>>> 6. stop consumer C1
>>> Note: now I have no consumers. Stopping and restarting C1 has no
>>> effect on the pending messages sitting on B1's queue.
>>> 7. stop both brokers
>>> 8. start B1, then start B2
>>> 9. start C1
>>> Note: now all messages have migrated from B2 to B1 and C1 is again
>>> processing messages.
>>>
>>> So after step 5, the only way to recover from the stuck messages is to
>>> restart the brokers.
>>>
>>> Below is my current connector config which I have on both brokers.
>>> I've tried playing with the various properties of the connector, but
>>> it seems as though no matter what I try the above scenario continues
>>> to occur.
>>>
>>> <networkConnector
>>> name="default-nc"
>>> uri="multicast://default?group=${broker.group}"
>>> dynamicOnly="true"
>>> networkTTL="25"
>>> suppressDuplicateQueueSubscriptions="true"/>
>>>
>>>
>>> So, is this an activemq bug? Am I mis-using activemq? Is there some
>>> other way to achieve a highly available distributed queue?
>>>
>>> Any help in this regard is greatly appreciated.
>>>
>>> --
>>> eric
>>>
>>>
>>
>> --
>> View this message in context: http://www.nabble.com/Issue-creating-a-distributed-queue-using-store-and-forward.-tp25790672p25803310.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
> --
> eric
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Issue creating a distributed queue using store and forward.
Posted by Eric Van Dewoestine <er...@gmail.com>.
Thanks for the link. Did you try applying that patch? If so did it
resolve your issue?
I'll give the patch a shot, but the note in the ticket regarding
failures for large message counts and Joe's warnings make me a little
apprehensive. I'll update my findings as soon as I get a chance to
test it out.
On Thu, Oct 8, 2009 at 10:17 AM, farshad <te...@hotmail.com> wrote:
>
> Hi Eric,
>
> I think you and I have run into the same problem. Please check below.
>
> Farshad
>
> http://www.nabble.com/Question-about-Queue-destinations-in-network-of-brokers-td25776018.html
>
>
> Eric Van wrote:
>>
>> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
>>
>> I'm running into an issue with the store and forward feature of
>> activemq, which I using in an attempt to create a highly available
>> distributed queue. I'm trying to figure out if the issue is a
>> misconfiguration on my part, expected behavior of activemq, or a bug.
>>
>> The summary of the problem is that given 2 brokers, B1 and B2, which
>> each have one consumer, C1 and C2, which are subscribed to the same
>> queue. If I stop a consumer on one of the brokers, the pending
>> messages from that broker are not always forwarded to the other broker
>> which still has a consumer, leading to those messages getting
>> indefinitely stuck.
>>
>> The steps I use to reproduce this scenario are as follows (Note: all
>> producing and consuming is performed over the stomp transport):
>>
>> Couple notes about the consumers:
>> - they have a prefetchSize of 40
>> - the processing of messages can take some time, so for the purposes
>> of this exercise, I've created a simple consumer that sleeps for 10
>> seconds before sending the message ack (using client-individual ack
>> mode)
>>
>> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
>> yet running.
>> 2. produce a few thousand messages to B1
>> Note: B1 now has a few thousand pending messages and B2 has 0.
>> 3. start consumer C2 (listing for messages from B2)
>> Note: messages are are successfully received and begin processing
>> (monitoring the brokers shows pending messages decreasing). Now B2
>> has all the pending messages and B1 has 0.
>> 4. start consumer C1 (listing for messages from B1)
>> Note: no messages are received, which is another issue I have since
>> B2 now has thousands of pending messages which C1 could help
>> process, but instead sits idle while C2 is forced to handle all the
>> messages.
>> 5. stop consumer C2
>> Note: now I have thousands of messages sitting on B2 and 0 on B1
>> where a C1 is alive and ready to handle them. So at this point,
>> despite having a consumer running, thousands of messages are stuck
>> in the queue.
>> 6. stop consumer C1
>> Note: now I have no consumers. Stopping and restarting C1 has no
>> effect on the pending messages sitting on B1's queue.
>> 7. stop both brokers
>> 8. start B1, then start B2
>> 9. start C1
>> Note: now all messages have migrated from B2 to B1 and C1 is again
>> processing messages.
>>
>> So after step 5, the only way to recover from the stuck messages is to
>> restart the brokers.
>>
>> Below is my current connector config which I have on both brokers.
>> I've tried playing with the various properties of the connector, but
>> it seems as though no matter what I try the above scenario continues
>> to occur.
>>
>> <networkConnector
>> name="default-nc"
>> uri="multicast://default?group=${broker.group}"
>> dynamicOnly="true"
>> networkTTL="25"
>> suppressDuplicateQueueSubscriptions="true"/>
>>
>>
>> So, is this an activemq bug? Am I mis-using activemq? Is there some
>> other way to achieve a highly available distributed queue?
>>
>> Any help in this regard is greatly appreciated.
>>
>> --
>> eric
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Issue-creating-a-distributed-queue-using-store-and-forward.-tp25790672p25803310.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
eric
Re: Issue creating a distributed queue using store and forward.
Posted by farshad <te...@hotmail.com>.
Hi Eric,
I think you and I have run into the same problem. Please check below.
Farshad
http://www.nabble.com/Question-about-Queue-destinations-in-network-of-brokers-td25776018.html
Eric Van wrote:
>
> ActiveMQ 5.3.0_SNAPSHOT (Sep 8th according to the snapshots listing)
>
> I'm running into an issue with the store and forward feature of
> activemq, which I using in an attempt to create a highly available
> distributed queue. I'm trying to figure out if the issue is a
> misconfiguration on my part, expected behavior of activemq, or a bug.
>
> The summary of the problem is that given 2 brokers, B1 and B2, which
> each have one consumer, C1 and C2, which are subscribed to the same
> queue. If I stop a consumer on one of the brokers, the pending
> messages from that broker are not always forwarded to the other broker
> which still has a consumer, leading to those messages getting
> indefinitely stuck.
>
> The steps I use to reproduce this scenario are as follows (Note: all
> producing and consuming is performed over the stomp transport):
>
> Couple notes about the consumers:
> - they have a prefetchSize of 40
> - the processing of messages can take some time, so for the purposes
> of this exercise, I've created a simple consumer that sleeps for 10
> seconds before sending the message ack (using client-individual ack
> mode)
>
> 1. start both brokers (B1 and B2). The consumers (C1 and C2) are not
> yet running.
> 2. produce a few thousand messages to B1
> Note: B1 now has a few thousand pending messages and B2 has 0.
> 3. start consumer C2 (listing for messages from B2)
> Note: messages are are successfully received and begin processing
> (monitoring the brokers shows pending messages decreasing). Now B2
> has all the pending messages and B1 has 0.
> 4. start consumer C1 (listing for messages from B1)
> Note: no messages are received, which is another issue I have since
> B2 now has thousands of pending messages which C1 could help
> process, but instead sits idle while C2 is forced to handle all the
> messages.
> 5. stop consumer C2
> Note: now I have thousands of messages sitting on B2 and 0 on B1
> where a C1 is alive and ready to handle them. So at this point,
> despite having a consumer running, thousands of messages are stuck
> in the queue.
> 6. stop consumer C1
> Note: now I have no consumers. Stopping and restarting C1 has no
> effect on the pending messages sitting on B1's queue.
> 7. stop both brokers
> 8. start B1, then start B2
> 9. start C1
> Note: now all messages have migrated from B2 to B1 and C1 is again
> processing messages.
>
> So after step 5, the only way to recover from the stuck messages is to
> restart the brokers.
>
> Below is my current connector config which I have on both brokers.
> I've tried playing with the various properties of the connector, but
> it seems as though no matter what I try the above scenario continues
> to occur.
>
> <networkConnector
> name="default-nc"
> uri="multicast://default?group=${broker.group}"
> dynamicOnly="true"
> networkTTL="25"
> suppressDuplicateQueueSubscriptions="true"/>
>
>
> So, is this an activemq bug? Am I mis-using activemq? Is there some
> other way to achieve a highly available distributed queue?
>
> Any help in this regard is greatly appreciated.
>
> --
> eric
>
>
--
View this message in context: http://www.nabble.com/Issue-creating-a-distributed-queue-using-store-and-forward.-tp25790672p25803310.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.