You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Andreas Gies <an...@wayofquality.de> on 2014/10/18 08:46:35 UTC
Duplicate messages durable subscriber
Hi
I am using ActiveMQ 5.10 in an application. So far the requirement for the remote locations has been for pure store and forward capabilities,
so that a single AMQ broker was sufficient. This has changed in a way that now 2 nodes should be present in the remote location for
resilience and load balancing. I had considered a master/configuration as the requirement for resilience is stronger than that for load balancing.
However, the situation in those locations is that I don’t have a shared db nor a shared filesystem. As far as I have understood the replicated level db
would require at least 3 nodes ?
This is why I have chosen a network of brokers in the end, which works well for any Queue based communication.
Now my problem is that there is one client application that is provided by a 3rd party and uses durable subscriptions. It would be quite an effort
to change that application towards using queues, so that I could consider virtual destinations.
The problem occurs two-fold:
Assume a Subscriber connects to BrokerA, then disconnects and reconnects to Broker B. It consumes messages for a while, than disconnects
and reconnects to Broker A. All messages that have already been consumed while it was connected to Broker B will be delivered again.
My question is now whether this could be avoided by means of ActiveMQ alone ? - I was contemplating a broker plugin to track messages that
have been consumed on other nodes so that I could avoid redelivering them again.
Sorry if thats a bit vague - I am fishing for ideas ….
Thanks and best regards
Andreas
Re: Duplicate messages durable subscriber
Posted by Andreas Gies <an...@wayofquality.de>.
That's the way I'd prefer it ;)
Will keep you posted
Andreas
On 04/11/14 18:55, Tim Bain wrote:
> The functionality you're contemplating for the broker plugin seems pretty
> far off from the standard ActiveMQ approach, since it involves knowing the
> state of a message across the full network of brokers rather than allowing
> each broker to operate as a stand-alone island that only needs to know
> about itself and how to pass message to its neighbors. So I'd be very
> hesitant to go down the path you're suggesting.
>
> And I suspect you won't have to; the functionality you're looking to use is
> how ActiveMQ should work, so I think you'll find that either there's a bug
> in ActiveMQ or a bug in your configuration. Or maybe there's a minor
> missing feature or one that's not been applied to all the transports it
> should have; either way, I suspect the actual fix will turn out to be much
> simpler than the plugin you're contemplating.
>
> We'll see what shakes out as you simplify your configuration to remove the
> less-common configuration options. Eventually we should get to a
> configuration that delivers the messages correctly but maybe doesn't have
> all the configuration options enabled that you want to use, and then it's a
> question of why it stops working with the less-common configuration options
> you're using.
>
> On Tue, Nov 4, 2014 at 8:16 AM, Andreas Gies <an...@wayofquality.de>
> wrote:
>
>> Hi Tim,
>>
>> I will make some time tomorrow to try your suggestions.
>>
>> A Virtual destination would simply replace a durable subscriber with a
>> Queue receiver.
>> Messages posted to a mapped topic would end up in the the underlying
>> queues.
>>
>> Now, a consumer that wasn't connected to the peer broker would immediately
>> receive
>> the messages on the broker - that would address the problem of temporarily
>> unvisible
>> messages.
>>
>> As for the duplicated delivery of messages - effectively we would have
>> concurrent consumers
>> (even though most likely onlz one of the would be connected at any one
>> point in time).
>> A message consumed on either side of the NWOB would effectively vanish
>> from the underlying
>> Queue and not be delivered again.
>>
>>
>> Due to that I was contemplating a distributed broker plugin that would
>> kind of synchronize the
>> consumption of messages of a DS within a NWOB.
>>
>> However, before going that path I wanted to make sure that I am not
>> missing something obvious.
>>
>>
>> Thanks for the answers so far and best regards
>> Andreas
>>
>> On 03/11/14 20:23, Tim Bain wrote:
>>
>>> Andreas,
>>>
>>> Your spec added two configuration elements your previous post didn't
>>> mention, and I'd like to eliminate each of them in turn to see if it's
>>> causing/contributing to the problem.
>>>
>>> 1. Your networkConnectors are apparently multicast. Please see what
>>> happens if you configure them as
>>> static:(tcp://host:port?tcpOptions)?staticOptions, to take the
>>> multicast
>>> (and the broker discovery that it's presumably doing) out of the
>>> equation.
>>> I recently experimented with what happens when the failover is
>>> allowed to
>>> perform a reconnect in a broker-to-broker networkConnector, and the
>>> result
>>> is duplicate and/or stale subscriptions between the brokers. That
>>> behavior
>>> could explain what you're seeing, if multicast is similarly performing
>>> reconnects without notifying the static wrapper so it can recreate the
>>> network bridge, so let's take it out of the equation to see if the
>>> behavior
>>> changes. (I've never used multicast, so this might not make sense; if
>>> someone knows that this can't be the issue, please say so.)
>>> 2. I don't know how gracefully conduitSubscriptions reacts to
>>> consumers
>>>
>>> moving around the network of brokers; I don't believe this should be
>>> the
>>> problem, but if #1 doesn't produce any change in behavior, can you set
>>> conduitSubscriptions=false and see if anything changes?
>>>
>>> I'm not clear on how Virtual Topics will solve the problem; can you
>>> explain? To me this feels like a problem with broker-to-broker management
>>> of subscriptions made on behalf of clients (most likely duplicate
>>> subscriptions for a client, one from each broker, after a failover), and
>>> I'm not sure how a Virtual Topic would make it any better if that's the
>>> case. But if you know of a way that it would, that might help me to
>>> understand what's going on.
>>>
>>> Tim
>>>
>>> On Sat, Nov 1, 2014 at 9:50 AM, Andreas Gies <an...@wayofquality.de>
>>> wrote:
>>>
>>> Hello Tim,
>>>> thanks for your answer. It took me a bit to digest it - so my apologies
>>>> for
>>>> the delay in my answer.
>>>>
>>>> I have come up with a test case that shows - and unfortunately confirms
>>>> my observations.The test case is located at [1].
>>>>
>>>> Here is the excerpt of my problem descriptions & observations:
>>>>
>>>> /**
>>>> * This specification shall help to investigate the duplicate delivery
>>>> of
>>>> messages for durable subscribers
>>>> * within a network of brokers. The problem has been posted on the
>>>> ActiveMQ mailing list on Oct. 18th 2014
>>>> * and was described as follows:
>>>> *
>>>> * Suppose you have a network of brokers consisting of two members
>>>> discovering each other via multicast.
>>>> * The network bridge is set up using conduit subscriptions. Now assume
>>>> that we have a durable subscriber
>>>> * named "S" that connects to the network of brokers using a failover
>>>> uri
>>>> pointing to both brokers.
>>>> *
>>>> * First, the subscriber connects to Broker A. It will consume all
>>>> messages published to either Broker A or B.
>>>> * Now the subscriber disconnects and stays offline for a bit, then it
>>>> reconnects to Broker B. Now it will pick
>>>> * up all messages that have been published while it was offline.
>>>> *
>>>> * Let's say then 10 messages are published. All is well as the
>>>> subscriber
>>>> consumes those messages.
>>>> * If the subscriber then disconnects and reconnects to Broker A, these
>>>> 10
>>>> messages will be consumed
>>>> * again by the reconnected subscriber.
>>>> *
>>>> * According to Tim Bain on Oct., 20th 2014 this indicates a bug rather
>>>> than a missing feature in ActiveMQ
>>>> * and this Spec shall pinpoint the behavior.
>>>> * *
>>>> * The test is based on ActiveMQ 5.10
>>>> *
>>>> * Observations:
>>>> * -------------
>>>> * Depending on when the durable subscriber is known to the members of
>>>> the
>>>> NWOB, messages can be either left pending
>>>> * or delivered repeatedly (see the last 2 test cases). Message gaps can
>>>> occur, if the DS has only connected
>>>> * to one broker so far. If the DS then disconnects and after a while
>>>> reconnects to the other broker it wasn't
>>>> * connected to so far, it will not see the messages that have been
>>>> produced while it was offline (it will see
>>>> * those messages after reconnecting to broker 1).
>>>> *
>>>> * Dupilcate delivery will happen if the DS was already connected to
>>>> both
>>>> brokers. From the broker's perspective
>>>> * it seems that those DS are handled as two distinct subscribers, so
>>>> effectively all messages that are published
>>>> * will eventually be delivered to both subscribers.
>>>> */
>>>>
>>>> I know that Virtual Topics could solve the problem - however we in the
>>>> middleware team are not in control of that
>>>> particular client application and therefor we cannot change the consumer
>>>> from a DS to a queue consumer.
>>>>
>>>> Can you confirm that we are indeed looking at a missing feature or a bug
>>>> in ActiveMQ 5.10 ? - Otherwise i would
>>>> need to get my thinking cap back on and see how I could solve the problem
>>>> without changing the client code.
>>>>
>>>> [1]
>>>> https://github.com/woq-blended/blended/blob/master/
>>>> blended-testing/blended-testing-activemq/src/test/
>>>> scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
>>>> <
>>>> https://github.com/woq-blended/blended/blob/master/
>>>> blended-testing/blended-testing-activemq/src/test/
>>>> scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
>>>> Thanks and best regards
>>>> Andreas
>>>>
>>>>
>>>> On 20 Oct 2014, at 17:40, Tim Bain <tb...@alumni.duke.edu> wrote:
>>>>> If you have a network of brokers, messages on topics will be forwarded
>>>>> to
>>>>> whichever broker the consumer connects to, without duplicate delivery of
>>>>> any messages so long as no messages were processed by the consumer
>>>>>
>>>> without
>>>>
>>>>> being ack'ed. If you were using queues, there's the potential for
>>>>>
>>>> messages
>>>>
>>>>> to get stranded on a broker if no consumers are left, but this isn't
>>>>> possible for topics. (I'm not clear on the reason that topics can't get
>>>>> messages stranded even when consumers bounce between brokers, and
>>>>> unfortunately http://activemq.apache.org/networks-of-brokers.html
>>>>>
>>>> doesn't
>>>>
>>>>> describe why that is.)
>>>>>
>>>>> So I think that ActiveMQ's base capabilities will do exactly what you
>>>>>
>>>> want,
>>>>
>>>>> and if you're seeing redelivery of messages that were successfully acked
>>>>> when the consumer bounces to another broker, I think that would indicate
>>>>>
>>>> a
>>>>
>>>>> bug in ActiveMQ rather than a missing feature.
>>>>>
>>>>> On Sun, Oct 19, 2014 at 6:05 PM, Noel OConnor <no...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Take a look at idempotent consumers in camel. This may help you out as
>>>>>> a
>>>>>> basis for your plugin if you decide to go with it.
>>>>>> On Oct 18, 2014 5:47 PM, "Andreas Gies" <an...@wayofquality.de>
>>>>>>
>>>>> wrote:
>>>>> Hi
>>>>>>> I am using ActiveMQ 5.10 in an application. So far the requirement for
>>>>>>>
>>>>>> the
>>>>>>
>>>>>>> remote locations has been for pure store and forward capabilities,
>>>>>>> so that a single AMQ broker was sufficient. This has changed in a way
>>>>>>>
>>>>>> that
>>>>>>
>>>>>>> now 2 nodes should be present in the remote location for
>>>>>>> resilience and load balancing. I had considered a master/configuration
>>>>>>>
>>>>>> as
>>>>> the requirement for resilience is stronger than that for load
>>>>>> balancing.
>>>>> However, the situation in those locations is that I don’t have a shared
>>>>>> db
>>>>>>
>>>>>>> nor a shared filesystem. As far as I have understood the replicated
>>>>>>>
>>>>>> level db
>>>>>>
>>>>>>> would require at least 3 nodes ?
>>>>>>>
>>>>>>> This is why I have chosen a network of brokers in the end, which works
>>>>>>> well for any Queue based communication.
>>>>>>>
>>>>>>> Now my problem is that there is one client application that is
>>>>>>> provided
>>>>>>>
>>>>>> by
>>>>>>
>>>>>>> a 3rd party and uses durable subscriptions. It would be quite an
>>>>>>> effort
>>>>>>> to change that application towards using queues, so that I could
>>>>>>>
>>>>>> consider
>>>>> virtual destinations.
>>>>>>> The problem occurs two-fold:
>>>>>>>
>>>>>>> Assume a Subscriber connects to BrokerA, then disconnects and
>>>>>>>
>>>>>> reconnects
>>>>> to Broker B. It consumes messages for a while, than disconnects
>>>>>>> and reconnects to Broker A. All messages that have already been
>>>>>>>
>>>>>> consumed
>>>>> while it was connected to Broker B will be delivered again.
>>>>>>> My question is now whether this could be avoided by means of ActiveMQ
>>>>>>> alone ? - I was contemplating a broker plugin to track messages that
>>>>>>> have been consumed on other nodes so that I could avoid redelivering
>>>>>>>
>>>>>> them
>>>>> again.
>>>>>>> Sorry if thats a bit vague - I am fishing for ideas ….
>>>>>>>
>>>>>>>
>>>>>>> Thanks and best regards
>>>>>>> Andreas
>>>>>>>
>> --
>>
>>
>> Andreas Gies
>>
>> WoQ – Way of Quality GmbH
>>
>> Geschäftsführer & CTO
>>
>> /eMail:/andreas@wayofquality.de <ma...@wayofquality.de>
>>
>> /Tel:/ +49 151 23470823
>>
>> /Fax:/ +49 1805 006534 2114
>>
>> /Twitter:/ andreasgies /Skype:/ giessonic
>>
>> /LinkedIn:/ <http://de.linkedin.com/pub/andreas-gies/0/594/aa5/> (
>> http://de.linkedin.com/pub/andreas-gies/0/594/aa5/)
>>
>> /Xing:/ <http://www.xing.com/profile/Andreas_Gies> (
>> http://www.xing.com/profile/Andreas_Gies)
>>
>> /Blog:/ <http://www.wayofquality.de/index.php/en/blog> (
>> http://www.wayofquality.de/index.php/en/blog)
>>
>> /Github:/ <https://github.com/atooni> (https://github.com/atooni)
>>
>> /Amtsgericht Landshut:/HRB 8352//
>>
>> //
>>
>> /Ust.-Id.:/ DE274771254
>>
>>
>> Haftungsausschluss
>>
>> Diese Email kann vertrauliche und/oder rechtlich geschützte Informationen
>> enthalten und ist ausschließlich für den/die benannten Adressaten bestimmt.
>> Sollten Sie nicht der beabsichtigte Empfänger sein oder diese Email
>> irrtümlich erhalten haben, ist es Ihnen nicht gestattet diese Mail oder
>> einen Teil davon ohne unsere Erlaubnis zu verbreiten, zu kopieren, unbefugt
>> weiterzuleiten oder zu behalten. Informieren Sie bitte sofort den Absender
>> telefonisch oder per Email und löschen Sie diese Email und alle Kopien aus
>> Ihrem System. Wir haften nicht für die Unversehrtheit von Emails, nachdem
>> sie unseren Einflussbereich verlassen haben.
>>
>>
>> Disclaimer
>>
>> This email may contain confidential and/or privileged information and is
>> intended solely for the attention and use of the named addressee(s). If you
>> are not the intended recipient, or a person responsible for delivering it
>> to the intended recipient, you are not authorized to and must not disclose,
>> copy, distribute, or retain this message or any part of it without our
>> authority. Please contact the sender by call or reply email immediately and
>> destroy all copies and the original message. We are not responsible for the
>> integrity of emails after they have left our sphere of control.
>>
>> //
>>
--
Andreas Gies
WoQ – Way of Quality GmbH
Geschäftsführer & CTO
/eMail:/andreas@wayofquality.de <ma...@wayofquality.de>
/Tel:/ +49 151 23470823
/Fax:/ +49 1805 006534 2114
/Twitter:/ andreasgies /Skype:/ giessonic
/LinkedIn:/ <http://de.linkedin.com/pub/andreas-gies/0/594/aa5/>
(http://de.linkedin.com/pub/andreas-gies/0/594/aa5/)
/Xing:/ <http://www.xing.com/profile/Andreas_Gies>
(http://www.xing.com/profile/Andreas_Gies)
/Blog:/ <http://www.wayofquality.de/index.php/en/blog>
(http://www.wayofquality.de/index.php/en/blog)
/Github:/ <https://github.com/atooni> (https://github.com/atooni)
/Amtsgericht Landshut:/HRB 8352//
//
/Ust.-Id.:/ DE274771254
Haftungsausschluss
Diese Email kann vertrauliche und/oder rechtlich geschützte
Informationen enthalten und ist ausschließlich für den/die benannten
Adressaten bestimmt. Sollten Sie nicht der beabsichtigte Empfänger sein
oder diese Email irrtümlich erhalten haben, ist es Ihnen nicht gestattet
diese Mail oder einen Teil davon ohne unsere Erlaubnis zu verbreiten, zu
kopieren, unbefugt weiterzuleiten oder zu behalten. Informieren Sie
bitte sofort den Absender telefonisch oder per Email und löschen Sie
diese Email und alle Kopien aus Ihrem System. Wir haften nicht für die
Unversehrtheit von Emails, nachdem sie unseren Einflussbereich verlassen
haben.
Disclaimer
This email may contain confidential and/or privileged information and is
intended solely for the attention and use of the named addressee(s). If
you are not the intended recipient, or a person responsible for
delivering it to the intended recipient, you are not authorized to and
must not disclose, copy, distribute, or retain this message or any part
of it without our authority. Please contact the sender by call or reply
email immediately and destroy all copies and the original message. We
are not responsible for the integrity of emails after they have left our
sphere of control.
//
Re: Duplicate messages durable subscriber
Posted by Tim Bain <tb...@alumni.duke.edu>.
The functionality you're contemplating for the broker plugin seems pretty
far off from the standard ActiveMQ approach, since it involves knowing the
state of a message across the full network of brokers rather than allowing
each broker to operate as a stand-alone island that only needs to know
about itself and how to pass message to its neighbors. So I'd be very
hesitant to go down the path you're suggesting.
And I suspect you won't have to; the functionality you're looking to use is
how ActiveMQ should work, so I think you'll find that either there's a bug
in ActiveMQ or a bug in your configuration. Or maybe there's a minor
missing feature or one that's not been applied to all the transports it
should have; either way, I suspect the actual fix will turn out to be much
simpler than the plugin you're contemplating.
We'll see what shakes out as you simplify your configuration to remove the
less-common configuration options. Eventually we should get to a
configuration that delivers the messages correctly but maybe doesn't have
all the configuration options enabled that you want to use, and then it's a
question of why it stops working with the less-common configuration options
you're using.
On Tue, Nov 4, 2014 at 8:16 AM, Andreas Gies <an...@wayofquality.de>
wrote:
> Hi Tim,
>
> I will make some time tomorrow to try your suggestions.
>
> A Virtual destination would simply replace a durable subscriber with a
> Queue receiver.
> Messages posted to a mapped topic would end up in the the underlying
> queues.
>
> Now, a consumer that wasn't connected to the peer broker would immediately
> receive
> the messages on the broker - that would address the problem of temporarily
> unvisible
> messages.
>
> As for the duplicated delivery of messages - effectively we would have
> concurrent consumers
> (even though most likely onlz one of the would be connected at any one
> point in time).
> A message consumed on either side of the NWOB would effectively vanish
> from the underlying
> Queue and not be delivered again.
>
>
> Due to that I was contemplating a distributed broker plugin that would
> kind of synchronize the
> consumption of messages of a DS within a NWOB.
>
> However, before going that path I wanted to make sure that I am not
> missing something obvious.
>
>
> Thanks for the answers so far and best regards
> Andreas
>
> On 03/11/14 20:23, Tim Bain wrote:
>
>> Andreas,
>>
>> Your spec added two configuration elements your previous post didn't
>> mention, and I'd like to eliminate each of them in turn to see if it's
>> causing/contributing to the problem.
>>
>> 1. Your networkConnectors are apparently multicast. Please see what
>> happens if you configure them as
>> static:(tcp://host:port?tcpOptions)?staticOptions, to take the
>> multicast
>> (and the broker discovery that it's presumably doing) out of the
>> equation.
>> I recently experimented with what happens when the failover is
>> allowed to
>> perform a reconnect in a broker-to-broker networkConnector, and the
>> result
>> is duplicate and/or stale subscriptions between the brokers. That
>> behavior
>> could explain what you're seeing, if multicast is similarly performing
>> reconnects without notifying the static wrapper so it can recreate the
>> network bridge, so let's take it out of the equation to see if the
>> behavior
>> changes. (I've never used multicast, so this might not make sense; if
>> someone knows that this can't be the issue, please say so.)
>> 2. I don't know how gracefully conduitSubscriptions reacts to
>> consumers
>>
>> moving around the network of brokers; I don't believe this should be
>> the
>> problem, but if #1 doesn't produce any change in behavior, can you set
>> conduitSubscriptions=false and see if anything changes?
>>
>> I'm not clear on how Virtual Topics will solve the problem; can you
>> explain? To me this feels like a problem with broker-to-broker management
>> of subscriptions made on behalf of clients (most likely duplicate
>> subscriptions for a client, one from each broker, after a failover), and
>> I'm not sure how a Virtual Topic would make it any better if that's the
>> case. But if you know of a way that it would, that might help me to
>> understand what's going on.
>>
>> Tim
>>
>> On Sat, Nov 1, 2014 at 9:50 AM, Andreas Gies <an...@wayofquality.de>
>> wrote:
>>
>> Hello Tim,
>>>
>>> thanks for your answer. It took me a bit to digest it - so my apologies
>>> for
>>> the delay in my answer.
>>>
>>> I have come up with a test case that shows - and unfortunately confirms
>>> my observations.The test case is located at [1].
>>>
>>> Here is the excerpt of my problem descriptions & observations:
>>>
>>> /**
>>> * This specification shall help to investigate the duplicate delivery
>>> of
>>> messages for durable subscribers
>>> * within a network of brokers. The problem has been posted on the
>>> ActiveMQ mailing list on Oct. 18th 2014
>>> * and was described as follows:
>>> *
>>> * Suppose you have a network of brokers consisting of two members
>>> discovering each other via multicast.
>>> * The network bridge is set up using conduit subscriptions. Now assume
>>> that we have a durable subscriber
>>> * named "S" that connects to the network of brokers using a failover
>>> uri
>>> pointing to both brokers.
>>> *
>>> * First, the subscriber connects to Broker A. It will consume all
>>> messages published to either Broker A or B.
>>> * Now the subscriber disconnects and stays offline for a bit, then it
>>> reconnects to Broker B. Now it will pick
>>> * up all messages that have been published while it was offline.
>>> *
>>> * Let's say then 10 messages are published. All is well as the
>>> subscriber
>>> consumes those messages.
>>> * If the subscriber then disconnects and reconnects to Broker A, these
>>> 10
>>> messages will be consumed
>>> * again by the reconnected subscriber.
>>> *
>>> * According to Tim Bain on Oct., 20th 2014 this indicates a bug rather
>>> than a missing feature in ActiveMQ
>>> * and this Spec shall pinpoint the behavior.
>>> * *
>>> * The test is based on ActiveMQ 5.10
>>> *
>>> * Observations:
>>> * -------------
>>> * Depending on when the durable subscriber is known to the members of
>>> the
>>> NWOB, messages can be either left pending
>>> * or delivered repeatedly (see the last 2 test cases). Message gaps can
>>> occur, if the DS has only connected
>>> * to one broker so far. If the DS then disconnects and after a while
>>> reconnects to the other broker it wasn't
>>> * connected to so far, it will not see the messages that have been
>>> produced while it was offline (it will see
>>> * those messages after reconnecting to broker 1).
>>> *
>>> * Dupilcate delivery will happen if the DS was already connected to
>>> both
>>> brokers. From the broker's perspective
>>> * it seems that those DS are handled as two distinct subscribers, so
>>> effectively all messages that are published
>>> * will eventually be delivered to both subscribers.
>>> */
>>>
>>> I know that Virtual Topics could solve the problem - however we in the
>>> middleware team are not in control of that
>>> particular client application and therefor we cannot change the consumer
>>> from a DS to a queue consumer.
>>>
>>> Can you confirm that we are indeed looking at a missing feature or a bug
>>> in ActiveMQ 5.10 ? - Otherwise i would
>>> need to get my thinking cap back on and see how I could solve the problem
>>> without changing the client code.
>>>
>>> [1]
>>> https://github.com/woq-blended/blended/blob/master/
>>> blended-testing/blended-testing-activemq/src/test/
>>> scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
>>> <
>>> https://github.com/woq-blended/blended/blob/master/
>>> blended-testing/blended-testing-activemq/src/test/
>>> scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
>>> Thanks and best regards
>>> Andreas
>>>
>>>
>>> On 20 Oct 2014, at 17:40, Tim Bain <tb...@alumni.duke.edu> wrote:
>>>>
>>>> If you have a network of brokers, messages on topics will be forwarded
>>>> to
>>>> whichever broker the consumer connects to, without duplicate delivery of
>>>> any messages so long as no messages were processed by the consumer
>>>>
>>> without
>>>
>>>> being ack'ed. If you were using queues, there's the potential for
>>>>
>>> messages
>>>
>>>> to get stranded on a broker if no consumers are left, but this isn't
>>>> possible for topics. (I'm not clear on the reason that topics can't get
>>>> messages stranded even when consumers bounce between brokers, and
>>>> unfortunately http://activemq.apache.org/networks-of-brokers.html
>>>>
>>> doesn't
>>>
>>>> describe why that is.)
>>>>
>>>> So I think that ActiveMQ's base capabilities will do exactly what you
>>>>
>>> want,
>>>
>>>> and if you're seeing redelivery of messages that were successfully acked
>>>> when the consumer bounces to another broker, I think that would indicate
>>>>
>>> a
>>>
>>>> bug in ActiveMQ rather than a missing feature.
>>>>
>>>> On Sun, Oct 19, 2014 at 6:05 PM, Noel OConnor <no...@gmail.com>
>>>> wrote:
>>>>
>>>> Take a look at idempotent consumers in camel. This may help you out as
>>>>> a
>>>>> basis for your plugin if you decide to go with it.
>>>>> On Oct 18, 2014 5:47 PM, "Andreas Gies" <an...@wayofquality.de>
>>>>>
>>>> wrote:
>>>
>>>> Hi
>>>>>>
>>>>>> I am using ActiveMQ 5.10 in an application. So far the requirement for
>>>>>>
>>>>> the
>>>>>
>>>>>> remote locations has been for pure store and forward capabilities,
>>>>>> so that a single AMQ broker was sufficient. This has changed in a way
>>>>>>
>>>>> that
>>>>>
>>>>>> now 2 nodes should be present in the remote location for
>>>>>> resilience and load balancing. I had considered a master/configuration
>>>>>>
>>>>> as
>>>
>>>> the requirement for resilience is stronger than that for load
>>>>>>
>>>>> balancing.
>>>
>>>> However, the situation in those locations is that I don’t have a shared
>>>>>>
>>>>> db
>>>>>
>>>>>> nor a shared filesystem. As far as I have understood the replicated
>>>>>>
>>>>> level db
>>>>>
>>>>>> would require at least 3 nodes ?
>>>>>>
>>>>>> This is why I have chosen a network of brokers in the end, which works
>>>>>> well for any Queue based communication.
>>>>>>
>>>>>> Now my problem is that there is one client application that is
>>>>>> provided
>>>>>>
>>>>> by
>>>>>
>>>>>> a 3rd party and uses durable subscriptions. It would be quite an
>>>>>> effort
>>>>>> to change that application towards using queues, so that I could
>>>>>>
>>>>> consider
>>>
>>>> virtual destinations.
>>>>>>
>>>>>> The problem occurs two-fold:
>>>>>>
>>>>>> Assume a Subscriber connects to BrokerA, then disconnects and
>>>>>>
>>>>> reconnects
>>>
>>>> to Broker B. It consumes messages for a while, than disconnects
>>>>>> and reconnects to Broker A. All messages that have already been
>>>>>>
>>>>> consumed
>>>
>>>> while it was connected to Broker B will be delivered again.
>>>>>>
>>>>>> My question is now whether this could be avoided by means of ActiveMQ
>>>>>> alone ? - I was contemplating a broker plugin to track messages that
>>>>>> have been consumed on other nodes so that I could avoid redelivering
>>>>>>
>>>>> them
>>>
>>>> again.
>>>>>>
>>>>>> Sorry if thats a bit vague - I am fishing for ideas ….
>>>>>>
>>>>>>
>>>>>> Thanks and best regards
>>>>>> Andreas
>>>>>>
>>>>>
>>>
> --
>
>
> Andreas Gies
>
> WoQ – Way of Quality GmbH
>
> Geschäftsführer & CTO
>
> /eMail:/andreas@wayofquality.de <ma...@wayofquality.de>
>
> /Tel:/ +49 151 23470823
>
> /Fax:/ +49 1805 006534 2114
>
> /Twitter:/ andreasgies /Skype:/ giessonic
>
> /LinkedIn:/ <http://de.linkedin.com/pub/andreas-gies/0/594/aa5/> (
> http://de.linkedin.com/pub/andreas-gies/0/594/aa5/)
>
> /Xing:/ <http://www.xing.com/profile/Andreas_Gies> (
> http://www.xing.com/profile/Andreas_Gies)
>
> /Blog:/ <http://www.wayofquality.de/index.php/en/blog> (
> http://www.wayofquality.de/index.php/en/blog)
>
> /Github:/ <https://github.com/atooni> (https://github.com/atooni)
>
> /Amtsgericht Landshut:/HRB 8352//
>
> //
>
> /Ust.-Id.:/ DE274771254
>
>
> Haftungsausschluss
>
> Diese Email kann vertrauliche und/oder rechtlich geschützte Informationen
> enthalten und ist ausschließlich für den/die benannten Adressaten bestimmt.
> Sollten Sie nicht der beabsichtigte Empfänger sein oder diese Email
> irrtümlich erhalten haben, ist es Ihnen nicht gestattet diese Mail oder
> einen Teil davon ohne unsere Erlaubnis zu verbreiten, zu kopieren, unbefugt
> weiterzuleiten oder zu behalten. Informieren Sie bitte sofort den Absender
> telefonisch oder per Email und löschen Sie diese Email und alle Kopien aus
> Ihrem System. Wir haften nicht für die Unversehrtheit von Emails, nachdem
> sie unseren Einflussbereich verlassen haben.
>
>
> Disclaimer
>
> This email may contain confidential and/or privileged information and is
> intended solely for the attention and use of the named addressee(s). If you
> are not the intended recipient, or a person responsible for delivering it
> to the intended recipient, you are not authorized to and must not disclose,
> copy, distribute, or retain this message or any part of it without our
> authority. Please contact the sender by call or reply email immediately and
> destroy all copies and the original message. We are not responsible for the
> integrity of emails after they have left our sphere of control.
>
> //
>
Re: Duplicate messages durable subscriber
Posted by Andreas Gies <an...@wayofquality.de>.
Hi Tim,
I will make some time tomorrow to try your suggestions.
A Virtual destination would simply replace a durable subscriber with a
Queue receiver.
Messages posted to a mapped topic would end up in the the underlying
queues.
Now, a consumer that wasn't connected to the peer broker would
immediately receive
the messages on the broker - that would address the problem of
temporarily unvisible
messages.
As for the duplicated delivery of messages - effectively we would have
concurrent consumers
(even though most likely onlz one of the would be connected at any one
point in time).
A message consumed on either side of the NWOB would effectively vanish
from the underlying
Queue and not be delivered again.
Due to that I was contemplating a distributed broker plugin that would
kind of synchronize the
consumption of messages of a DS within a NWOB.
However, before going that path I wanted to make sure that I am not
missing something obvious.
Thanks for the answers so far and best regards
Andreas
On 03/11/14 20:23, Tim Bain wrote:
> Andreas,
>
> Your spec added two configuration elements your previous post didn't
> mention, and I'd like to eliminate each of them in turn to see if it's
> causing/contributing to the problem.
>
> 1. Your networkConnectors are apparently multicast. Please see what
> happens if you configure them as
> static:(tcp://host:port?tcpOptions)?staticOptions, to take the multicast
> (and the broker discovery that it's presumably doing) out of the equation.
> I recently experimented with what happens when the failover is allowed to
> perform a reconnect in a broker-to-broker networkConnector, and the result
> is duplicate and/or stale subscriptions between the brokers. That behavior
> could explain what you're seeing, if multicast is similarly performing
> reconnects without notifying the static wrapper so it can recreate the
> network bridge, so let's take it out of the equation to see if the behavior
> changes. (I've never used multicast, so this might not make sense; if
> someone knows that this can't be the issue, please say so.)
> 2. I don't know how gracefully conduitSubscriptions reacts to consumers
> moving around the network of brokers; I don't believe this should be the
> problem, but if #1 doesn't produce any change in behavior, can you set
> conduitSubscriptions=false and see if anything changes?
>
> I'm not clear on how Virtual Topics will solve the problem; can you
> explain? To me this feels like a problem with broker-to-broker management
> of subscriptions made on behalf of clients (most likely duplicate
> subscriptions for a client, one from each broker, after a failover), and
> I'm not sure how a Virtual Topic would make it any better if that's the
> case. But if you know of a way that it would, that might help me to
> understand what's going on.
>
> Tim
>
> On Sat, Nov 1, 2014 at 9:50 AM, Andreas Gies <an...@wayofquality.de>
> wrote:
>
>> Hello Tim,
>>
>> thanks for your answer. It took me a bit to digest it - so my apologies for
>> the delay in my answer.
>>
>> I have come up with a test case that shows - and unfortunately confirms
>> my observations.The test case is located at [1].
>>
>> Here is the excerpt of my problem descriptions & observations:
>>
>> /**
>> * This specification shall help to investigate the duplicate delivery of
>> messages for durable subscribers
>> * within a network of brokers. The problem has been posted on the
>> ActiveMQ mailing list on Oct. 18th 2014
>> * and was described as follows:
>> *
>> * Suppose you have a network of brokers consisting of two members
>> discovering each other via multicast.
>> * The network bridge is set up using conduit subscriptions. Now assume
>> that we have a durable subscriber
>> * named "S" that connects to the network of brokers using a failover uri
>> pointing to both brokers.
>> *
>> * First, the subscriber connects to Broker A. It will consume all
>> messages published to either Broker A or B.
>> * Now the subscriber disconnects and stays offline for a bit, then it
>> reconnects to Broker B. Now it will pick
>> * up all messages that have been published while it was offline.
>> *
>> * Let's say then 10 messages are published. All is well as the subscriber
>> consumes those messages.
>> * If the subscriber then disconnects and reconnects to Broker A, these 10
>> messages will be consumed
>> * again by the reconnected subscriber.
>> *
>> * According to Tim Bain on Oct., 20th 2014 this indicates a bug rather
>> than a missing feature in ActiveMQ
>> * and this Spec shall pinpoint the behavior.
>> * *
>> * The test is based on ActiveMQ 5.10
>> *
>> * Observations:
>> * -------------
>> * Depending on when the durable subscriber is known to the members of the
>> NWOB, messages can be either left pending
>> * or delivered repeatedly (see the last 2 test cases). Message gaps can
>> occur, if the DS has only connected
>> * to one broker so far. If the DS then disconnects and after a while
>> reconnects to the other broker it wasn't
>> * connected to so far, it will not see the messages that have been
>> produced while it was offline (it will see
>> * those messages after reconnecting to broker 1).
>> *
>> * Dupilcate delivery will happen if the DS was already connected to both
>> brokers. From the broker's perspective
>> * it seems that those DS are handled as two distinct subscribers, so
>> effectively all messages that are published
>> * will eventually be delivered to both subscribers.
>> */
>>
>> I know that Virtual Topics could solve the problem - however we in the
>> middleware team are not in control of that
>> particular client application and therefor we cannot change the consumer
>> from a DS to a queue consumer.
>>
>> Can you confirm that we are indeed looking at a missing feature or a bug
>> in ActiveMQ 5.10 ? - Otherwise i would
>> need to get my thinking cap back on and see how I could solve the problem
>> without changing the client code.
>>
>> [1]
>> https://github.com/woq-blended/blended/blob/master/blended-testing/blended-testing-activemq/src/test/scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
>> <
>> https://github.com/woq-blended/blended/blob/master/blended-testing/blended-testing-activemq/src/test/scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
>> Thanks and best regards
>> Andreas
>>
>>
>>> On 20 Oct 2014, at 17:40, Tim Bain <tb...@alumni.duke.edu> wrote:
>>>
>>> If you have a network of brokers, messages on topics will be forwarded to
>>> whichever broker the consumer connects to, without duplicate delivery of
>>> any messages so long as no messages were processed by the consumer
>> without
>>> being ack'ed. If you were using queues, there's the potential for
>> messages
>>> to get stranded on a broker if no consumers are left, but this isn't
>>> possible for topics. (I'm not clear on the reason that topics can't get
>>> messages stranded even when consumers bounce between brokers, and
>>> unfortunately http://activemq.apache.org/networks-of-brokers.html
>> doesn't
>>> describe why that is.)
>>>
>>> So I think that ActiveMQ's base capabilities will do exactly what you
>> want,
>>> and if you're seeing redelivery of messages that were successfully acked
>>> when the consumer bounces to another broker, I think that would indicate
>> a
>>> bug in ActiveMQ rather than a missing feature.
>>>
>>> On Sun, Oct 19, 2014 at 6:05 PM, Noel OConnor <no...@gmail.com>
>>> wrote:
>>>
>>>> Take a look at idempotent consumers in camel. This may help you out as a
>>>> basis for your plugin if you decide to go with it.
>>>> On Oct 18, 2014 5:47 PM, "Andreas Gies" <an...@wayofquality.de>
>> wrote:
>>>>> Hi
>>>>>
>>>>> I am using ActiveMQ 5.10 in an application. So far the requirement for
>>>> the
>>>>> remote locations has been for pure store and forward capabilities,
>>>>> so that a single AMQ broker was sufficient. This has changed in a way
>>>> that
>>>>> now 2 nodes should be present in the remote location for
>>>>> resilience and load balancing. I had considered a master/configuration
>> as
>>>>> the requirement for resilience is stronger than that for load
>> balancing.
>>>>> However, the situation in those locations is that I don’t have a shared
>>>> db
>>>>> nor a shared filesystem. As far as I have understood the replicated
>>>> level db
>>>>> would require at least 3 nodes ?
>>>>>
>>>>> This is why I have chosen a network of brokers in the end, which works
>>>>> well for any Queue based communication.
>>>>>
>>>>> Now my problem is that there is one client application that is provided
>>>> by
>>>>> a 3rd party and uses durable subscriptions. It would be quite an effort
>>>>> to change that application towards using queues, so that I could
>> consider
>>>>> virtual destinations.
>>>>>
>>>>> The problem occurs two-fold:
>>>>>
>>>>> Assume a Subscriber connects to BrokerA, then disconnects and
>> reconnects
>>>>> to Broker B. It consumes messages for a while, than disconnects
>>>>> and reconnects to Broker A. All messages that have already been
>> consumed
>>>>> while it was connected to Broker B will be delivered again.
>>>>>
>>>>> My question is now whether this could be avoided by means of ActiveMQ
>>>>> alone ? - I was contemplating a broker plugin to track messages that
>>>>> have been consumed on other nodes so that I could avoid redelivering
>> them
>>>>> again.
>>>>>
>>>>> Sorry if thats a bit vague - I am fishing for ideas ….
>>>>>
>>>>>
>>>>> Thanks and best regards
>>>>> Andreas
>>
--
Andreas Gies
WoQ – Way of Quality GmbH
Geschäftsführer & CTO
/eMail:/andreas@wayofquality.de <ma...@wayofquality.de>
/Tel:/ +49 151 23470823
/Fax:/ +49 1805 006534 2114
/Twitter:/ andreasgies /Skype:/ giessonic
/LinkedIn:/ <http://de.linkedin.com/pub/andreas-gies/0/594/aa5/>
(http://de.linkedin.com/pub/andreas-gies/0/594/aa5/)
/Xing:/ <http://www.xing.com/profile/Andreas_Gies>
(http://www.xing.com/profile/Andreas_Gies)
/Blog:/ <http://www.wayofquality.de/index.php/en/blog>
(http://www.wayofquality.de/index.php/en/blog)
/Github:/ <https://github.com/atooni> (https://github.com/atooni)
/Amtsgericht Landshut:/HRB 8352//
//
/Ust.-Id.:/ DE274771254
Haftungsausschluss
Diese Email kann vertrauliche und/oder rechtlich geschützte
Informationen enthalten und ist ausschließlich für den/die benannten
Adressaten bestimmt. Sollten Sie nicht der beabsichtigte Empfänger sein
oder diese Email irrtümlich erhalten haben, ist es Ihnen nicht gestattet
diese Mail oder einen Teil davon ohne unsere Erlaubnis zu verbreiten, zu
kopieren, unbefugt weiterzuleiten oder zu behalten. Informieren Sie
bitte sofort den Absender telefonisch oder per Email und löschen Sie
diese Email und alle Kopien aus Ihrem System. Wir haften nicht für die
Unversehrtheit von Emails, nachdem sie unseren Einflussbereich verlassen
haben.
Disclaimer
This email may contain confidential and/or privileged information and is
intended solely for the attention and use of the named addressee(s). If
you are not the intended recipient, or a person responsible for
delivering it to the intended recipient, you are not authorized to and
must not disclose, copy, distribute, or retain this message or any part
of it without our authority. Please contact the sender by call or reply
email immediately and destroy all copies and the original message. We
are not responsible for the integrity of emails after they have left our
sphere of control.
//
Re: Duplicate messages durable subscriber
Posted by Tim Bain <tb...@alumni.duke.edu>.
Andreas,
Your spec added two configuration elements your previous post didn't
mention, and I'd like to eliminate each of them in turn to see if it's
causing/contributing to the problem.
1. Your networkConnectors are apparently multicast. Please see what
happens if you configure them as
static:(tcp://host:port?tcpOptions)?staticOptions, to take the multicast
(and the broker discovery that it's presumably doing) out of the equation.
I recently experimented with what happens when the failover is allowed to
perform a reconnect in a broker-to-broker networkConnector, and the result
is duplicate and/or stale subscriptions between the brokers. That behavior
could explain what you're seeing, if multicast is similarly performing
reconnects without notifying the static wrapper so it can recreate the
network bridge, so let's take it out of the equation to see if the behavior
changes. (I've never used multicast, so this might not make sense; if
someone knows that this can't be the issue, please say so.)
2. I don't know how gracefully conduitSubscriptions reacts to consumers
moving around the network of brokers; I don't believe this should be the
problem, but if #1 doesn't produce any change in behavior, can you set
conduitSubscriptions=false and see if anything changes?
I'm not clear on how Virtual Topics will solve the problem; can you
explain? To me this feels like a problem with broker-to-broker management
of subscriptions made on behalf of clients (most likely duplicate
subscriptions for a client, one from each broker, after a failover), and
I'm not sure how a Virtual Topic would make it any better if that's the
case. But if you know of a way that it would, that might help me to
understand what's going on.
Tim
On Sat, Nov 1, 2014 at 9:50 AM, Andreas Gies <an...@wayofquality.de>
wrote:
> Hello Tim,
>
> thanks for your answer. It took me a bit to digest it - so my apologies for
> the delay in my answer.
>
> I have come up with a test case that shows - and unfortunately confirms
> my observations.The test case is located at [1].
>
> Here is the excerpt of my problem descriptions & observations:
>
> /**
> * This specification shall help to investigate the duplicate delivery of
> messages for durable subscribers
> * within a network of brokers. The problem has been posted on the
> ActiveMQ mailing list on Oct. 18th 2014
> * and was described as follows:
> *
> * Suppose you have a network of brokers consisting of two members
> discovering each other via multicast.
> * The network bridge is set up using conduit subscriptions. Now assume
> that we have a durable subscriber
> * named "S" that connects to the network of brokers using a failover uri
> pointing to both brokers.
> *
> * First, the subscriber connects to Broker A. It will consume all
> messages published to either Broker A or B.
> * Now the subscriber disconnects and stays offline for a bit, then it
> reconnects to Broker B. Now it will pick
> * up all messages that have been published while it was offline.
> *
> * Let's say then 10 messages are published. All is well as the subscriber
> consumes those messages.
> * If the subscriber then disconnects and reconnects to Broker A, these 10
> messages will be consumed
> * again by the reconnected subscriber.
> *
> * According to Tim Bain on Oct., 20th 2014 this indicates a bug rather
> than a missing feature in ActiveMQ
> * and this Spec shall pinpoint the behavior.
> * *
> * The test is based on ActiveMQ 5.10
> *
> * Observations:
> * -------------
> * Depending on when the durable subscriber is known to the members of the
> NWOB, messages can be either left pending
> * or delivered repeatedly (see the last 2 test cases). Message gaps can
> occur, if the DS has only connected
> * to one broker so far. If the DS then disconnects and after a while
> reconnects to the other broker it wasn't
> * connected to so far, it will not see the messages that have been
> produced while it was offline (it will see
> * those messages after reconnecting to broker 1).
> *
> * Dupilcate delivery will happen if the DS was already connected to both
> brokers. From the broker's perspective
> * it seems that those DS are handled as two distinct subscribers, so
> effectively all messages that are published
> * will eventually be delivered to both subscribers.
> */
>
> I know that Virtual Topics could solve the problem - however we in the
> middleware team are not in control of that
> particular client application and therefor we cannot change the consumer
> from a DS to a queue consumer.
>
> Can you confirm that we are indeed looking at a missing feature or a bug
> in ActiveMQ 5.10 ? - Otherwise i would
> need to get my thinking cap back on and see how I could solve the problem
> without changing the client code.
>
> [1]
> https://github.com/woq-blended/blended/blob/master/blended-testing/blended-testing-activemq/src/test/scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
> <
> https://github.com/woq-blended/blended/blob/master/blended-testing/blended-testing-activemq/src/test/scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala
> >
>
> Thanks and best regards
> Andreas
>
>
> > On 20 Oct 2014, at 17:40, Tim Bain <tb...@alumni.duke.edu> wrote:
> >
> > If you have a network of brokers, messages on topics will be forwarded to
> > whichever broker the consumer connects to, without duplicate delivery of
> > any messages so long as no messages were processed by the consumer
> without
> > being ack'ed. If you were using queues, there's the potential for
> messages
> > to get stranded on a broker if no consumers are left, but this isn't
> > possible for topics. (I'm not clear on the reason that topics can't get
> > messages stranded even when consumers bounce between brokers, and
> > unfortunately http://activemq.apache.org/networks-of-brokers.html
> doesn't
> > describe why that is.)
> >
> > So I think that ActiveMQ's base capabilities will do exactly what you
> want,
> > and if you're seeing redelivery of messages that were successfully acked
> > when the consumer bounces to another broker, I think that would indicate
> a
> > bug in ActiveMQ rather than a missing feature.
> >
> > On Sun, Oct 19, 2014 at 6:05 PM, Noel OConnor <no...@gmail.com>
> > wrote:
> >
> >> Take a look at idempotent consumers in camel. This may help you out as a
> >> basis for your plugin if you decide to go with it.
> >> On Oct 18, 2014 5:47 PM, "Andreas Gies" <an...@wayofquality.de>
> wrote:
> >>
> >>> Hi
> >>>
> >>> I am using ActiveMQ 5.10 in an application. So far the requirement for
> >> the
> >>> remote locations has been for pure store and forward capabilities,
> >>> so that a single AMQ broker was sufficient. This has changed in a way
> >> that
> >>> now 2 nodes should be present in the remote location for
> >>> resilience and load balancing. I had considered a master/configuration
> as
> >>> the requirement for resilience is stronger than that for load
> balancing.
> >>>
> >>> However, the situation in those locations is that I don’t have a shared
> >> db
> >>> nor a shared filesystem. As far as I have understood the replicated
> >> level db
> >>> would require at least 3 nodes ?
> >>>
> >>> This is why I have chosen a network of brokers in the end, which works
> >>> well for any Queue based communication.
> >>>
> >>> Now my problem is that there is one client application that is provided
> >> by
> >>> a 3rd party and uses durable subscriptions. It would be quite an effort
> >>> to change that application towards using queues, so that I could
> consider
> >>> virtual destinations.
> >>>
> >>> The problem occurs two-fold:
> >>>
> >>> Assume a Subscriber connects to BrokerA, then disconnects and
> reconnects
> >>> to Broker B. It consumes messages for a while, than disconnects
> >>> and reconnects to Broker A. All messages that have already been
> consumed
> >>> while it was connected to Broker B will be delivered again.
> >>>
> >>> My question is now whether this could be avoided by means of ActiveMQ
> >>> alone ? - I was contemplating a broker plugin to track messages that
> >>> have been consumed on other nodes so that I could avoid redelivering
> them
> >>> again.
> >>>
> >>> Sorry if thats a bit vague - I am fishing for ideas ….
> >>>
> >>>
> >>> Thanks and best regards
> >>> Andreas
> >>
>
>
Re: Duplicate messages durable subscriber
Posted by Andreas Gies <an...@wayofquality.de>.
Hello Tim,
thanks for your answer. It took me a bit to digest it - so my apologies for
the delay in my answer.
I have come up with a test case that shows - and unfortunately confirms
my observations.The test case is located at [1].
Here is the excerpt of my problem descriptions & observations:
/**
* This specification shall help to investigate the duplicate delivery of messages for durable subscribers
* within a network of brokers. The problem has been posted on the ActiveMQ mailing list on Oct. 18th 2014
* and was described as follows:
*
* Suppose you have a network of brokers consisting of two members discovering each other via multicast.
* The network bridge is set up using conduit subscriptions. Now assume that we have a durable subscriber
* named "S" that connects to the network of brokers using a failover uri pointing to both brokers.
*
* First, the subscriber connects to Broker A. It will consume all messages published to either Broker A or B.
* Now the subscriber disconnects and stays offline for a bit, then it reconnects to Broker B. Now it will pick
* up all messages that have been published while it was offline.
*
* Let's say then 10 messages are published. All is well as the subscriber consumes those messages.
* If the subscriber then disconnects and reconnects to Broker A, these 10 messages will be consumed
* again by the reconnected subscriber.
*
* According to Tim Bain on Oct., 20th 2014 this indicates a bug rather than a missing feature in ActiveMQ
* and this Spec shall pinpoint the behavior.
* *
* The test is based on ActiveMQ 5.10
*
* Observations:
* -------------
* Depending on when the durable subscriber is known to the members of the NWOB, messages can be either left pending
* or delivered repeatedly (see the last 2 test cases). Message gaps can occur, if the DS has only connected
* to one broker so far. If the DS then disconnects and after a while reconnects to the other broker it wasn't
* connected to so far, it will not see the messages that have been produced while it was offline (it will see
* those messages after reconnecting to broker 1).
*
* Dupilcate delivery will happen if the DS was already connected to both brokers. From the broker's perspective
* it seems that those DS are handled as two distinct subscribers, so effectively all messages that are published
* will eventually be delivered to both subscribers.
*/
I know that Virtual Topics could solve the problem - however we in the middleware team are not in control of that
particular client application and therefor we cannot change the consumer from a DS to a queue consumer.
Can you confirm that we are indeed looking at a missing feature or a bug in ActiveMQ 5.10 ? - Otherwise i would
need to get my thinking cap back on and see how I could solve the problem without changing the client code.
[1] https://github.com/woq-blended/blended/blob/master/blended-testing/blended-testing-activemq/src/test/scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala <https://github.com/woq-blended/blended/blob/master/blended-testing/blended-testing-activemq/src/test/scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala>
Thanks and best regards
Andreas
> On 20 Oct 2014, at 17:40, Tim Bain <tb...@alumni.duke.edu> wrote:
>
> If you have a network of brokers, messages on topics will be forwarded to
> whichever broker the consumer connects to, without duplicate delivery of
> any messages so long as no messages were processed by the consumer without
> being ack'ed. If you were using queues, there's the potential for messages
> to get stranded on a broker if no consumers are left, but this isn't
> possible for topics. (I'm not clear on the reason that topics can't get
> messages stranded even when consumers bounce between brokers, and
> unfortunately http://activemq.apache.org/networks-of-brokers.html doesn't
> describe why that is.)
>
> So I think that ActiveMQ's base capabilities will do exactly what you want,
> and if you're seeing redelivery of messages that were successfully acked
> when the consumer bounces to another broker, I think that would indicate a
> bug in ActiveMQ rather than a missing feature.
>
> On Sun, Oct 19, 2014 at 6:05 PM, Noel OConnor <no...@gmail.com>
> wrote:
>
>> Take a look at idempotent consumers in camel. This may help you out as a
>> basis for your plugin if you decide to go with it.
>> On Oct 18, 2014 5:47 PM, "Andreas Gies" <an...@wayofquality.de> wrote:
>>
>>> Hi
>>>
>>> I am using ActiveMQ 5.10 in an application. So far the requirement for
>> the
>>> remote locations has been for pure store and forward capabilities,
>>> so that a single AMQ broker was sufficient. This has changed in a way
>> that
>>> now 2 nodes should be present in the remote location for
>>> resilience and load balancing. I had considered a master/configuration as
>>> the requirement for resilience is stronger than that for load balancing.
>>>
>>> However, the situation in those locations is that I don’t have a shared
>> db
>>> nor a shared filesystem. As far as I have understood the replicated
>> level db
>>> would require at least 3 nodes ?
>>>
>>> This is why I have chosen a network of brokers in the end, which works
>>> well for any Queue based communication.
>>>
>>> Now my problem is that there is one client application that is provided
>> by
>>> a 3rd party and uses durable subscriptions. It would be quite an effort
>>> to change that application towards using queues, so that I could consider
>>> virtual destinations.
>>>
>>> The problem occurs two-fold:
>>>
>>> Assume a Subscriber connects to BrokerA, then disconnects and reconnects
>>> to Broker B. It consumes messages for a while, than disconnects
>>> and reconnects to Broker A. All messages that have already been consumed
>>> while it was connected to Broker B will be delivered again.
>>>
>>> My question is now whether this could be avoided by means of ActiveMQ
>>> alone ? - I was contemplating a broker plugin to track messages that
>>> have been consumed on other nodes so that I could avoid redelivering them
>>> again.
>>>
>>> Sorry if thats a bit vague - I am fishing for ideas ….
>>>
>>>
>>> Thanks and best regards
>>> Andreas
>>
Re: Duplicate messages durable subscriber
Posted by Tim Bain <tb...@alumni.duke.edu>.
If you have a network of brokers, messages on topics will be forwarded to
whichever broker the consumer connects to, without duplicate delivery of
any messages so long as no messages were processed by the consumer without
being ack'ed. If you were using queues, there's the potential for messages
to get stranded on a broker if no consumers are left, but this isn't
possible for topics. (I'm not clear on the reason that topics can't get
messages stranded even when consumers bounce between brokers, and
unfortunately http://activemq.apache.org/networks-of-brokers.html doesn't
describe why that is.)
So I think that ActiveMQ's base capabilities will do exactly what you want,
and if you're seeing redelivery of messages that were successfully acked
when the consumer bounces to another broker, I think that would indicate a
bug in ActiveMQ rather than a missing feature.
On Sun, Oct 19, 2014 at 6:05 PM, Noel OConnor <no...@gmail.com>
wrote:
> Take a look at idempotent consumers in camel. This may help you out as a
> basis for your plugin if you decide to go with it.
> On Oct 18, 2014 5:47 PM, "Andreas Gies" <an...@wayofquality.de> wrote:
>
> > Hi
> >
> > I am using ActiveMQ 5.10 in an application. So far the requirement for
> the
> > remote locations has been for pure store and forward capabilities,
> > so that a single AMQ broker was sufficient. This has changed in a way
> that
> > now 2 nodes should be present in the remote location for
> > resilience and load balancing. I had considered a master/configuration as
> > the requirement for resilience is stronger than that for load balancing.
> >
> > However, the situation in those locations is that I don’t have a shared
> db
> > nor a shared filesystem. As far as I have understood the replicated
> level db
> > would require at least 3 nodes ?
> >
> > This is why I have chosen a network of brokers in the end, which works
> > well for any Queue based communication.
> >
> > Now my problem is that there is one client application that is provided
> by
> > a 3rd party and uses durable subscriptions. It would be quite an effort
> > to change that application towards using queues, so that I could consider
> > virtual destinations.
> >
> > The problem occurs two-fold:
> >
> > Assume a Subscriber connects to BrokerA, then disconnects and reconnects
> > to Broker B. It consumes messages for a while, than disconnects
> > and reconnects to Broker A. All messages that have already been consumed
> > while it was connected to Broker B will be delivered again.
> >
> > My question is now whether this could be avoided by means of ActiveMQ
> > alone ? - I was contemplating a broker plugin to track messages that
> > have been consumed on other nodes so that I could avoid redelivering them
> > again.
> >
> > Sorry if thats a bit vague - I am fishing for ideas ….
> >
> >
> > Thanks and best regards
> > Andreas
>
Re: Duplicate messages durable subscriber
Posted by Noel OConnor <no...@gmail.com>.
Take a look at idempotent consumers in camel. This may help you out as a
basis for your plugin if you decide to go with it.
On Oct 18, 2014 5:47 PM, "Andreas Gies" <an...@wayofquality.de> wrote:
> Hi
>
> I am using ActiveMQ 5.10 in an application. So far the requirement for the
> remote locations has been for pure store and forward capabilities,
> so that a single AMQ broker was sufficient. This has changed in a way that
> now 2 nodes should be present in the remote location for
> resilience and load balancing. I had considered a master/configuration as
> the requirement for resilience is stronger than that for load balancing.
>
> However, the situation in those locations is that I don’t have a shared db
> nor a shared filesystem. As far as I have understood the replicated level db
> would require at least 3 nodes ?
>
> This is why I have chosen a network of brokers in the end, which works
> well for any Queue based communication.
>
> Now my problem is that there is one client application that is provided by
> a 3rd party and uses durable subscriptions. It would be quite an effort
> to change that application towards using queues, so that I could consider
> virtual destinations.
>
> The problem occurs two-fold:
>
> Assume a Subscriber connects to BrokerA, then disconnects and reconnects
> to Broker B. It consumes messages for a while, than disconnects
> and reconnects to Broker A. All messages that have already been consumed
> while it was connected to Broker B will be delivered again.
>
> My question is now whether this could be avoided by means of ActiveMQ
> alone ? - I was contemplating a broker plugin to track messages that
> have been consumed on other nodes so that I could avoid redelivering them
> again.
>
> Sorry if thats a bit vague - I am fishing for ideas ….
>
>
> Thanks and best regards
> Andreas