You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Ramit Arora <ra...@gmail.com> on 2008/03/28 10:42:56 UTC
Consuming persistent messages from a non-durable subscriber
Hi All,
I have a situation where I have to consume persisent messages from many
non-durable subscribers. My publishers push persistent messages to a topic,
which are consumed asynchronously by non-durable subscribers. I am using the
AMQ Persistence Adapter & CLIENT_ACKNOWLEDGE. If I kill a non-durable
subscriber & bring it up again, some messages are lost, which is expected.
But consider the following case:
1. Start a publisher publishing persistent messages to a topic.
2. Start a non-durable subscriber which is slow compared to the publisher.
3. Kill the broker when 100 messages have been published but only 50 have
been consumed.
4. Since the publisher & subscriber are failover clients, they wait for the
broker to come up again.
5. Restart the broker.
6. Try to get the remaining 50 messages to the non-durable subscriber.
Step 6 does not work in both 5.0 & the latest 5.1 snapshot. Shouldn't the
remaining 50 messages be delivered? The JMS spec says:
"A JMS provider must deliver a PERSISTENT message once-and-only-once. This
means a JMS provider failure must not cause it to be lost, and it must not
deliver it twice. PERSISTENT (once-and-only-once) and NON_PERSISTENT
(at-most-once)
message delivery are a way for a JMS client to select between delivery
techniques that may lose a messages if a JMS provider dies and those which
take extra effort to insure that messages can survive such a failure. There
is
typically a performance/reliability trade-off implied by this choice. When a
client selects the NON_PERSISTENT delivery mode, it is indicating that it
values performance over reliability; a selection of PERSISTENT reverses the
requested trade-off."
Queues donot suffer from this problem. Please let me know if I am missing
something.
Thanks,
Ramit
--
View this message in context: http://www.nabble.com/Consuming-persistent-messages-from-a-non-durable-subscriber-tp16348721s2354p16348721.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consuming persistent messages from a non-durable subscriber
Posted by ttmdev <jo...@ttmsolutions.com>.
I think I would be a bit more inclined to look into virtual topics. Primarily
because they're easier to work with - vs durable subscriptions - and they
also offer the functionality provided by queues. So you get the best of
both worlds ;)
Joe
Ramit Arora wrote:
>
> Hi Joe,
> Thanks for your time. My application requires a hub & spoke architecture
> with around 100 publishers & 500 consumers at peak load. All publishers
> would be pushing messages to different topics. One consumer would be
> durable as it needs to collect all messages. The rest would be online
> sporadically (actually they are end-user GUIs & I intend to stream
> messages in real time to them). So I intended to make them non-durable.
> What I want is that if my broker goes down, all my consumers should
> failover to the slave & not loose any messages. I am using NFS based
> master/slave. I am observing that my non-durable subscribers are losing
> messages on failover if they are lagging behind the producers.
> I am concerned about the cost of using 500 durable subscribers. I'll be
> considering the use of retroactive consumers & virtual destinations. What
> would be the recommended thing to do in my case?
> And BTW, the user guide is awesome!
>
> Thanks,
> Ramit
>
> ttmdev wrote:
>>
>> Your non-durable subscriber will only get those messages that were
>> published while it is active. When you bounce the broker it does not
>> retain the identity of your non-durable subscriber. So when the broker
>> comes back up, it doesn't consider your non-durable subscriber to have
>> been active when the persisted messages were published.
>>
>> Can you use durable subscribers? If not, look into retroactive consumers
>> or better yet, virtual topics. With the latter you get the functionality
>> of both a topic and queue and you don't have to worry about setting up
>> unique identifiers as you would have to when using durable subscriptions.
>>
>> http://activemq.apache.org/virtual-destinations.html
>>
>> http://activemq.apache.org/retroactive-consumer.html
>>
>> Joe
>> Goto www.ttmsolutions.com for a free ActiveMQ user guide
>>
>>
>>
>> Ramit Arora wrote:
>>>
>>> Hi All,
>>> I have a situation where I have to consume persisent messages from many
>>> non-durable subscribers. My publishers push persistent messages to a
>>> topic, which are consumed asynchronously by non-durable subscribers. I
>>> am using the AMQ Persistence Adapter & CLIENT_ACKNOWLEDGE. If I kill a
>>> non-durable subscriber & bring it up again, some messages are lost,
>>> which is expected. But consider the following case:
>>> 1. Start a publisher publishing persistent messages to a topic.
>>> 2. Start a non-durable subscriber which is slow compared to the
>>> publisher.
>>> 3. Kill the broker when 100 messages have been published but only 50
>>> have been consumed.
>>> 4. Since the publisher & subscriber are failover clients, they wait for
>>> the broker to come up again.
>>> 5. Restart the broker.
>>> 6. Try to get the remaining 50 messages to the non-durable subscriber.
>>>
>>> Step 6 does not work in both 5.0 & the latest 5.1 snapshot. Shouldn't
>>> the remaining 50 messages be delivered? The JMS spec says:
>>> "A JMS provider must deliver a PERSISTENT message once-and-only-once.
>>> This
>>> means a JMS provider failure must not cause it to be lost, and it must
>>> not
>>> deliver it twice. PERSISTENT (once-and-only-once) and NON_PERSISTENT
>>> (at-most-once)
>>> message delivery are a way for a JMS client to select between delivery
>>> techniques that may lose a messages if a JMS provider dies and those
>>> which
>>> take extra effort to insure that messages can survive such a failure.
>>> There is
>>> typically a performance/reliability trade-off implied by this choice.
>>> When a
>>> client selects the NON_PERSISTENT delivery mode, it is indicating that
>>> it
>>> values performance over reliability; a selection of PERSISTENT reverses
>>> the
>>> requested trade-off."
>>>
>>> Queues donot suffer from this problem. Please let me know if I am
>>> missing something.
>>>
>>> Thanks,
>>> Ramit
>>>
>>>
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Consuming-persistent-messages-from-a-non-durable-subscriber-tp16348721s2354p16356200.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consuming persistent messages from a non-durable subscriber
Posted by Ramit Arora <ra...@gmail.com>.
Hi Joe,
Thanks for your time. My application requires a hub & spoke architecture
with around 100 publishers & 500 consumers at peak load. All publishers
would be pushing messages to different topics. One consumer would be durable
as it needs to collect all messages. The rest would be online sporadically
(actually they are end-user GUIs & I intend to stream messages in real time
to them). So I intended to make them non-durable. What I want is that if my
broker goes down, all my consumers should failover to the slave & not loose
any messages. I am using NFS based master/slave. I am observing that my
non-durable subscribers are losing messages on failover if they are lagging
behind the producers.
I am concerned about the cost of using 500 durable subscribers. I'll be
considering the use of retroactive consumers & virtual destinations. What
would be the recommended thing to do in my case?
And BTW, the user guide is awesome!
Thanks,
Ramit
ttmdev wrote:
>
> Your non-durable subscriber will only get those messages that were
> published while it is active. When you bounce the broker it does not
> retain the identity of your non-durable subscriber. So when the broker
> comes back up, it doesn't consider your non-durable subscriber to have
> been active when the persisted messages were published.
>
> Can you use durable subscribers? If not, look into retroactive consumers
> or better yet, virtual topics. With the latter you get the functionality
> of both a topic and queue and you don't have to worry about setting up
> unique identifiers as you would have to when using durable subscriptions.
>
> http://activemq.apache.org/virtual-destinations.html
>
> http://activemq.apache.org/retroactive-consumer.html
>
> Joe
> Goto www.ttmsolutions.com for a free ActiveMQ user guide
>
>
>
> Ramit Arora wrote:
>>
>> Hi All,
>> I have a situation where I have to consume persisent messages from many
>> non-durable subscribers. My publishers push persistent messages to a
>> topic, which are consumed asynchronously by non-durable subscribers. I am
>> using the AMQ Persistence Adapter & CLIENT_ACKNOWLEDGE. If I kill a
>> non-durable subscriber & bring it up again, some messages are lost, which
>> is expected. But consider the following case:
>> 1. Start a publisher publishing persistent messages to a topic.
>> 2. Start a non-durable subscriber which is slow compared to the
>> publisher.
>> 3. Kill the broker when 100 messages have been published but only 50 have
>> been consumed.
>> 4. Since the publisher & subscriber are failover clients, they wait for
>> the broker to come up again.
>> 5. Restart the broker.
>> 6. Try to get the remaining 50 messages to the non-durable subscriber.
>>
>> Step 6 does not work in both 5.0 & the latest 5.1 snapshot. Shouldn't the
>> remaining 50 messages be delivered? The JMS spec says:
>> "A JMS provider must deliver a PERSISTENT message once-and-only-once.
>> This
>> means a JMS provider failure must not cause it to be lost, and it must
>> not
>> deliver it twice. PERSISTENT (once-and-only-once) and NON_PERSISTENT
>> (at-most-once)
>> message delivery are a way for a JMS client to select between delivery
>> techniques that may lose a messages if a JMS provider dies and those
>> which
>> take extra effort to insure that messages can survive such a failure.
>> There is
>> typically a performance/reliability trade-off implied by this choice.
>> When a
>> client selects the NON_PERSISTENT delivery mode, it is indicating that it
>> values performance over reliability; a selection of PERSISTENT reverses
>> the
>> requested trade-off."
>>
>> Queues donot suffer from this problem. Please let me know if I am missing
>> something.
>>
>> Thanks,
>> Ramit
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Consuming-persistent-messages-from-a-non-durable-subscriber-tp16348721s2354p16350322.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consuming persistent messages from a non-durable subscriber
Posted by ttmdev <jo...@ttmsolutions.com>.
Your non-durable subscriber will only get those messages that were published
while it is active. When you bounce the broker it does not retain the
identity of your non-durable subscriber. So when the broker comes back up,
it doesn't consider your non-durable subscriber to have been active when the
persisted messages were published.
Can you use durable subscribers? If not, look into retroactive consumers or
better yet, virtual topics. With the latter you get the functionality of
both a topic and queue and you don't have to worry about setting up unique
identifiers as you would have to when using durable subscriptions.
http://activemq.apache.org/virtual-destinations.html
http://activemq.apache.org/retroactive-consumer.html
Joe
Goto www.ttmsolutions.com for a free ActiveMQ user guide
Ramit Arora wrote:
>
> Hi All,
> I have a situation where I have to consume persisent messages from many
> non-durable subscribers. My publishers push persistent messages to a
> topic, which are consumed asynchronously by non-durable subscribers. I am
> using the AMQ Persistence Adapter & CLIENT_ACKNOWLEDGE. If I kill a
> non-durable subscriber & bring it up again, some messages are lost, which
> is expected. But consider the following case:
> 1. Start a publisher publishing persistent messages to a topic.
> 2. Start a non-durable subscriber which is slow compared to the publisher.
> 3. Kill the broker when 100 messages have been published but only 50 have
> been consumed.
> 4. Since the publisher & subscriber are failover clients, they wait for
> the broker to come up again.
> 5. Restart the broker.
> 6. Try to get the remaining 50 messages to the non-durable subscriber.
>
> Step 6 does not work in both 5.0 & the latest 5.1 snapshot. Shouldn't the
> remaining 50 messages be delivered? The JMS spec says:
> "A JMS provider must deliver a PERSISTENT message once-and-only-once. This
> means a JMS provider failure must not cause it to be lost, and it must not
> deliver it twice. PERSISTENT (once-and-only-once) and NON_PERSISTENT
> (at-most-once)
> message delivery are a way for a JMS client to select between delivery
> techniques that may lose a messages if a JMS provider dies and those which
> take extra effort to insure that messages can survive such a failure.
> There is
> typically a performance/reliability trade-off implied by this choice. When
> a
> client selects the NON_PERSISTENT delivery mode, it is indicating that it
> values performance over reliability; a selection of PERSISTENT reverses
> the
> requested trade-off."
>
> Queues donot suffer from this problem. Please let me know if I am missing
> something.
>
> Thanks,
> Ramit
>
>
>
--
View this message in context: http://www.nabble.com/Consuming-persistent-messages-from-a-non-durable-subscriber-tp16348721s2354p16349812.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consuming persistent messages from a non-durable subscriber
Posted by Rob Davies <ra...@gmail.com>.
On 28 Mar 2008, at 09:42, Ramit Arora wrote:
>
> Hi All,
> I have a situation where I have to consume persisent messages from
> many
> non-durable subscribers. My publishers push persistent messages to a
> topic,
> which are consumed asynchronously by non-durable subscribers. I am
> using the
> AMQ Persistence Adapter & CLIENT_ACKNOWLEDGE. If I kill a non-durable
> subscriber & bring it up again, some messages are lost, which is
> expected.
> But consider the following case:
> 1. Start a publisher publishing persistent messages to a topic.
> 2. Start a non-durable subscriber which is slow compared to the
> publisher.
> 3. Kill the broker when 100 messages have been published but only 50
> have
> been consumed.
> 4. Since the publisher & subscriber are failover clients, they wait
> for the
> broker to come up again.
> 5. Restart the broker.
> 6. Try to get the remaining 50 messages to the non-durable subscriber.
>
> Step 6 does not work in both 5.0 & the latest 5.1 snapshot.
> Shouldn't the
> remaining 50 messages be delivered? The JMS spec says:
> "A JMS provider must deliver a PERSISTENT message once-and-only-
> once. This
> means a JMS provider failure must not cause it to be lost, and it
> must not
> deliver it twice. PERSISTENT (once-and-only-once) and NON_PERSISTENT
> (at-most-once)
> message delivery are a way for a JMS client to select between delivery
> techniques that may lose a messages if a JMS provider dies and those
> which
> take extra effort to insure that messages can survive such a
> failure. There
> is
> typically a performance/reliability trade-off implied by this
> choice. When a
> client selects the NON_PERSISTENT delivery mode, it is indicating
> that it
> values performance over reliability; a selection of PERSISTENT
> reverses the
> requested trade-off."
>
> Queues donot suffer from this problem. Please let me know if I am
> missing
> something.
>
> Thanks,
> Ramit
>
>
> --
> View this message in context: http://www.nabble.com/Consuming-persistent-messages-from-a-non-durable-subscriber-tp16348721s2354p16348721.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Message delivery by default is persistent - but there is a fundamental
difference in the contract between queues and topics:
Queue messages are always persisted
Topic messages are persisted if there exists a durable subscriber for
that topic.
You can get around this by using virtual destinations - see here: http://activemq.apache.org/virtual-destinations.html
cheers,
Rob