You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Sven Vintges <sv...@vintges.org> on 2010/11/01 13:16:54 UTC
Re: Virtual Topic and missed messages gone missing withselectorAware=true
Perhaps a work-around would be to create a garbage-service. Ie. a
services that pops off all messages not needed by the consumer to
/dev/null? Though you probably don't know which messages are not needed
by the subscriber since you don't know which subscribers you will
have... Perhaps you could create an algorithm that does something like
"if the consumer is up and the message is older than xxx time" pop it
off?
On Mon, 1 Nov 2010 10:25:40 +0000, "Maciej Rakowicz"
<ma...@homeaway.co.uk> wrote:
> in case of selectorAware=true and consumer down, what happens to
> those messages? are they discarded? Is there anything sent to one of
> the advisories? I can't seem to find it anywhere.
>
> On 1 Nov 2010, at 09:48, Maciej Rakowicz wrote:
>
>> I guess, only if we keep downtimes short enough so 'matched'
>> messages don't expire and big enough maxPageSize or something to keep
>> matched messages consumed.
>>
>> Does anyone know if this (i.e. AMQ-3004) is going to be worked on in
>> the near future?
>>
>> m.
>>
>> On 1 Nov 2010, at 09:32, Gary Tully wrote:
>>
>>> Message expiry may help in the short term, set a time to live on
>>> the
>>> messages such that the built up unmatched messages expire and are
>>> removed.
>>>
>>> On 1 November 2010 09:20, Maciej Rakowicz <ma...@homeaway.co.uk>
>>> wrote:
>>>> All,
>>>>
>>>> amq 5.4.0. jdk 1.6
>>>>
>>>> Scenario:
>>>> broker persistence=true, selectorAware=false
>>>> Virtual Topic with say one queue. There is a consumer A connected
>>>> to that
>>>> queue with a selector. All works fine. Consumer A dies, messages
>>>> pile up,
>>>> consumer A starts back up, missed messages are redelivered. You
>>>> can easily
>>>> verify that queue receives all posted messages while consumer's
>>>> down.
>>>> Now, adding consumer B, since selectorAware=false and consumer B
>>>> uses an
>>>> exclusive selector messages sent to consumer A are not consumed by
>>>> consumer
>>>> B. All good save the fact that all unmatched messages end up
>>>> polluting
>>>> consumer's B queue. All according to the documentation.
>>>> Flipping selectorAware to true solves one problem but introduces
>>>> another
>>>> (way more important in my setup). Unmatched messages won't pile up
>>>> on
>>>> consumer's B queue which is fine. However, if consumer A dies they
>>>> are not
>>>> sent to consumer's A queue (disappearing after reaching the topic
>>>> - I dunno
>>>> where they go) hence disabling any missed message delivery on
>>>> consumer's A
>>>> startup.
>>>>
>>>> I there any way around it? I really need to handle all missed
>>>> messages with
>>>> no exception. At the same time I cannot allow stacking up messages
>>>> in each
>>>> and every queue although they are not matched.
>>>>
>>>> Btw. I do not have a list of consumers in advance, the
>>>> subscription is fully
>>>> dynamic, although since broker runs persistence, once subscribed I
>>>> do have
>>>> an idea of who's subscribed even if connection is currently down.
>>>>
>>>> I would appreciate any hints.
>>>>
>>>> mac
>>>
>>>
>>>
>>> -- http://blog.garytully.com
>>> http://fusesource.com
>>
Re: Virtual Topic and missed messages gone missing with selectorAware=true
Posted by Roelof Naude <ro...@gmail.com>.
hi maciej,
that sounds interesting. would you mind sharing the filter and changes
to SelectorAwareVirtualTopicInterceptor?
On 11/02/2010 03:48 PM, Maciej Rakowicz wrote:
> Ok, i do have a somewhat contrived workaround. Basically, I decided to
> cache selector string of any queue subscription (well vtopic queue)
> that I intercept (BrokerFilter). Then I have modified
> org
> .apache
> .activemq.broker.region.virtual.SelectorAwareVirtualTopicInterceptor
> so that if there is no subscription for a destination (here we talk
> about queue rather than vtopic again) it tries looking it up in the
> cache - well, just the selector string as it's all I need, plus due to
> the way cache is distributed it has to be easily serializable. It
> compiles the selector and tries to match it against
> MessageEvaluationContext. Of course this works with several
> assumptions in mind. First, I need this consumer to be connected at
> least once (that's when I can cache its selector). Secondly, selector
> <-> queue is 1-to-1 relation. That's perfectly acceptable in my setup.
>
> m.
>
> On 1 Nov 2010, at 12:16, Sven Vintges wrote:
>
>> Perhaps a work-around would be to create a garbage-service. Ie. a
>> services that pops off all messages not needed by the consumer to /
>> dev/null? Though you probably don't know which messages are not
>> needed by the subscriber since you don't know which subscribers you
>> will have... Perhaps you could create an algorithm that does
>> something like "if the consumer is up and the message is older than
>> xxx time" pop it off?
>>
>> On Mon, 1 Nov 2010 10:25:40 +0000, "Maciej Rakowicz"<maciek@homeaway.co.uk
>>> wrote:
>>> in case of selectorAware=true and consumer down, what happens to
>>> those messages? are they discarded? Is there anything sent to one of
>>> the advisories? I can't seem to find it anywhere.
>>>
>>> On 1 Nov 2010, at 09:48, Maciej Rakowicz wrote:
>>>
>>>> I guess, only if we keep downtimes short enough so 'matched'
>>>> messages don't expire and big enough maxPageSize or something to
>>>> keep matched messages consumed.
>>>>
>>>> Does anyone know if this (i.e. AMQ-3004) is going to be worked on
>>>> in the near future?
>>>>
>>>> m.
>>>>
>>>> On 1 Nov 2010, at 09:32, Gary Tully wrote:
>>>>
>>>>> Message expiry may help in the short term, set a time to live on
>>>>> the
>>>>> messages such that the built up unmatched messages expire and are
>>>>> removed.
>>>>>
>>>>> On 1 November 2010 09:20, Maciej Rakowicz
>>>>> <ma...@homeaway.co.uk> wrote:
>>>>>> All,
>>>>>>
>>>>>> amq 5.4.0. jdk 1.6
>>>>>>
>>>>>> Scenario:
>>>>>> broker persistence=true, selectorAware=false
>>>>>> Virtual Topic with say one queue. There is a consumer A
>>>>>> connected to that
>>>>>> queue with a selector. All works fine. Consumer A dies,
>>>>>> messages pile up,
>>>>>> consumer A starts back up, missed messages are redelivered. You
>>>>>> can easily
>>>>>> verify that queue receives all posted messages while consumer's
>>>>>> down.
>>>>>> Now, adding consumer B, since selectorAware=false and consumer
>>>>>> B uses an
>>>>>> exclusive selector messages sent to consumer A are not consumed
>>>>>> by consumer
>>>>>> B. All good save the fact that all unmatched messages end up
>>>>>> polluting
>>>>>> consumer's B queue. All according to the documentation.
>>>>>> Flipping selectorAware to true solves one problem but
>>>>>> introduces another
>>>>>> (way more important in my setup). Unmatched messages won't pile
>>>>>> up on
>>>>>> consumer's B queue which is fine. However, if consumer A dies
>>>>>> they are not
>>>>>> sent to consumer's A queue (disappearing after reaching the
>>>>>> topic - I dunno
>>>>>> where they go) hence disabling any missed message delivery on
>>>>>> consumer's A
>>>>>> startup.
>>>>>>
>>>>>> I there any way around it? I really need to handle all missed
>>>>>> messages with
>>>>>> no exception. At the same time I cannot allow stacking up
>>>>>> messages in each
>>>>>> and every queue although they are not matched.
>>>>>>
>>>>>> Btw. I do not have a list of consumers in advance, the
>>>>>> subscription is fully
>>>>>> dynamic, although since broker runs persistence, once subscribed
>>>>>> I do have
>>>>>> an idea of who's subscribed even if connection is currently down.
>>>>>>
>>>>>> I would appreciate any hints.
>>>>>>
>>>>>> mac
>>>>>
>>>>>
>>>>>
>>>>> -- http://blog.garytully.com
>>>>> http://fusesource.com
>>>>
>>
>
Re: Virtual Topic and missed messages gone missing with selectorAware=true
Posted by Maciej Rakowicz <ma...@homeaway.co.uk>.
Ok, i do have a somewhat contrived workaround. Basically, I decided to
cache selector string of any queue subscription (well vtopic queue)
that I intercept (BrokerFilter). Then I have modified
org
.apache
.activemq.broker.region.virtual.SelectorAwareVirtualTopicInterceptor
so that if there is no subscription for a destination (here we talk
about queue rather than vtopic again) it tries looking it up in the
cache - well, just the selector string as it's all I need, plus due to
the way cache is distributed it has to be easily serializable. It
compiles the selector and tries to match it against
MessageEvaluationContext. Of course this works with several
assumptions in mind. First, I need this consumer to be connected at
least once (that's when I can cache its selector). Secondly, selector
<-> queue is 1-to-1 relation. That's perfectly acceptable in my setup.
m.
On 1 Nov 2010, at 12:16, Sven Vintges wrote:
> Perhaps a work-around would be to create a garbage-service. Ie. a
> services that pops off all messages not needed by the consumer to /
> dev/null? Though you probably don't know which messages are not
> needed by the subscriber since you don't know which subscribers you
> will have... Perhaps you could create an algorithm that does
> something like "if the consumer is up and the message is older than
> xxx time" pop it off?
>
> On Mon, 1 Nov 2010 10:25:40 +0000, "Maciej Rakowicz" <maciek@homeaway.co.uk
> > wrote:
>> in case of selectorAware=true and consumer down, what happens to
>> those messages? are they discarded? Is there anything sent to one of
>> the advisories? I can't seem to find it anywhere.
>>
>> On 1 Nov 2010, at 09:48, Maciej Rakowicz wrote:
>>
>>> I guess, only if we keep downtimes short enough so 'matched'
>>> messages don't expire and big enough maxPageSize or something to
>>> keep matched messages consumed.
>>>
>>> Does anyone know if this (i.e. AMQ-3004) is going to be worked on
>>> in the near future?
>>>
>>> m.
>>>
>>> On 1 Nov 2010, at 09:32, Gary Tully wrote:
>>>
>>>> Message expiry may help in the short term, set a time to live on
>>>> the
>>>> messages such that the built up unmatched messages expire and are
>>>> removed.
>>>>
>>>> On 1 November 2010 09:20, Maciej Rakowicz
>>>> <ma...@homeaway.co.uk> wrote:
>>>>> All,
>>>>>
>>>>> amq 5.4.0. jdk 1.6
>>>>>
>>>>> Scenario:
>>>>> broker persistence=true, selectorAware=false
>>>>> Virtual Topic with say one queue. There is a consumer A
>>>>> connected to that
>>>>> queue with a selector. All works fine. Consumer A dies,
>>>>> messages pile up,
>>>>> consumer A starts back up, missed messages are redelivered. You
>>>>> can easily
>>>>> verify that queue receives all posted messages while consumer's
>>>>> down.
>>>>> Now, adding consumer B, since selectorAware=false and consumer
>>>>> B uses an
>>>>> exclusive selector messages sent to consumer A are not consumed
>>>>> by consumer
>>>>> B. All good save the fact that all unmatched messages end up
>>>>> polluting
>>>>> consumer's B queue. All according to the documentation.
>>>>> Flipping selectorAware to true solves one problem but
>>>>> introduces another
>>>>> (way more important in my setup). Unmatched messages won't pile
>>>>> up on
>>>>> consumer's B queue which is fine. However, if consumer A dies
>>>>> they are not
>>>>> sent to consumer's A queue (disappearing after reaching the
>>>>> topic - I dunno
>>>>> where they go) hence disabling any missed message delivery on
>>>>> consumer's A
>>>>> startup.
>>>>>
>>>>> I there any way around it? I really need to handle all missed
>>>>> messages with
>>>>> no exception. At the same time I cannot allow stacking up
>>>>> messages in each
>>>>> and every queue although they are not matched.
>>>>>
>>>>> Btw. I do not have a list of consumers in advance, the
>>>>> subscription is fully
>>>>> dynamic, although since broker runs persistence, once subscribed
>>>>> I do have
>>>>> an idea of who's subscribed even if connection is currently down.
>>>>>
>>>>> I would appreciate any hints.
>>>>>
>>>>> mac
>>>>
>>>>
>>>>
>>>> -- http://blog.garytully.com
>>>> http://fusesource.com
>>>
>