You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chris Hampson <ch...@hampson.dev> on 2019/07/19 13:01:01 UTC
Question regarding pattern subscriptions
Hi all,
Tl;dr: I want to set up a subscription based on patterns that will match non-existent topics and receive all messages that match (e.g. test.*), missing first message to any new matching topic, set auto.create.topics.enable=true and allow.auto.create.topics=true and very low metadata.max.age.ms
I've been playing around with Kafka trying to build a PoC for a cluster event notification system. I can't /quite/ get it doing what I'm after (at the moment). No doubt this is me missing some little nugget in the docs, or mis-understanding something. I've also searched around, but perhaps am using the wrong terms.
I'd like to set up a consumer with a topic pattern along the lines of "test.*" then have it pick up all messages sent to topics that match that pattern. So far I'm missing the first new message sent to a new pattern.
e.g. it will miss the first message sent to test.a, but catch subsequent ones.
I've set the following:
auto.create.topics.enable=true
allow.auto.create.topics=true
And in the consumer properties I've set metadata.max.age.ms very low (around 50). (I'm not sure what kind of performance problems doing this could do in production with multiple consumers.)
Yet I'll still (almost always) miss the first message to a new topic in the pattern.
If anyone has any ideas, or even better things to google for to help solve this I'd be most appreciative even if the result is "you just won't be able to do what you want" ;-)
Many thanks in advance,
Chris
Re: Question regarding pattern subscriptions
Posted by Chris Hampson <ch...@hampson.dev>.
Thanks Alok,
I'd come across that and begun playing with different values for that. Tweaking it has done exactly what I was after. I'm now getting the desired behaviour.
Thanks again for the speedy response,
Cheers,
Chris
> On 22 Jul 2019, at 09:36, Alok Dwivedi <al...@instaclustr.com> wrote:
>
> Hi Chris
> Since you are only missing the very first message sent to that topic, I wonder its related to " auto.offset.reset" consumer setting. This setting controls what happens when no offset is present for a topic (most commonly the case when there has been no consumer yet for that topic)
>
> From documentation:
> "Use auto.offset.reset to define the behaviour of the consumer when there is no committed position (which would be the case when the group is first initialized) or when an offset is out of range. You can choose either to reset the position to the "earliest" offset or the "latest" offset (the default). You can also select "none" if you would rather set the initial offset yourself and you are willing to handle out of range errors manually."
>
> With default behaviour of "latest" there is a possibility that consumer is not yet ready to listen to messages by the time 1st message was sent but then its ready and starts consuming from 2nd (latest since its stared listening). You can prove or disprove this theory of mine by changing default to "earliest" and see if you always get messages from beginning once default is overridden (or not).
>
> Thanks
> Alok Dwivedi
> Senior Consultant
> https://www.instaclustr.com/platform/
>
>
> On 19/07/2019, 14:01, "Chris Hampson" <ch...@hampson.dev> wrote:
>
> Hi all,
>
> Tl;dr: I want to set up a subscription based on patterns that will match non-existent topics and receive all messages that match (e.g. test.*), missing first message to any new matching topic, set auto.create.topics.enable=true and allow.auto.create.topics=true and very low metadata.max.age.ms
>
> I've been playing around with Kafka trying to build a PoC for a cluster event notification system. I can't /quite/ get it doing what I'm after (at the moment). No doubt this is me missing some little nugget in the docs, or mis-understanding something. I've also searched around, but perhaps am using the wrong terms.
>
> I'd like to set up a consumer with a topic pattern along the lines of "test.*" then have it pick up all messages sent to topics that match that pattern. So far I'm missing the first new message sent to a new pattern.
>
> e.g. it will miss the first message sent to test.a, but catch subsequent ones.
>
> I've set the following:
> auto.create.topics.enable=true
> allow.auto.create.topics=true
>
> And in the consumer properties I've set metadata.max.age.ms very low (around 50). (I'm not sure what kind of performance problems doing this could do in production with multiple consumers.)
>
> Yet I'll still (almost always) miss the first message to a new topic in the pattern.
>
> If anyone has any ideas, or even better things to google for to help solve this I'd be most appreciative even if the result is "you just won't be able to do what you want" ;-)
>
> Many thanks in advance,
>
> Chris
>
Re: Question regarding pattern subscriptions
Posted by Alok Dwivedi <al...@instaclustr.com>.
Hi Chris
Since you are only missing the very first message sent to that topic, I wonder its related to " auto.offset.reset" consumer setting. This setting controls what happens when no offset is present for a topic (most commonly the case when there has been no consumer yet for that topic)
From documentation:
"Use auto.offset.reset to define the behaviour of the consumer when there is no committed position (which would be the case when the group is first initialized) or when an offset is out of range. You can choose either to reset the position to the "earliest" offset or the "latest" offset (the default). You can also select "none" if you would rather set the initial offset yourself and you are willing to handle out of range errors manually."
With default behaviour of "latest" there is a possibility that consumer is not yet ready to listen to messages by the time 1st message was sent but then its ready and starts consuming from 2nd (latest since its stared listening). You can prove or disprove this theory of mine by changing default to "earliest" and see if you always get messages from beginning once default is overridden (or not).
Thanks
Alok Dwivedi
Senior Consultant
https://www.instaclustr.com/platform/
On 19/07/2019, 14:01, "Chris Hampson" <ch...@hampson.dev> wrote:
Hi all,
Tl;dr: I want to set up a subscription based on patterns that will match non-existent topics and receive all messages that match (e.g. test.*), missing first message to any new matching topic, set auto.create.topics.enable=true and allow.auto.create.topics=true and very low metadata.max.age.ms
I've been playing around with Kafka trying to build a PoC for a cluster event notification system. I can't /quite/ get it doing what I'm after (at the moment). No doubt this is me missing some little nugget in the docs, or mis-understanding something. I've also searched around, but perhaps am using the wrong terms.
I'd like to set up a consumer with a topic pattern along the lines of "test.*" then have it pick up all messages sent to topics that match that pattern. So far I'm missing the first new message sent to a new pattern.
e.g. it will miss the first message sent to test.a, but catch subsequent ones.
I've set the following:
auto.create.topics.enable=true
allow.auto.create.topics=true
And in the consumer properties I've set metadata.max.age.ms very low (around 50). (I'm not sure what kind of performance problems doing this could do in production with multiple consumers.)
Yet I'll still (almost always) miss the first message to a new topic in the pattern.
If anyone has any ideas, or even better things to google for to help solve this I'd be most appreciative even if the result is "you just won't be able to do what you want" ;-)
Many thanks in advance,
Chris