Unexpected mirror maker behavior with new consumer

[Luke Steensen <lu...@braintreepayments.com>]

Hello,

I've been experimenting with mirror maker and am a bit confused about some
behavior I'm seeing.

I have two simple single broker clusters setup locally, source and target.
The source cluster has a topic foo with a few message in it.

When I run mirror maker with --whitelist '*' (as noted in the
documentation), nothing is written to the target cluster. Switching '.*'
makes no difference.

With --whitelist '\w*', everything is mirrored, including
__consumer_offsets.

With --whitelist foo, the foo topic is mirrored correctly from source to
target.

With --whitelist 'foo|bar' (where foo already exists and bar does not yet),
foo is mirrored correctly, but bar is ignored when it gets created.
Restarting mirror maker with the same command causes it to pick up on bar.

I assume there is some outdated documentation around old vs new consumer in
mirror maker, but it's still not obvious how to automatically mirror all
non-internal topics. Any pointers would be appreciated.

Thanks,
Luke

Re: Unexpected mirror maker behavior with new consumer

[Luke Steensen <lu...@braintreepayments.com>]

I've opened https://issues.apache.org/jira/browse/KAFKA-3285 to track the
issue.

I will test again with metadata.max.age.ms in mind, that makes sense.
Thanks!


On Wed, Feb 24, 2016 at 8:22 PM, tao xiao <xi...@gmail.com> wrote:

> One more additional note to new consumer. The new topic will not be picked
> up by new consumer immediately. It takes as long as metadata.max.age.ms to
> refresh metadata and pick up topics that match the pattern at the time of
> check.
>
> On Thu, 25 Feb 2016 at 08:48 Jason Gustafson <ja...@confluent.io> wrote:
>
> > Hey Luke,
> >
> > I took a look at the code and it does look like the whitelist argument is
> > handled differently between the old and new consumers. For the new
> > consumer, we just treat it as a raw regular expression, but the old
> > consumer does some preprocessing. We should probably do the preprocessing
> > in both cases for consistency. Would you mind opening a JIRA?
> >
> > Thanks,
> > Jason
> >
> > On Wed, Feb 24, 2016 at 4:17 PM, Luke Steensen <
> > luke.steensen@braintreepayments.com> wrote:
> >
> > > Hello,
> > >
> > > I've been experimenting with mirror maker and am a bit confused about
> > some
> > > behavior I'm seeing.
> > >
> > > I have two simple single broker clusters setup locally, source and
> > target.
> > > The source cluster has a topic foo with a few message in it.
> > >
> > > When I run mirror maker with --whitelist '*' (as noted in the
> > > documentation), nothing is written to the target cluster. Switching
> '.*'
> > > makes no difference.
> > >
> > > With --whitelist '\w*', everything is mirrored, including
> > > __consumer_offsets.
> > >
> > > With --whitelist foo, the foo topic is mirrored correctly from source
> to
> > > target.
> > >
> > > With --whitelist 'foo|bar' (where foo already exists and bar does not
> > yet),
> > > foo is mirrored correctly, but bar is ignored when it gets created.
> > > Restarting mirror maker with the same command causes it to pick up on
> > bar.
> > >
> > > I assume there is some outdated documentation around old vs new
> consumer
> > in
> > > mirror maker, but it's still not obvious how to automatically mirror
> all
> > > non-internal topics. Any pointers would be appreciated.
> > >
> > > Thanks,
> > > Luke
> > >
> >
>

Re: Unexpected mirror maker behavior with new consumer

[tao xiao <xi...@gmail.com>]

One more additional note to new consumer. The new topic will not be picked
up by new consumer immediately. It takes as long as metadata.max.age.ms to
refresh metadata and pick up topics that match the pattern at the time of
check.

On Thu, 25 Feb 2016 at 08:48 Jason Gustafson <ja...@confluent.io> wrote:

> Hey Luke,
>
> I took a look at the code and it does look like the whitelist argument is
> handled differently between the old and new consumers. For the new
> consumer, we just treat it as a raw regular expression, but the old
> consumer does some preprocessing. We should probably do the preprocessing
> in both cases for consistency. Would you mind opening a JIRA?
>
> Thanks,
> Jason
>
> On Wed, Feb 24, 2016 at 4:17 PM, Luke Steensen <
> luke.steensen@braintreepayments.com> wrote:
>
> > Hello,
> >
> > I've been experimenting with mirror maker and am a bit confused about
> some
> > behavior I'm seeing.
> >
> > I have two simple single broker clusters setup locally, source and
> target.
> > The source cluster has a topic foo with a few message in it.
> >
> > When I run mirror maker with --whitelist '*' (as noted in the
> > documentation), nothing is written to the target cluster. Switching '.*'
> > makes no difference.
> >
> > With --whitelist '\w*', everything is mirrored, including
> > __consumer_offsets.
> >
> > With --whitelist foo, the foo topic is mirrored correctly from source to
> > target.
> >
> > With --whitelist 'foo|bar' (where foo already exists and bar does not
> yet),
> > foo is mirrored correctly, but bar is ignored when it gets created.
> > Restarting mirror maker with the same command causes it to pick up on
> bar.
> >
> > I assume there is some outdated documentation around old vs new consumer
> in
> > mirror maker, but it's still not obvious how to automatically mirror all
> > non-internal topics. Any pointers would be appreciated.
> >
> > Thanks,
> > Luke
> >
>

Re: Unexpected mirror maker behavior with new consumer

[Jason Gustafson <ja...@confluent.io>]

Hey Luke,

I took a look at the code and it does look like the whitelist argument is
handled differently between the old and new consumers. For the new
consumer, we just treat it as a raw regular expression, but the old
consumer does some preprocessing. We should probably do the preprocessing
in both cases for consistency. Would you mind opening a JIRA?

Thanks,
Jason

On Wed, Feb 24, 2016 at 4:17 PM, Luke Steensen <
luke.steensen@braintreepayments.com> wrote:

> Hello,
>
> I've been experimenting with mirror maker and am a bit confused about some
> behavior I'm seeing.
>
> I have two simple single broker clusters setup locally, source and target.
> The source cluster has a topic foo with a few message in it.
>
> When I run mirror maker with --whitelist '*' (as noted in the
> documentation), nothing is written to the target cluster. Switching '.*'
> makes no difference.
>
> With --whitelist '\w*', everything is mirrored, including
> __consumer_offsets.
>
> With --whitelist foo, the foo topic is mirrored correctly from source to
> target.
>
> With --whitelist 'foo|bar' (where foo already exists and bar does not yet),
> foo is mirrored correctly, but bar is ignored when it gets created.
> Restarting mirror maker with the same command causes it to pick up on bar.
>
> I assume there is some outdated documentation around old vs new consumer in
> mirror maker, but it's still not obvious how to automatically mirror all
> non-internal topics. Any pointers would be appreciated.
>
> Thanks,
> Luke
>