kafka mirroring ...!

[sunil kalva <sa...@gmail.com>]

I ran kafka mirroring tool after producing data in source cluster, and this
is not copied to destination cluster. If i produce data after running tool
those data are copied to destination cluster. Am i missing something ?

-- 
SunilKalva

Re: kafka mirroring ...!

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

Hi Sunilkalva,

We are rewriting mirror maker in KAFKA-1997 with a handful of enhancement.
With that new mirror maker, you will be able to mirror to a different
topic by using the message handler.

Jiangjie (Becket) Qin

On 3/9/15, 4:41 AM, "sunil kalva" <sa...@gmail.com> wrote:

>I think it will be very usefull if we can mirror to a different topic name
>on destination side.
>We have a use case to merge data from multiple colos to one central colo.
>
>SunilKalva
>
>On Mon, Mar 9, 2015 at 4:29 PM, tao xiao <xi...@gmail.com> wrote:
>
>> I don't think you can mirror messages to a different topic name in the
>> current mirror maker implementation. Mirror maker sends the message to
>> destination topic based on the topic name it reads from source
>>
>> On Mon, Mar 9, 2015 at 5:00 PM, sunil kalva <sa...@gmail.com>
>>wrote:
>>
>> > Can i configure different topic name in destination cluster, i mean
>>can i
>> > have different topic names for source and destination cluster for
>> > mirroring. If yes how can i map source topic with destination topic
>>name
>> ?
>> >
>> > SunilKalva
>> >
>> > On Mon, Mar 9, 2015 at 6:41 AM, tao xiao <xi...@gmail.com> wrote:
>> >
>> > > Ctrl+c is clean shutdown. kill -9 is not
>> > >
>> > > On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville <am...@g.hmc.edu>
>> > wrote:
>> > >
>> > > > What does a "clean shutdown" of the MM entail? So far I've just
>>been
>> > > using
>> > > > Ctrl + C to send an interrupt to kill it.
>> > > >
>> > > >
>> > > > Alex
>> > > >
>> > > > On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin
>> > <jqin@linkedin.com.invalid
>> > > >
>> > > > wrote:
>> > > >
>> > > > > If auto.offset.reset is set to smallest, it does not mean the
>> > consumer
>> > > > > will always consume from the smallest. It means that if no
>>previous
>> > > > offset
>> > > > > commit is found for this consumer group, then it will consume
>>from
>> > the
>> > > > > smallest. So for mirror maker, you probably want to always use
>>the
>> > same
>> > > > > consumer group id. This could be configured in the consumer
>>config
>> > file
>> > > > > you pass into mirror maker.
>> > > > > Another thing about duplicate messages is that if mirror maker
>>is
>> > > > shutdown
>> > > > > cleanly, next time when you start it again with same consumer
>>group
>> > id,
>> > > > > there should be no duplicates. But if mirror maker shutdown
>> > > > uncleanly(e.g.
>> > > > > By a kill -9), then next time it starts up you might still have
>> > > duplicate
>> > > > > messages after the last committed offsets.
>> > > > >
>> > > > > Jiangjie (Becket) Qin
>> > > > >
>> > > > > On 3/7/15, 11:45 PM, "sunil kalva" <sa...@gmail.com> wrote:
>> > > > >
>> > > > > >Qin
>> > > > > >Partition problem is solved by passing "--new.producer true"
>> option
>> > in
>> > > > > >command line,  but adding auto.offset.rese=smallest config,
>>every
>> > > time i
>> > > > > >restart the Mirror tool it copies from starting ends up having
>>lot
>> > of
>> > > > > >duplicate messages in destination cluster.
>> > > > > >Could you please tell me how do i configure to make sure that
>> > > > destination
>> > > > > >cluster is always insync with source cluster.
>> > > > > >
>> > > > > >SunilKalva
>> > > > > >
>> > > > > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin
>> > > <jqin@linkedin.com.invalid
>> > > > >
>> > > > > >wrote:
>> > > > > >
>> > > > > >> For data not showing up, you need to make sure mirror maker
>> > consumer
>> > > > > >> auto.offset.reset is set to smallest, otherwise when you run
>> > mirror
>> > > > > >>maker
>> > > > > >> for the first time, all the pre-existing messages won¹t be
>> > consumed.
>> > > > > >> For partition sticking, can you verify if your messages are
>> keyed
>> > > > > >>messages
>> > > > > >> or not? If they are not keyed messages, can you check if you
>>are
>> > > using
>> > > > > >>old
>> > > > > >> producer or new producer? For old producer, the default
>>behavior
>> > is
>> > > > > >> sticking to one partition for 10 min and then move to the
>>next
>> > > > > >>partition.
>> > > > > >> So if you wait for more than 10 min, you should see messages
>>in
>> > two
>> > > > > >> different partitions.
>> > > > > >>
>> > > > > >> Jiangjie (Becket) Qin
>> > > > > >>
>> > > > > >> On 3/7/15, 8:28 AM, "sunil kalva" <sa...@gmail.com>
>>wrote:
>> > > > > >>
>> > > > > >> >And i also observed ,all the data is moving to one
>>partition in
>> > > > > >> >destination
>> > > > > >> >cluster though i have multiple partitions for that topic in
>> > source
>> > > > and
>> > > > > >> >destination clusters.
>> > > > > >> >
>> > > > > >> >SunilKalva
>> > > > > >> >
>> > > > > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <
>> > sambarcode@gmail.com>
>> > > > > >>wrote:
>> > > > > >> >
>> > > > > >> >> I ran kafka mirroring tool after producing data in source
>> > > cluster,
>> > > > > >>and
>> > > > > >> >> this is not copied to destination cluster. If i produce
>>data
>> > > after
>> > > > > >> >>running
>> > > > > >> >> tool those data are copied to destination cluster. Am i
>> missing
>> > > > > >> >>something ?
>> > > > > >> >>
>> > > > > >> >> --
>> > > > > >> >> SunilKalva
>> > > > > >> >>
>> > > > > >> >
>> > > > > >> >
>> > > > > >> >
>> > > > > >> >--
>> > > > > >> >SunilKalva
>> > > > > >>
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > > >--
>> > > > > >SunilKalva
>> > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Tao
>> > >
>> >
>> >
>> >
>> > --
>> > SunilKalva
>> >
>>
>>
>>
>> --
>> Regards,
>> Tao
>>
>
>
>
>-- 
>SunilKalva

Re: kafka mirroring ...!

[sunil kalva <sa...@gmail.com>]

I think it will be very usefull if we can mirror to a different topic name
on destination side.
We have a use case to merge data from multiple colos to one central colo.

SunilKalva

On Mon, Mar 9, 2015 at 4:29 PM, tao xiao <xi...@gmail.com> wrote:

> I don't think you can mirror messages to a different topic name in the
> current mirror maker implementation. Mirror maker sends the message to
> destination topic based on the topic name it reads from source
>
> On Mon, Mar 9, 2015 at 5:00 PM, sunil kalva <sa...@gmail.com> wrote:
>
> > Can i configure different topic name in destination cluster, i mean can i
> > have different topic names for source and destination cluster for
> > mirroring. If yes how can i map source topic with destination topic name
> ?
> >
> > SunilKalva
> >
> > On Mon, Mar 9, 2015 at 6:41 AM, tao xiao <xi...@gmail.com> wrote:
> >
> > > Ctrl+c is clean shutdown. kill -9 is not
> > >
> > > On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville <am...@g.hmc.edu>
> > wrote:
> > >
> > > > What does a "clean shutdown" of the MM entail? So far I've just been
> > > using
> > > > Ctrl + C to send an interrupt to kill it.
> > > >
> > > >
> > > > Alex
> > > >
> > > > On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin
> > <jqin@linkedin.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > If auto.offset.reset is set to smallest, it does not mean the
> > consumer
> > > > > will always consume from the smallest. It means that if no previous
> > > > offset
> > > > > commit is found for this consumer group, then it will consume from
> > the
> > > > > smallest. So for mirror maker, you probably want to always use the
> > same
> > > > > consumer group id. This could be configured in the consumer config
> > file
> > > > > you pass into mirror maker.
> > > > > Another thing about duplicate messages is that if mirror maker is
> > > > shutdown
> > > > > cleanly, next time when you start it again with same consumer group
> > id,
> > > > > there should be no duplicates. But if mirror maker shutdown
> > > > uncleanly(e.g.
> > > > > By a kill -9), then next time it starts up you might still have
> > > duplicate
> > > > > messages after the last committed offsets.
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On 3/7/15, 11:45 PM, "sunil kalva" <sa...@gmail.com> wrote:
> > > > >
> > > > > >Qin
> > > > > >Partition problem is solved by passing "--new.producer true"
> option
> > in
> > > > > >command line,  but adding auto.offset.rese=smallest config, every
> > > time i
> > > > > >restart the Mirror tool it copies from starting ends up having lot
> > of
> > > > > >duplicate messages in destination cluster.
> > > > > >Could you please tell me how do i configure to make sure that
> > > > destination
> > > > > >cluster is always insync with source cluster.
> > > > > >
> > > > > >SunilKalva
> > > > > >
> > > > > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin
> > > <jqin@linkedin.com.invalid
> > > > >
> > > > > >wrote:
> > > > > >
> > > > > >> For data not showing up, you need to make sure mirror maker
> > consumer
> > > > > >> auto.offset.reset is set to smallest, otherwise when you run
> > mirror
> > > > > >>maker
> > > > > >> for the first time, all the pre-existing messages won¹t be
> > consumed.
> > > > > >> For partition sticking, can you verify if your messages are
> keyed
> > > > > >>messages
> > > > > >> or not? If they are not keyed messages, can you check if you are
> > > using
> > > > > >>old
> > > > > >> producer or new producer? For old producer, the default behavior
> > is
> > > > > >> sticking to one partition for 10 min and then move to the next
> > > > > >>partition.
> > > > > >> So if you wait for more than 10 min, you should see messages in
> > two
> > > > > >> different partitions.
> > > > > >>
> > > > > >> Jiangjie (Becket) Qin
> > > > > >>
> > > > > >> On 3/7/15, 8:28 AM, "sunil kalva" <sa...@gmail.com> wrote:
> > > > > >>
> > > > > >> >And i also observed ,all the data is moving to one partition in
> > > > > >> >destination
> > > > > >> >cluster though i have multiple partitions for that topic in
> > source
> > > > and
> > > > > >> >destination clusters.
> > > > > >> >
> > > > > >> >SunilKalva
> > > > > >> >
> > > > > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <
> > sambarcode@gmail.com>
> > > > > >>wrote:
> > > > > >> >
> > > > > >> >> I ran kafka mirroring tool after producing data in source
> > > cluster,
> > > > > >>and
> > > > > >> >> this is not copied to destination cluster. If i produce data
> > > after
> > > > > >> >>running
> > > > > >> >> tool those data are copied to destination cluster. Am i
> missing
> > > > > >> >>something ?
> > > > > >> >>
> > > > > >> >> --
> > > > > >> >> SunilKalva
> > > > > >> >>
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >--
> > > > > >> >SunilKalva
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > >--
> > > > > >SunilKalva
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
> >
> >
> > --
> > SunilKalva
> >
>
>
>
> --
> Regards,
> Tao
>



-- 
SunilKalva

Re: kafka mirroring ...!

[tao xiao <xi...@gmail.com>]

I don't think you can mirror messages to a different topic name in the
current mirror maker implementation. Mirror maker sends the message to
destination topic based on the topic name it reads from source

On Mon, Mar 9, 2015 at 5:00 PM, sunil kalva <sa...@gmail.com> wrote:

> Can i configure different topic name in destination cluster, i mean can i
> have different topic names for source and destination cluster for
> mirroring. If yes how can i map source topic with destination topic name ?
>
> SunilKalva
>
> On Mon, Mar 9, 2015 at 6:41 AM, tao xiao <xi...@gmail.com> wrote:
>
> > Ctrl+c is clean shutdown. kill -9 is not
> >
> > On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville <am...@g.hmc.edu>
> wrote:
> >
> > > What does a "clean shutdown" of the MM entail? So far I've just been
> > using
> > > Ctrl + C to send an interrupt to kill it.
> > >
> > >
> > > Alex
> > >
> > > On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin
> <jqin@linkedin.com.invalid
> > >
> > > wrote:
> > >
> > > > If auto.offset.reset is set to smallest, it does not mean the
> consumer
> > > > will always consume from the smallest. It means that if no previous
> > > offset
> > > > commit is found for this consumer group, then it will consume from
> the
> > > > smallest. So for mirror maker, you probably want to always use the
> same
> > > > consumer group id. This could be configured in the consumer config
> file
> > > > you pass into mirror maker.
> > > > Another thing about duplicate messages is that if mirror maker is
> > > shutdown
> > > > cleanly, next time when you start it again with same consumer group
> id,
> > > > there should be no duplicates. But if mirror maker shutdown
> > > uncleanly(e.g.
> > > > By a kill -9), then next time it starts up you might still have
> > duplicate
> > > > messages after the last committed offsets.
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On 3/7/15, 11:45 PM, "sunil kalva" <sa...@gmail.com> wrote:
> > > >
> > > > >Qin
> > > > >Partition problem is solved by passing "--new.producer true" option
> in
> > > > >command line,  but adding auto.offset.rese=smallest config, every
> > time i
> > > > >restart the Mirror tool it copies from starting ends up having lot
> of
> > > > >duplicate messages in destination cluster.
> > > > >Could you please tell me how do i configure to make sure that
> > > destination
> > > > >cluster is always insync with source cluster.
> > > > >
> > > > >SunilKalva
> > > > >
> > > > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin
> > <jqin@linkedin.com.invalid
> > > >
> > > > >wrote:
> > > > >
> > > > >> For data not showing up, you need to make sure mirror maker
> consumer
> > > > >> auto.offset.reset is set to smallest, otherwise when you run
> mirror
> > > > >>maker
> > > > >> for the first time, all the pre-existing messages won¹t be
> consumed.
> > > > >> For partition sticking, can you verify if your messages are keyed
> > > > >>messages
> > > > >> or not? If they are not keyed messages, can you check if you are
> > using
> > > > >>old
> > > > >> producer or new producer? For old producer, the default behavior
> is
> > > > >> sticking to one partition for 10 min and then move to the next
> > > > >>partition.
> > > > >> So if you wait for more than 10 min, you should see messages in
> two
> > > > >> different partitions.
> > > > >>
> > > > >> Jiangjie (Becket) Qin
> > > > >>
> > > > >> On 3/7/15, 8:28 AM, "sunil kalva" <sa...@gmail.com> wrote:
> > > > >>
> > > > >> >And i also observed ,all the data is moving to one partition in
> > > > >> >destination
> > > > >> >cluster though i have multiple partitions for that topic in
> source
> > > and
> > > > >> >destination clusters.
> > > > >> >
> > > > >> >SunilKalva
> > > > >> >
> > > > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <
> sambarcode@gmail.com>
> > > > >>wrote:
> > > > >> >
> > > > >> >> I ran kafka mirroring tool after producing data in source
> > cluster,
> > > > >>and
> > > > >> >> this is not copied to destination cluster. If i produce data
> > after
> > > > >> >>running
> > > > >> >> tool those data are copied to destination cluster. Am i missing
> > > > >> >>something ?
> > > > >> >>
> > > > >> >> --
> > > > >> >> SunilKalva
> > > > >> >>
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >--
> > > > >> >SunilKalva
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >SunilKalva
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
>
>
> --
> SunilKalva
>



-- 
Regards,
Tao

Re: kafka mirroring ...!

[sunil kalva <sa...@gmail.com>]

Can i configure different topic name in destination cluster, i mean can i
have different topic names for source and destination cluster for
mirroring. If yes how can i map source topic with destination topic name ?

SunilKalva

On Mon, Mar 9, 2015 at 6:41 AM, tao xiao <xi...@gmail.com> wrote:

> Ctrl+c is clean shutdown. kill -9 is not
>
> On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville <am...@g.hmc.edu> wrote:
>
> > What does a "clean shutdown" of the MM entail? So far I've just been
> using
> > Ctrl + C to send an interrupt to kill it.
> >
> >
> > Alex
> >
> > On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin <jqin@linkedin.com.invalid
> >
> > wrote:
> >
> > > If auto.offset.reset is set to smallest, it does not mean the consumer
> > > will always consume from the smallest. It means that if no previous
> > offset
> > > commit is found for this consumer group, then it will consume from the
> > > smallest. So for mirror maker, you probably want to always use the same
> > > consumer group id. This could be configured in the consumer config file
> > > you pass into mirror maker.
> > > Another thing about duplicate messages is that if mirror maker is
> > shutdown
> > > cleanly, next time when you start it again with same consumer group id,
> > > there should be no duplicates. But if mirror maker shutdown
> > uncleanly(e.g.
> > > By a kill -9), then next time it starts up you might still have
> duplicate
> > > messages after the last committed offsets.
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On 3/7/15, 11:45 PM, "sunil kalva" <sa...@gmail.com> wrote:
> > >
> > > >Qin
> > > >Partition problem is solved by passing "--new.producer true" option in
> > > >command line,  but adding auto.offset.rese=smallest config, every
> time i
> > > >restart the Mirror tool it copies from starting ends up having lot of
> > > >duplicate messages in destination cluster.
> > > >Could you please tell me how do i configure to make sure that
> > destination
> > > >cluster is always insync with source cluster.
> > > >
> > > >SunilKalva
> > > >
> > > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin
> <jqin@linkedin.com.invalid
> > >
> > > >wrote:
> > > >
> > > >> For data not showing up, you need to make sure mirror maker consumer
> > > >> auto.offset.reset is set to smallest, otherwise when you run mirror
> > > >>maker
> > > >> for the first time, all the pre-existing messages won¹t be consumed.
> > > >> For partition sticking, can you verify if your messages are keyed
> > > >>messages
> > > >> or not? If they are not keyed messages, can you check if you are
> using
> > > >>old
> > > >> producer or new producer? For old producer, the default behavior is
> > > >> sticking to one partition for 10 min and then move to the next
> > > >>partition.
> > > >> So if you wait for more than 10 min, you should see messages in two
> > > >> different partitions.
> > > >>
> > > >> Jiangjie (Becket) Qin
> > > >>
> > > >> On 3/7/15, 8:28 AM, "sunil kalva" <sa...@gmail.com> wrote:
> > > >>
> > > >> >And i also observed ,all the data is moving to one partition in
> > > >> >destination
> > > >> >cluster though i have multiple partitions for that topic in source
> > and
> > > >> >destination clusters.
> > > >> >
> > > >> >SunilKalva
> > > >> >
> > > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sa...@gmail.com>
> > > >>wrote:
> > > >> >
> > > >> >> I ran kafka mirroring tool after producing data in source
> cluster,
> > > >>and
> > > >> >> this is not copied to destination cluster. If i produce data
> after
> > > >> >>running
> > > >> >> tool those data are copied to destination cluster. Am i missing
> > > >> >>something ?
> > > >> >>
> > > >> >> --
> > > >> >> SunilKalva
> > > >> >>
> > > >> >
> > > >> >
> > > >> >
> > > >> >--
> > > >> >SunilKalva
> > > >>
> > > >>
> > > >
> > > >
> > > >--
> > > >SunilKalva
> > >
> > >
> >
>
>
>
> --
> Regards,
> Tao
>



-- 
SunilKalva

Re: kafka mirroring ...!

[tao xiao <xi...@gmail.com>]

Ctrl+c is clean shutdown. kill -9 is not

On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville <am...@g.hmc.edu> wrote:

> What does a "clean shutdown" of the MM entail? So far I've just been using
> Ctrl + C to send an interrupt to kill it.
>
>
> Alex
>
> On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
> > If auto.offset.reset is set to smallest, it does not mean the consumer
> > will always consume from the smallest. It means that if no previous
> offset
> > commit is found for this consumer group, then it will consume from the
> > smallest. So for mirror maker, you probably want to always use the same
> > consumer group id. This could be configured in the consumer config file
> > you pass into mirror maker.
> > Another thing about duplicate messages is that if mirror maker is
> shutdown
> > cleanly, next time when you start it again with same consumer group id,
> > there should be no duplicates. But if mirror maker shutdown
> uncleanly(e.g.
> > By a kill -9), then next time it starts up you might still have duplicate
> > messages after the last committed offsets.
> >
> > Jiangjie (Becket) Qin
> >
> > On 3/7/15, 11:45 PM, "sunil kalva" <sa...@gmail.com> wrote:
> >
> > >Qin
> > >Partition problem is solved by passing "--new.producer true" option in
> > >command line,  but adding auto.offset.rese=smallest config, every time i
> > >restart the Mirror tool it copies from starting ends up having lot of
> > >duplicate messages in destination cluster.
> > >Could you please tell me how do i configure to make sure that
> destination
> > >cluster is always insync with source cluster.
> > >
> > >SunilKalva
> > >
> > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin <jqin@linkedin.com.invalid
> >
> > >wrote:
> > >
> > >> For data not showing up, you need to make sure mirror maker consumer
> > >> auto.offset.reset is set to smallest, otherwise when you run mirror
> > >>maker
> > >> for the first time, all the pre-existing messages won¹t be consumed.
> > >> For partition sticking, can you verify if your messages are keyed
> > >>messages
> > >> or not? If they are not keyed messages, can you check if you are using
> > >>old
> > >> producer or new producer? For old producer, the default behavior is
> > >> sticking to one partition for 10 min and then move to the next
> > >>partition.
> > >> So if you wait for more than 10 min, you should see messages in two
> > >> different partitions.
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> > >> On 3/7/15, 8:28 AM, "sunil kalva" <sa...@gmail.com> wrote:
> > >>
> > >> >And i also observed ,all the data is moving to one partition in
> > >> >destination
> > >> >cluster though i have multiple partitions for that topic in source
> and
> > >> >destination clusters.
> > >> >
> > >> >SunilKalva
> > >> >
> > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sa...@gmail.com>
> > >>wrote:
> > >> >
> > >> >> I ran kafka mirroring tool after producing data in source cluster,
> > >>and
> > >> >> this is not copied to destination cluster. If i produce data after
> > >> >>running
> > >> >> tool those data are copied to destination cluster. Am i missing
> > >> >>something ?
> > >> >>
> > >> >> --
> > >> >> SunilKalva
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> >--
> > >> >SunilKalva
> > >>
> > >>
> > >
> > >
> > >--
> > >SunilKalva
> >
> >
>



-- 
Regards,
Tao

Re: kafka mirroring ...!

[Alex Melville <am...@g.hmc.edu>]

What does a "clean shutdown" of the MM entail? So far I've just been using
Ctrl + C to send an interrupt to kill it.


Alex

On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:

> If auto.offset.reset is set to smallest, it does not mean the consumer
> will always consume from the smallest. It means that if no previous offset
> commit is found for this consumer group, then it will consume from the
> smallest. So for mirror maker, you probably want to always use the same
> consumer group id. This could be configured in the consumer config file
> you pass into mirror maker.
> Another thing about duplicate messages is that if mirror maker is shutdown
> cleanly, next time when you start it again with same consumer group id,
> there should be no duplicates. But if mirror maker shutdown uncleanly(e.g.
> By a kill -9), then next time it starts up you might still have duplicate
> messages after the last committed offsets.
>
> Jiangjie (Becket) Qin
>
> On 3/7/15, 11:45 PM, "sunil kalva" <sa...@gmail.com> wrote:
>
> >Qin
> >Partition problem is solved by passing "--new.producer true" option in
> >command line,  but adding auto.offset.rese=smallest config, every time i
> >restart the Mirror tool it copies from starting ends up having lot of
> >duplicate messages in destination cluster.
> >Could you please tell me how do i configure to make sure that destination
> >cluster is always insync with source cluster.
> >
> >SunilKalva
> >
> >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
> >wrote:
> >
> >> For data not showing up, you need to make sure mirror maker consumer
> >> auto.offset.reset is set to smallest, otherwise when you run mirror
> >>maker
> >> for the first time, all the pre-existing messages won¹t be consumed.
> >> For partition sticking, can you verify if your messages are keyed
> >>messages
> >> or not? If they are not keyed messages, can you check if you are using
> >>old
> >> producer or new producer? For old producer, the default behavior is
> >> sticking to one partition for 10 min and then move to the next
> >>partition.
> >> So if you wait for more than 10 min, you should see messages in two
> >> different partitions.
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On 3/7/15, 8:28 AM, "sunil kalva" <sa...@gmail.com> wrote:
> >>
> >> >And i also observed ,all the data is moving to one partition in
> >> >destination
> >> >cluster though i have multiple partitions for that topic in source and
> >> >destination clusters.
> >> >
> >> >SunilKalva
> >> >
> >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sa...@gmail.com>
> >>wrote:
> >> >
> >> >> I ran kafka mirroring tool after producing data in source cluster,
> >>and
> >> >> this is not copied to destination cluster. If i produce data after
> >> >>running
> >> >> tool those data are copied to destination cluster. Am i missing
> >> >>something ?
> >> >>
> >> >> --
> >> >> SunilKalva
> >> >>
> >> >
> >> >
> >> >
> >> >--
> >> >SunilKalva
> >>
> >>
> >
> >
> >--
> >SunilKalva
>
>

Re: kafka mirroring ...!

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

If auto.offset.reset is set to smallest, it does not mean the consumer
will always consume from the smallest. It means that if no previous offset
commit is found for this consumer group, then it will consume from the
smallest. So for mirror maker, you probably want to always use the same
consumer group id. This could be configured in the consumer config file
you pass into mirror maker.
Another thing about duplicate messages is that if mirror maker is shutdown
cleanly, next time when you start it again with same consumer group id,
there should be no duplicates. But if mirror maker shutdown uncleanly(e.g.
By a kill -9), then next time it starts up you might still have duplicate
messages after the last committed offsets.

Jiangjie (Becket) Qin

On 3/7/15, 11:45 PM, "sunil kalva" <sa...@gmail.com> wrote:

>Qin
>Partition problem is solved by passing "--new.producer true" option in
>command line,  but adding auto.offset.rese=smallest config, every time i
>restart the Mirror tool it copies from starting ends up having lot of
>duplicate messages in destination cluster.
>Could you please tell me how do i configure to make sure that destination
>cluster is always insync with source cluster.
>
>SunilKalva
>
>On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
>wrote:
>
>> For data not showing up, you need to make sure mirror maker consumer
>> auto.offset.reset is set to smallest, otherwise when you run mirror
>>maker
>> for the first time, all the pre-existing messages won¹t be consumed.
>> For partition sticking, can you verify if your messages are keyed
>>messages
>> or not? If they are not keyed messages, can you check if you are using
>>old
>> producer or new producer? For old producer, the default behavior is
>> sticking to one partition for 10 min and then move to the next
>>partition.
>> So if you wait for more than 10 min, you should see messages in two
>> different partitions.
>>
>> Jiangjie (Becket) Qin
>>
>> On 3/7/15, 8:28 AM, "sunil kalva" <sa...@gmail.com> wrote:
>>
>> >And i also observed ,all the data is moving to one partition in
>> >destination
>> >cluster though i have multiple partitions for that topic in source and
>> >destination clusters.
>> >
>> >SunilKalva
>> >
>> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sa...@gmail.com>
>>wrote:
>> >
>> >> I ran kafka mirroring tool after producing data in source cluster,
>>and
>> >> this is not copied to destination cluster. If i produce data after
>> >>running
>> >> tool those data are copied to destination cluster. Am i missing
>> >>something ?
>> >>
>> >> --
>> >> SunilKalva
>> >>
>> >
>> >
>> >
>> >--
>> >SunilKalva
>>
>>
>
>
>-- 
>SunilKalva

Re: kafka mirroring ...!

[sunil kalva <sa...@gmail.com>]

Qin
Partition problem is solved by passing "--new.producer true" option in
command line,  but adding auto.offset.rese=smallest config, every time i
restart the Mirror tool it copies from starting ends up having lot of
duplicate messages in destination cluster.
Could you please tell me how do i configure to make sure that destination
cluster is always insync with source cluster.

SunilKalva

On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:

> For data not showing up, you need to make sure mirror maker consumer
> auto.offset.reset is set to smallest, otherwise when you run mirror maker
> for the first time, all the pre-existing messages won¹t be consumed.
> For partition sticking, can you verify if your messages are keyed messages
> or not? If they are not keyed messages, can you check if you are using old
> producer or new producer? For old producer, the default behavior is
> sticking to one partition for 10 min and then move to the next partition.
> So if you wait for more than 10 min, you should see messages in two
> different partitions.
>
> Jiangjie (Becket) Qin
>
> On 3/7/15, 8:28 AM, "sunil kalva" <sa...@gmail.com> wrote:
>
> >And i also observed ,all the data is moving to one partition in
> >destination
> >cluster though i have multiple partitions for that topic in source and
> >destination clusters.
> >
> >SunilKalva
> >
> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sa...@gmail.com> wrote:
> >
> >> I ran kafka mirroring tool after producing data in source cluster, and
> >> this is not copied to destination cluster. If i produce data after
> >>running
> >> tool those data are copied to destination cluster. Am i missing
> >>something ?
> >>
> >> --
> >> SunilKalva
> >>
> >
> >
> >
> >--
> >SunilKalva
>
>


-- 
SunilKalva

Re: kafka mirroring ...!

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

For data not showing up, you need to make sure mirror maker consumer
auto.offset.reset is set to smallest, otherwise when you run mirror maker
for the first time, all the pre-existing messages won¹t be consumed.
For partition sticking, can you verify if your messages are keyed messages
or not? If they are not keyed messages, can you check if you are using old
producer or new producer? For old producer, the default behavior is
sticking to one partition for 10 min and then move to the next partition.
So if you wait for more than 10 min, you should see messages in two
different partitions.

Jiangjie (Becket) Qin

On 3/7/15, 8:28 AM, "sunil kalva" <sa...@gmail.com> wrote:

>And i also observed ,all the data is moving to one partition in
>destination
>cluster though i have multiple partitions for that topic in source and
>destination clusters.
>
>SunilKalva
>
>On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sa...@gmail.com> wrote:
>
>> I ran kafka mirroring tool after producing data in source cluster, and
>> this is not copied to destination cluster. If i produce data after
>>running
>> tool those data are copied to destination cluster. Am i missing
>>something ?
>>
>> --
>> SunilKalva
>>
>
>
>
>-- 
>SunilKalva

Re: kafka mirroring ...!

[sunil kalva <sa...@gmail.com>]

And i also observed ,all the data is moving to one partition in destination
cluster though i have multiple partitions for that topic in source and
destination clusters.

SunilKalva

On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sa...@gmail.com> wrote:

> I ran kafka mirroring tool after producing data in source cluster, and
> this is not copied to destination cluster. If i produce data after running
> tool those data are copied to destination cluster. Am i missing something ?
>
> --
> SunilKalva
>



-- 
SunilKalva