You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Yu, Libo " <li...@citi.com> on 2013/07/02 22:26:39 UTC
partition numbers and mirrormaker
Hi
I asked this question previously and didn't get an answer.
If a topic has N partitions on source cluster, after being mirrored, on the destination
cluster it has only one partition.
To avoid this issue, before mirroring, I create the same topic with N partitions on the
Destination cluster. But then the mirrormaker cannot fetch all partitions and stop
randomly.
So what should we expect from mirrormake and what is the right way to use it? Thanks.
Regards,
Libo
RE: partition numbers and mirrormaker
Posted by "Yu, Libo " <li...@citi.com>.
num.partition will set a number of partition for all topics. If the topics have different partition numbers,
then this will not work.
Regards,
Libo
-----Original Message-----
From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
Sent: Wednesday, July 03, 2013 1:27 PM
To: users@kafka.apache.org
Subject: Re: partition numbers and mirrormaker
Libo, if you set num.partitions on the target brokers and set topic auto creation to true then you don't need to create the topic beforehand.
On Wed, Jul 3, 2013 at 9:15 AM, Yu, Libo <li...@citi.com> wrote:
> Thanks, Jun. Previously I created the topic with the specified number
> of partitions on the destination
> (target) cluster before I launched mirrormaker to do mirroring. But
> the mirrormaker only downloaded data for some of partitions, and in
> some cases, it stopped randomly. I will test this scenario again with beta1.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Wednesday, July 03, 2013 11:49 AM
> To: users@kafka.apache.org
> Subject: Re: partition numbers and mirrormaker
>
> Yes, but you can change num.partitions in the target brokers.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jul 3, 2013 at 6:21 AM, Yu, Libo <li...@citi.com> wrote:
>
>> For a topic with N partitions on the source cluster, if I let
>> mirrormaker to create topic automatically, the same topic on the
>> destination cluster will have only one partition.
>> Is that expected behavior? Could you just give a yes/no answer? Thanks.
>>
>> Regards,
>>
>> Libo
>>
>>
>> -----Original Message-----
>> From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
>> Sent: Tuesday, July 02, 2013 5:23 PM
>> To: users@kafka.apache.org
>> Subject: Re: partition numbers and mirrormaker
>>
>> You can override the default number of partitions on the source and
>> target cluster to N. (num.partitions in the broker config). If you
>> need a higher (or lower) number of partitions for specific topics you
>> can override that on a per-topic basis (again, the config can be kept
>> identical between the source and target cluster).
>>
>> Thanks,
>>
>> Joel
>>
>> On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <li...@citi.com> wrote:
>> > Hi
>> >
>> > I asked this question previously and didn't get an answer.
>> > If a topic has N partitions on source cluster, after being
>> > mirrored, on the destination cluster it has only one partition.
>> >
>> > To avoid this issue, before mirroring, I create the same topic with
>> > N partitions on the Destination cluster. But then the mirrormaker
>> > cannot fetch all partitions and stop randomly.
>> >
>> > So what should we expect from mirrormake and what is the right way
>> > to
>> use it? Thanks.
>> >
>> >
>> > Regards,
>> >
>> > Libo
>> >
>>
Re: partition numbers and mirrormaker
Posted by Joel Koshy <jj...@gmail.com>.
Libo, if you set num.partitions on the target brokers and set topic
auto creation to true then you don't need to create the topic
beforehand.
On Wed, Jul 3, 2013 at 9:15 AM, Yu, Libo <li...@citi.com> wrote:
> Thanks, Jun. Previously I created the topic with the specified number of partitions on the destination
> (target) cluster before I launched mirrormaker to do mirroring. But the mirrormaker only downloaded
> data for some of partitions, and in some cases, it stopped randomly. I will test this scenario again with
> beta1.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Wednesday, July 03, 2013 11:49 AM
> To: users@kafka.apache.org
> Subject: Re: partition numbers and mirrormaker
>
> Yes, but you can change num.partitions in the target brokers.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jul 3, 2013 at 6:21 AM, Yu, Libo <li...@citi.com> wrote:
>
>> For a topic with N partitions on the source cluster, if I let
>> mirrormaker to create topic automatically, the same topic on the
>> destination cluster will have only one partition.
>> Is that expected behavior? Could you just give a yes/no answer? Thanks.
>>
>> Regards,
>>
>> Libo
>>
>>
>> -----Original Message-----
>> From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
>> Sent: Tuesday, July 02, 2013 5:23 PM
>> To: users@kafka.apache.org
>> Subject: Re: partition numbers and mirrormaker
>>
>> You can override the default number of partitions on the source and
>> target cluster to N. (num.partitions in the broker config). If you
>> need a higher (or lower) number of partitions for specific topics you
>> can override that on a per-topic basis (again, the config can be kept
>> identical between the source and target cluster).
>>
>> Thanks,
>>
>> Joel
>>
>> On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <li...@citi.com> wrote:
>> > Hi
>> >
>> > I asked this question previously and didn't get an answer.
>> > If a topic has N partitions on source cluster, after being mirrored,
>> > on the destination cluster it has only one partition.
>> >
>> > To avoid this issue, before mirroring, I create the same topic with
>> > N partitions on the Destination cluster. But then the mirrormaker
>> > cannot fetch all partitions and stop randomly.
>> >
>> > So what should we expect from mirrormake and what is the right way
>> > to
>> use it? Thanks.
>> >
>> >
>> > Regards,
>> >
>> > Libo
>> >
>>
RE: partition numbers and mirrormaker
Posted by "Yu, Libo " <li...@citi.com>.
Thanks, Jun. Previously I created the topic with the specified number of partitions on the destination
(target) cluster before I launched mirrormaker to do mirroring. But the mirrormaker only downloaded
data for some of partitions, and in some cases, it stopped randomly. I will test this scenario again with
beta1.
Regards,
Libo
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Wednesday, July 03, 2013 11:49 AM
To: users@kafka.apache.org
Subject: Re: partition numbers and mirrormaker
Yes, but you can change num.partitions in the target brokers.
Thanks,
Jun
On Wed, Jul 3, 2013 at 6:21 AM, Yu, Libo <li...@citi.com> wrote:
> For a topic with N partitions on the source cluster, if I let
> mirrormaker to create topic automatically, the same topic on the
> destination cluster will have only one partition.
> Is that expected behavior? Could you just give a yes/no answer? Thanks.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> Sent: Tuesday, July 02, 2013 5:23 PM
> To: users@kafka.apache.org
> Subject: Re: partition numbers and mirrormaker
>
> You can override the default number of partitions on the source and
> target cluster to N. (num.partitions in the broker config). If you
> need a higher (or lower) number of partitions for specific topics you
> can override that on a per-topic basis (again, the config can be kept
> identical between the source and target cluster).
>
> Thanks,
>
> Joel
>
> On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <li...@citi.com> wrote:
> > Hi
> >
> > I asked this question previously and didn't get an answer.
> > If a topic has N partitions on source cluster, after being mirrored,
> > on the destination cluster it has only one partition.
> >
> > To avoid this issue, before mirroring, I create the same topic with
> > N partitions on the Destination cluster. But then the mirrormaker
> > cannot fetch all partitions and stop randomly.
> >
> > So what should we expect from mirrormake and what is the right way
> > to
> use it? Thanks.
> >
> >
> > Regards,
> >
> > Libo
> >
>
Re: partition numbers and mirrormaker
Posted by Jun Rao <ju...@gmail.com>.
Yes, but you can change num.partitions in the target brokers.
Thanks,
Jun
On Wed, Jul 3, 2013 at 6:21 AM, Yu, Libo <li...@citi.com> wrote:
> For a topic with N partitions on the source cluster, if I let mirrormaker
> to create topic
> automatically, the same topic on the destination cluster will have only
> one partition.
> Is that expected behavior? Could you just give a yes/no answer? Thanks.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> Sent: Tuesday, July 02, 2013 5:23 PM
> To: users@kafka.apache.org
> Subject: Re: partition numbers and mirrormaker
>
> You can override the default number of partitions on the source and target
> cluster to N. (num.partitions in the broker config). If you need a higher
> (or lower) number of partitions for specific topics you can override that
> on a per-topic basis (again, the config can be kept identical between the
> source and target cluster).
>
> Thanks,
>
> Joel
>
> On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <li...@citi.com> wrote:
> > Hi
> >
> > I asked this question previously and didn't get an answer.
> > If a topic has N partitions on source cluster, after being mirrored,
> > on the destination cluster it has only one partition.
> >
> > To avoid this issue, before mirroring, I create the same topic with N
> > partitions on the Destination cluster. But then the mirrormaker cannot
> > fetch all partitions and stop randomly.
> >
> > So what should we expect from mirrormake and what is the right way to
> use it? Thanks.
> >
> >
> > Regards,
> >
> > Libo
> >
>
RE: partition numbers and mirrormaker
Posted by "Yu, Libo " <li...@citi.com>.
For a topic with N partitions on the source cluster, if I let mirrormaker to create topic
automatically, the same topic on the destination cluster will have only one partition.
Is that expected behavior? Could you just give a yes/no answer? Thanks.
Regards,
Libo
-----Original Message-----
From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
Sent: Tuesday, July 02, 2013 5:23 PM
To: users@kafka.apache.org
Subject: Re: partition numbers and mirrormaker
You can override the default number of partitions on the source and target cluster to N. (num.partitions in the broker config). If you need a higher (or lower) number of partitions for specific topics you can override that on a per-topic basis (again, the config can be kept identical between the source and target cluster).
Thanks,
Joel
On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <li...@citi.com> wrote:
> Hi
>
> I asked this question previously and didn't get an answer.
> If a topic has N partitions on source cluster, after being mirrored,
> on the destination cluster it has only one partition.
>
> To avoid this issue, before mirroring, I create the same topic with N
> partitions on the Destination cluster. But then the mirrormaker cannot
> fetch all partitions and stop randomly.
>
> So what should we expect from mirrormake and what is the right way to use it? Thanks.
>
>
> Regards,
>
> Libo
>
Re: partition numbers and mirrormaker
Posted by Joel Koshy <jj...@gmail.com>.
You can override the default number of partitions on the source and
target cluster to N. (num.partitions in the broker config). If you
need a higher (or lower) number of partitions for specific topics you
can override that on a per-topic basis (again, the config can be kept
identical between the source and target cluster).
Thanks,
Joel
On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <li...@citi.com> wrote:
> Hi
>
> I asked this question previously and didn't get an answer.
> If a topic has N partitions on source cluster, after being mirrored, on the destination
> cluster it has only one partition.
>
> To avoid this issue, before mirroring, I create the same topic with N partitions on the
> Destination cluster. But then the mirrormaker cannot fetch all partitions and stop
> randomly.
>
> So what should we expect from mirrormake and what is the right way to use it? Thanks.
>
>
> Regards,
>
> Libo
>