moving replications

[sunil kalva <sa...@gmail.com>]

Hi
How to move replications from one broker to another broker ?

-- 
SunilKalva

Re: moving replications

[Joel Koshy <jj...@gmail.com>]

I think what you may be looking for is being discussed here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+rebalancing

On Wed, Mar 04, 2015 at 12:34:30PM +0530, sunil kalva wrote:
> Is there any way to automate
> On Mar 3, 2015 11:57 AM, "sunil kalva" <sa...@gmail.com> wrote:
> 
> > Why can't kafka automatically rebalances partitions with new broker and
> > adjust with existing brokers ?
> > Why should we run manually ?
> >
> > On Tue, Mar 3, 2015 at 6:41 AM, Gwen Shapira <gs...@cloudera.com>
> > wrote:
> >
> >> I think the ReassignPartitionsTool does what you need, at least partially.
> >>
> >> It will move partitions of given topics to a new set of brokers - this
> >> includes replicas and leaders from what I can tell.
> >>
> >> Here's the documentation of one of the options:
> >>
> >> topics-to-move-json-file: Generate a reassignment configuration to move
> >> the partitions of the specified topics to the list of brokers specified by
> >> the --broker-list option. The format to use is...
> >>
> >> So it sounds like what you looked for?
> >>
> >> Since the tool runs in "dry run" mode, you can try and see without
> >> actually moving anything.
> >>
> >> Gwen
> >>
> >>
> >>
> >> On Mar 2, 2015 10:05 AM, "sunil kalva" <sa...@gmail.com> wrote:
> >>
> >>> Shapira
> >>>
> >>> tx for quick reply, but this tool elects a new leader for a given
> >>> partition from the existing replicas of that partition.
> >>> But my problem is basically move one replica completely from old broker
> >>> to new broker and eventually move leader also to new broker (with out
> >>> incrementing replica count for that partition)
> >>> Please let me know if more information required.
> >>>
> >>> t
> >>> SunilKalva
> >>>
> >>> On Mon, Mar 2, 2015 at 10:43 PM, Gwen Shapira <gs...@cloudera.com>
> >>> wrote:
> >>>
> >>>> Take a look at the Reassign Partition Tool. It lets you specify which
> >>>> replica exists on which broker:
> >>>>
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
> >>>>
> >>>> Its a bit tricky to use, so feel free to follow up with more questions
> >>>> :)
> >>>>
> >>>> Gwen
> >>>>
> >>>> On Mon, Mar 2, 2015 at 7:59 AM, sunil kalva <sa...@gmail.com>
> >>>> wrote:
> >>>> > Hi
> >>>> > How to move replications from one broker to another broker ?
> >>>> >
> >>>> > --
> >>>> > SunilKalva
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> SunilKalva
> >>>
> >>
> >
> >
> > --
> > SunilKalva
> >

Re: moving replications

[sunil kalva <sa...@gmail.com>]

Is there any way to automate
On Mar 3, 2015 11:57 AM, "sunil kalva" <sa...@gmail.com> wrote:

> Why can't kafka automatically rebalances partitions with new broker and
> adjust with existing brokers ?
> Why should we run manually ?
>
> On Tue, Mar 3, 2015 at 6:41 AM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
>> I think the ReassignPartitionsTool does what you need, at least partially.
>>
>> It will move partitions of given topics to a new set of brokers - this
>> includes replicas and leaders from what I can tell.
>>
>> Here's the documentation of one of the options:
>>
>> topics-to-move-json-file: Generate a reassignment configuration to move
>> the partitions of the specified topics to the list of brokers specified by
>> the --broker-list option. The format to use is...
>>
>> So it sounds like what you looked for?
>>
>> Since the tool runs in "dry run" mode, you can try and see without
>> actually moving anything.
>>
>> Gwen
>>
>>
>>
>> On Mar 2, 2015 10:05 AM, "sunil kalva" <sa...@gmail.com> wrote:
>>
>>> Shapira
>>>
>>> tx for quick reply, but this tool elects a new leader for a given
>>> partition from the existing replicas of that partition.
>>> But my problem is basically move one replica completely from old broker
>>> to new broker and eventually move leader also to new broker (with out
>>> incrementing replica count for that partition)
>>> Please let me know if more information required.
>>>
>>> t
>>> SunilKalva
>>>
>>> On Mon, Mar 2, 2015 at 10:43 PM, Gwen Shapira <gs...@cloudera.com>
>>> wrote:
>>>
>>>> Take a look at the Reassign Partition Tool. It lets you specify which
>>>> replica exists on which broker:
>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
>>>>
>>>> Its a bit tricky to use, so feel free to follow up with more questions
>>>> :)
>>>>
>>>> Gwen
>>>>
>>>> On Mon, Mar 2, 2015 at 7:59 AM, sunil kalva <sa...@gmail.com>
>>>> wrote:
>>>> > Hi
>>>> > How to move replications from one broker to another broker ?
>>>> >
>>>> > --
>>>> > SunilKalva
>>>>
>>>
>>>
>>>
>>> --
>>> SunilKalva
>>>
>>
>
>
> --
> SunilKalva
>

Re: moving replications

[sunil kalva <sa...@gmail.com>]

Why can't kafka automatically rebalances partitions with new broker and
adjust with existing brokers ?
Why should we run manually ?

On Tue, Mar 3, 2015 at 6:41 AM, Gwen Shapira <gs...@cloudera.com> wrote:

> I think the ReassignPartitionsTool does what you need, at least partially.
>
> It will move partitions of given topics to a new set of brokers - this
> includes replicas and leaders from what I can tell.
>
> Here's the documentation of one of the options:
>
> topics-to-move-json-file: Generate a reassignment configuration to move
> the partitions of the specified topics to the list of brokers specified by
> the --broker-list option. The format to use is...
>
> So it sounds like what you looked for?
>
> Since the tool runs in "dry run" mode, you can try and see without
> actually moving anything.
>
> Gwen
>
>
>
> On Mar 2, 2015 10:05 AM, "sunil kalva" <sa...@gmail.com> wrote:
>
>> Shapira
>>
>> tx for quick reply, but this tool elects a new leader for a given
>> partition from the existing replicas of that partition.
>> But my problem is basically move one replica completely from old broker
>> to new broker and eventually move leader also to new broker (with out
>> incrementing replica count for that partition)
>> Please let me know if more information required.
>>
>> t
>> SunilKalva
>>
>> On Mon, Mar 2, 2015 at 10:43 PM, Gwen Shapira <gs...@cloudera.com>
>> wrote:
>>
>>> Take a look at the Reassign Partition Tool. It lets you specify which
>>> replica exists on which broker:
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
>>>
>>> Its a bit tricky to use, so feel free to follow up with more questions :)
>>>
>>> Gwen
>>>
>>> On Mon, Mar 2, 2015 at 7:59 AM, sunil kalva <sa...@gmail.com>
>>> wrote:
>>> > Hi
>>> > How to move replications from one broker to another broker ?
>>> >
>>> > --
>>> > SunilKalva
>>>
>>
>>
>>
>> --
>> SunilKalva
>>
>


-- 
SunilKalva

Re: moving replications

[Gwen Shapira <gs...@cloudera.com>]

I think the ReassignPartitionsTool does what you need, at least partially.

It will move partitions of given topics to a new set of brokers - this
includes replicas and leaders from what I can tell.

Here's the documentation of one of the options:

topics-to-move-json-file: Generate a reassignment configuration to move the
partitions of the specified topics to the list of brokers specified by the
--broker-list option. The format to use is...

So it sounds like what you looked for?

Since the tool runs in "dry run" mode, you can try and see without actually
moving anything.

Gwen



On Mar 2, 2015 10:05 AM, "sunil kalva" <sa...@gmail.com> wrote:

> Shapira
>
> tx for quick reply, but this tool elects a new leader for a given
> partition from the existing replicas of that partition.
> But my problem is basically move one replica completely from old broker to
> new broker and eventually move leader also to new broker (with out
> incrementing replica count for that partition)
> Please let me know if more information required.
>
> t
> SunilKalva
>
> On Mon, Mar 2, 2015 at 10:43 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
>> Take a look at the Reassign Partition Tool. It lets you specify which
>> replica exists on which broker:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
>>
>> Its a bit tricky to use, so feel free to follow up with more questions :)
>>
>> Gwen
>>
>> On Mon, Mar 2, 2015 at 7:59 AM, sunil kalva <sa...@gmail.com> wrote:
>> > Hi
>> > How to move replications from one broker to another broker ?
>> >
>> > --
>> > SunilKalva
>>
>
>
>
> --
> SunilKalva
>

Re: moving replications

[sunil kalva <sa...@gmail.com>]

Shapira

tx for quick reply, but this tool elects a new leader for a given partition
from the existing replicas of that partition.
But my problem is basically move one replica completely from old broker to
new broker and eventually move leader also to new broker (with out
incrementing replica count for that partition)
Please let me know if more information required.

t
SunilKalva

On Mon, Mar 2, 2015 at 10:43 PM, Gwen Shapira <gs...@cloudera.com> wrote:

> Take a look at the Reassign Partition Tool. It lets you specify which
> replica exists on which broker:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
>
> Its a bit tricky to use, so feel free to follow up with more questions :)
>
> Gwen
>
> On Mon, Mar 2, 2015 at 7:59 AM, sunil kalva <sa...@gmail.com> wrote:
> > Hi
> > How to move replications from one broker to another broker ?
> >
> > --
> > SunilKalva
>



-- 
SunilKalva

Re: moving replications

[Gwen Shapira <gs...@cloudera.com>]

Take a look at the Reassign Partition Tool. It lets you specify which
replica exists on which broker:
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool

Its a bit tricky to use, so feel free to follow up with more questions :)

Gwen

On Mon, Mar 2, 2015 at 7:59 AM, sunil kalva <sa...@gmail.com> wrote:
> Hi
> How to move replications from one broker to another broker ?
>
> --
> SunilKalva