You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Henry Cai <hc...@pinterest.com.INVALID> on 2015/05/25 04:45:56 UTC
Replication tools to move topics/partitions gradually
We have a kafka cluster with 10 brokers and we are using the kafka
replication tool (kafka-reassign-partitions.sh) when we need to add more
brokers to the cluster. But this tool tends to move too many
topic/partitions around at the same time which causes instability. Do we
have an option to do it more slowly (e.g. move one topic/partition at a
step) or did some one build a tool on top of 'kafka-reassign-partitions.sh'?
Another use case is when a broker node went down, do we have a tool to move
the topic/partitions serviced by this node to the remaining nodes (and
doing that in a fashion which doesn't cause too much instability)?
Re: Replication tools to move topics/partitions gradually
Posted by Henry Cai <hc...@pinterest.com.INVALID>.
Thanks Todd. Let us know when these tools are announced.
On Sun, May 24, 2015 at 11:00 PM, Todd Palino <tp...@gmail.com> wrote:
> See, this is why I should never say anything :)
>
> The version I have right now is very limited - it only does a clone (we
> needed it for some hardware testing) and a leader balance (does it using
> partition reassignment without actually moving partitions). We have some
> scripts that the other SREs have written that need to be incorporated which
> do partition balancing by count and by size.
>
> Let me figure out where we are going to host this, but we can probably put
> out the script we have in the next couple weeks. Then we can just iterate
> on it.
>
> -Todd
>
> > On May 24, 2015, at 8:46 PM, Henry Cai <hc...@pinterest.com.INVALID>
> wrote:
> >
> > Todd,
> >
> > This is very promising. Do you know when will we be able to see your
> tools
> > released to public?
> >
> >> On Sun, May 24, 2015 at 7:54 PM, Todd Palino <tp...@gmail.com> wrote:
> >>
> >> We've built tools on top of it that both build the list based on less
> >> information (like "clone this broker to that one") and break it down
> into a
> >> configurable number of discrete moves so it doesn't tank the cluster.
> >>
> >> And yes, I've finally started the process of departing them from the
> >> LinkedIn-specific tooling so we can release them to everyone else :)
> >>
> >> -Todd
> >>
> >>>> On May 24, 2015, at 7:45 PM, Henry Cai <hc...@pinterest.com.INVALID>
> >>> wrote:
> >>>
> >>> We have a kafka cluster with 10 brokers and we are using the kafka
> >>> replication tool (kafka-reassign-partitions.sh) when we need to add
> more
> >>> brokers to the cluster. But this tool tends to move too many
> >>> topic/partitions around at the same time which causes instability. Do
> we
> >>> have an option to do it more slowly (e.g. move one topic/partition at a
> >>> step) or did some one build a tool on top of
> >> 'kafka-reassign-partitions.sh'?
> >>>
> >>> Another use case is when a broker node went down, do we have a tool to
> >> move
> >>> the topic/partitions serviced by this node to the remaining nodes (and
> >>> doing that in a fashion which doesn't cause too much instability)?
> >>
>
Re: Replication tools to move topics/partitions gradually
Posted by Todd Palino <tp...@gmail.com>.
See, this is why I should never say anything :)
The version I have right now is very limited - it only does a clone (we needed it for some hardware testing) and a leader balance (does it using partition reassignment without actually moving partitions). We have some scripts that the other SREs have written that need to be incorporated which do partition balancing by count and by size.
Let me figure out where we are going to host this, but we can probably put out the script we have in the next couple weeks. Then we can just iterate on it.
-Todd
> On May 24, 2015, at 8:46 PM, Henry Cai <hc...@pinterest.com.INVALID> wrote:
>
> Todd,
>
> This is very promising. Do you know when will we be able to see your tools
> released to public?
>
>> On Sun, May 24, 2015 at 7:54 PM, Todd Palino <tp...@gmail.com> wrote:
>>
>> We've built tools on top of it that both build the list based on less
>> information (like "clone this broker to that one") and break it down into a
>> configurable number of discrete moves so it doesn't tank the cluster.
>>
>> And yes, I've finally started the process of departing them from the
>> LinkedIn-specific tooling so we can release them to everyone else :)
>>
>> -Todd
>>
>>>> On May 24, 2015, at 7:45 PM, Henry Cai <hc...@pinterest.com.INVALID>
>>> wrote:
>>>
>>> We have a kafka cluster with 10 brokers and we are using the kafka
>>> replication tool (kafka-reassign-partitions.sh) when we need to add more
>>> brokers to the cluster. But this tool tends to move too many
>>> topic/partitions around at the same time which causes instability. Do we
>>> have an option to do it more slowly (e.g. move one topic/partition at a
>>> step) or did some one build a tool on top of
>> 'kafka-reassign-partitions.sh'?
>>>
>>> Another use case is when a broker node went down, do we have a tool to
>> move
>>> the topic/partitions serviced by this node to the remaining nodes (and
>>> doing that in a fashion which doesn't cause too much instability)?
>>
Re: Replication tools to move topics/partitions gradually
Posted by Henry Cai <hc...@pinterest.com.INVALID>.
Todd,
This is very promising. Do you know when will we be able to see your tools
released to public?
On Sun, May 24, 2015 at 7:54 PM, Todd Palino <tp...@gmail.com> wrote:
> We've built tools on top of it that both build the list based on less
> information (like "clone this broker to that one") and break it down into a
> configurable number of discrete moves so it doesn't tank the cluster.
>
> And yes, I've finally started the process of departing them from the
> LinkedIn-specific tooling so we can release them to everyone else :)
>
> -Todd
>
> > On May 24, 2015, at 7:45 PM, Henry Cai <hc...@pinterest.com.INVALID>
> wrote:
> >
> > We have a kafka cluster with 10 brokers and we are using the kafka
> > replication tool (kafka-reassign-partitions.sh) when we need to add more
> > brokers to the cluster. But this tool tends to move too many
> > topic/partitions around at the same time which causes instability. Do we
> > have an option to do it more slowly (e.g. move one topic/partition at a
> > step) or did some one build a tool on top of
> 'kafka-reassign-partitions.sh'?
> >
> > Another use case is when a broker node went down, do we have a tool to
> move
> > the topic/partitions serviced by this node to the remaining nodes (and
> > doing that in a fashion which doesn't cause too much instability)?
>
Re: Replication tools to move topics/partitions gradually
Posted by Todd Palino <tp...@gmail.com>.
We've built tools on top of it that both build the list based on less information (like "clone this broker to that one") and break it down into a configurable number of discrete moves so it doesn't tank the cluster.
And yes, I've finally started the process of departing them from the LinkedIn-specific tooling so we can release them to everyone else :)
-Todd
> On May 24, 2015, at 7:45 PM, Henry Cai <hc...@pinterest.com.INVALID> wrote:
>
> We have a kafka cluster with 10 brokers and we are using the kafka
> replication tool (kafka-reassign-partitions.sh) when we need to add more
> brokers to the cluster. But this tool tends to move too many
> topic/partitions around at the same time which causes instability. Do we
> have an option to do it more slowly (e.g. move one topic/partition at a
> step) or did some one build a tool on top of 'kafka-reassign-partitions.sh'?
>
> Another use case is when a broker node went down, do we have a tool to move
> the topic/partitions serviced by this node to the remaining nodes (and
> doing that in a fashion which doesn't cause too much instability)?