You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Todd Palino <tp...@gmail.com> on 2015/03/05 03:44:11 UTC
Re: [DISCUSS] KIP-6 - New reassignment partition logic for re-balancing
Apologize for the late comment on this...
So fair assignment by count (taking into account the current partition
count of each broker) is very good. However, it's worth noting that all
partitions are not created equal. We have actually been performing more
rebalance work based on the partition size on disk, as given equal
retention of all topics, the size on disk is a better indicator of the
amount of traffic a partition gets, both in terms of storage and network
traffic. Overall, this seems to be a better balance.
In addition to this, I think there is very much a need to have Kafka be
rack-aware. That is, to be able to assure that for a given cluster, you
never assign all replicas for a given partition in the same rack. This
would allow us to guard against maintenances or power failures that affect
a full rack of systems (or a given switch).
I think it would make sense to implement the reassignment logic as a
pluggable component. That way it would be easy to select a scheme when
performing a reassignment (count, size, rack aware). Configuring a default
scheme for a cluster would allow for the brokers to create new topics and
partitions in compliance with the requested policy.
-Todd
On Thu, Jan 22, 2015 at 10:13 PM, Joe Stein <jo...@stealth.ly> wrote:
> I will go back through the ticket and code and write more up. Should be
> able to-do that sometime next week. The intention was to not replace
> existing functionality by issue a WARN on use. The following version it is
> released we could then deprecate it... I will fix the KIP for that too.
>
> On Fri, Jan 23, 2015 at 12:34 AM, Neha Narkhede <ne...@confluent.io> wrote:
>
> > Hey Joe,
> >
> > 1. Could you add details to the Public Interface section of the KIP? This
> > should include the proposed changes to the partition reassignment tool.
> > Also, maybe the new option can be named --rebalance instead of
> > --re-balance?
> > 2. It makes sense to list --decommission-broker as part of this KIP.
> > Similarly, shouldn't we also have an --add-broker option? The way I see
> > this is that there are several events when a partition reassignment is
> > required. Before this functionality is automated on the broker, the tool
> > will generate an ideal replica placement for each such event. The users
> > should merely have to specify the nature of the event e.g. adding a
> broker
> > or decommissioning an existing broker or merely rebalancing.
> > 3. If I understand the KIP correctly, the upgrade plan for this feature
> > includes removing the existing --generate option on the partition
> > reassignment tool in 0.8.3 while adding all the new options in the same
> > release. Is that correct?
> >
> > Thanks,
> > Neha
> >
> > On Thu, Jan 22, 2015 at 9:23 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > Ditto on this one. Can you give the algorithm we want to implement?
> > >
> > > Also I think in terms of scope this is just proposing to change the
> logic
> > > in ReassignPartitionsCommand? I think we've had the discussion various
> > > times on the mailing list that what people really want is just for
> Kafka
> > to
> > > do it's best to balance data in an online fashion (for some definition
> of
> > > balance). i.e. if you add a new node partitions would slowly migrate to
> > it,
> > > and if a node dies, partitions slowly migrate off it. This could
> > > potentially be more work, but I'm not sure how much more. Has anyone
> > > thought about how to do it?
> > >
> > > -Jay
> > >
> > > On Wed, Jan 21, 2015 at 10:11 PM, Joe Stein <jo...@stealth.ly>
> > wrote:
> > >
> > > > Posted a KIP for --re-balance for partition assignment in
> reassignment
> > > > tool.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+re-balancing
> > > >
> > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1792
> > > >
> > > > While going through the KIP I thought of one thing from the JIRA that
> > we
> > > > should change. We should preserve --generate to be existing
> > functionality
> > > > for the next release it is in. If folks want to use --re-balance then
> > > > great, it just won't break any upgrade paths, yet.
> > > >
> > > > /*******************************************
> > > > Joe Stein
> > > > Founder, Principal Consultant
> > > > Big Data Open Source Security LLC
> > > > http://www.stealth.ly
> > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > > ********************************************/
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Neha
> >
>
RE: [DISCUSS] KIP-6 - New reassignment partition logic for
re-balancing
Posted by Aditya Auradkar <aa...@linkedin.com.INVALID>.
Thanks for the writeup and RB Joe/Dmitry.
Is it possible to describe the proposed partition reassignment algorithm in more detail on the KIP? In fact, it would be really easy to understand if we had some concrete examples comparing partition assignment with the old algorithm and the new.
Aditya
________________________________________
From: Tong Li [litong01@us.ibm.com]
Sent: Wednesday, March 04, 2015 7:33 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-6 - New reassignment partition logic for re-balancing
Todd,
I think plugable design is good with solid default. The only issue I
feel is when you use one and switch to another, will we end up with some
unread messages hanging around and no one thinks or knows it is their
responsibility to take care of them?
Thanks.
Tong
Sent from my iPhone
> On Mar 5, 2015, at 10:46 AM, Todd Palino <tp...@gmail.com> wrote:
>
> Apologize for the late comment on this...
>
> So fair assignment by count (taking into account the current partition
> count of each broker) is very good. However, it's worth noting that all
> partitions are not created equal. We have actually been performing more
> rebalance work based on the partition size on disk, as given equal
> retention of all topics, the size on disk is a better indicator of the
> amount of traffic a partition gets, both in terms of storage and network
> traffic. Overall, this seems to be a better balance.
>
> In addition to this, I think there is very much a need to have Kafka be
> rack-aware. That is, to be able to assure that for a given cluster, you
> never assign all replicas for a given partition in the same rack. This
> would allow us to guard against maintenances or power failures that
affect
> a full rack of systems (or a given switch).
>
> I think it would make sense to implement the reassignment logic as a
> pluggable component. That way it would be easy to select a scheme when
> performing a reassignment (count, size, rack aware). Configuring a
default
> scheme for a cluster would allow for the brokers to create new topics and
> partitions in compliance with the requested policy.
>
> -Todd
>
>
> On Thu, Jan 22, 2015 at 10:13 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > I will go back through the ticket and code and write more up. Should be
> > able to-do that sometime next week. The intention was to not replace
> > existing functionality by issue a WARN on use. The following version it
is
> > released we could then deprecate it... I will fix the KIP for that too.
> >
> > On Fri, Jan 23, 2015 at 12:34 AM, Neha Narkhede <ne...@confluent.io>
wrote:
> >
> > > Hey Joe,
> > >
> > > 1. Could you add details to the Public Interface section of the KIP?
This
> > > should include the proposed changes to the partition reassignment
tool.
> > > Also, maybe the new option can be named --rebalance instead of
> > > --re-balance?
> > > 2. It makes sense to list --decommission-broker as part of this KIP.
> > > Similarly, shouldn't we also have an --add-broker option? The way I
see
> > > this is that there are several events when a partition reassignment
is
> > > required. Before this functionality is automated on the broker, the
tool
> > > will generate an ideal replica placement for each such event. The
users
> > > should merely have to specify the nature of the event e.g. adding a
> > broker
> > > or decommissioning an existing broker or merely rebalancing.
> > > 3. If I understand the KIP correctly, the upgrade plan for this
feature
> > > includes removing the existing --generate option on the partition
> > > reassignment tool in 0.8.3 while adding all the new options in the
same
> > > release. Is that correct?
> > >
> > > Thanks,
> > > Neha
> > >
> > > On Thu, Jan 22, 2015 at 9:23 PM, Jay Kreps <ja...@gmail.com>
wrote:
> > >
> > > > Ditto on this one. Can you give the algorithm we want to implement?
> > > >
> > > > Also I think in terms of scope this is just proposing to change the
> > logic
> > > > in ReassignPartitionsCommand? I think we've had the discussion
various
> > > > times on the mailing list that what people really want is just for
> > Kafka
> > > to
> > > > do it's best to balance data in an online fashion (for some
definition
> > of
> > > > balance). i.e. if you add a new node partitions would slowly
migrate to
> > > it,
> > > > and if a node dies, partitions slowly migrate off it. This could
> > > > potentially be more work, but I'm not sure how much more. Has
anyone
> > > > thought about how to do it?
> > > >
> > > > -Jay
> > > >
> > > > On Wed, Jan 21, 2015 at 10:11 PM, Joe Stein <jo...@stealth.ly>
> > > wrote:
> > > >
> > > > > Posted a KIP for --re-balance for partition assignment in
> > reassignment
> > > > > tool.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New
+reassignment+partition+logic+for+re-balancing
> > > > >
> > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1792
> > > > >
> > > > > While going through the KIP I thought of one thing from the JIRA
that
> > > we
> > > > > should change. We should preserve --generate to be existing
> > > functionality
> > > > > for the next release it is in. If folks want to use --re-balance
then
> > > > > great, it just won't break any upgrade paths, yet.
> > > > >
> > > > > /*******************************************
> > > > > Joe Stein
> > > > > Founder, Principal Consultant
> > > > > Big Data Open Source Security LLC
> > > > > http://www.stealth.ly
> > > > > Twitter: @allthingshadoop
<http://www.twitter.com/allthingshadoop>
> > > > > ********************************************/
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Neha
> > >
> >
Re: [DISCUSS] KIP-6 - New reassignment partition logic for re-balancing
Posted by Guozhang Wang <wa...@gmail.com>.
I am +1 Todd's suggestion, the default reassignment scheme is only used
when a reassignment command is issued with no scheme specified, and
changing this default scheme should not automatically trigger a
reassignment of all existing topics: it will only take effect when the next
reassignment command with no specific scheme is issued.
On Thu, Mar 5, 2015 at 10:16 AM, Todd Palino <tp...@gmail.com> wrote:
> I would not think that partitions moving would cause any orphaned messages
> like that. I would be more concerned about what happens when you change the
> default on a running cluster from one scheme to another. Would we want to
> support some kind of automated reassignment of existing partitions
> (personally - no. I want to trigger that manually because it is a very disk
> and network intensive process)?
>
> -Todd
>
> On Wed, Mar 4, 2015 at 7:33 PM, Tong Li <li...@us.ibm.com> wrote:
>
> >
> >
> > Todd,
> > I think plugable design is good with solid default. The only issue I
> > feel is when you use one and switch to another, will we end up with some
> > unread messages hanging around and no one thinks or knows it is their
> > responsibility to take care of them?
> >
> > Thanks.
> >
> > Tong
> >
> > Sent from my iPhone
> >
> > > On Mar 5, 2015, at 10:46 AM, Todd Palino <tp...@gmail.com> wrote:
> > >
> > > Apologize for the late comment on this...
> > >
> > > So fair assignment by count (taking into account the current partition
> > > count of each broker) is very good. However, it's worth noting that all
> > > partitions are not created equal. We have actually been performing more
> > > rebalance work based on the partition size on disk, as given equal
> > > retention of all topics, the size on disk is a better indicator of the
> > > amount of traffic a partition gets, both in terms of storage and
> network
> > > traffic. Overall, this seems to be a better balance.
> > >
> > > In addition to this, I think there is very much a need to have Kafka be
> > > rack-aware. That is, to be able to assure that for a given cluster, you
> > > never assign all replicas for a given partition in the same rack. This
> > > would allow us to guard against maintenances or power failures that
> > affect
> > > a full rack of systems (or a given switch).
> > >
> > > I think it would make sense to implement the reassignment logic as a
> > > pluggable component. That way it would be easy to select a scheme when
> > > performing a reassignment (count, size, rack aware). Configuring a
> > default
> > > scheme for a cluster would allow for the brokers to create new topics
> and
> > > partitions in compliance with the requested policy.
> > >
> > > -Todd
> > >
> > >
> > > On Thu, Jan 22, 2015 at 10:13 PM, Joe Stein <jo...@stealth.ly>
> > wrote:
> > >
> > > > I will go back through the ticket and code and write more up. Should
> be
> > > > able to-do that sometime next week. The intention was to not replace
> > > > existing functionality by issue a WARN on use. The following version
> it
> > is
> > > > released we could then deprecate it... I will fix the KIP for that
> too.
> > > >
> > > > On Fri, Jan 23, 2015 at 12:34 AM, Neha Narkhede <ne...@confluent.io>
> > wrote:
> > > >
> > > > > Hey Joe,
> > > > >
> > > > > 1. Could you add details to the Public Interface section of the
> KIP?
> > This
> > > > > should include the proposed changes to the partition reassignment
> > tool.
> > > > > Also, maybe the new option can be named --rebalance instead of
> > > > > --re-balance?
> > > > > 2. It makes sense to list --decommission-broker as part of this
> KIP.
> > > > > Similarly, shouldn't we also have an --add-broker option? The way I
> > see
> > > > > this is that there are several events when a partition reassignment
> > is
> > > > > required. Before this functionality is automated on the broker, the
> > tool
> > > > > will generate an ideal replica placement for each such event. The
> > users
> > > > > should merely have to specify the nature of the event e.g. adding a
> > > > broker
> > > > > or decommissioning an existing broker or merely rebalancing.
> > > > > 3. If I understand the KIP correctly, the upgrade plan for this
> > feature
> > > > > includes removing the existing --generate option on the partition
> > > > > reassignment tool in 0.8.3 while adding all the new options in the
> > same
> > > > > release. Is that correct?
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > > On Thu, Jan 22, 2015 at 9:23 PM, Jay Kreps <ja...@gmail.com>
> > wrote:
> > > > >
> > > > > > Ditto on this one. Can you give the algorithm we want to
> implement?
> > > > > >
> > > > > > Also I think in terms of scope this is just proposing to change
> the
> > > > logic
> > > > > > in ReassignPartitionsCommand? I think we've had the discussion
> > various
> > > > > > times on the mailing list that what people really want is just
> for
> > > > Kafka
> > > > > to
> > > > > > do it's best to balance data in an online fashion (for some
> > definition
> > > > of
> > > > > > balance). i.e. if you add a new node partitions would slowly
> > migrate to
> > > > > it,
> > > > > > and if a node dies, partitions slowly migrate off it. This could
> > > > > > potentially be more work, but I'm not sure how much more. Has
> > anyone
> > > > > > thought about how to do it?
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > > On Wed, Jan 21, 2015 at 10:11 PM, Joe Stein <
> joe.stein@stealth.ly>
> > > > > wrote:
> > > > > >
> > > > > > > Posted a KIP for --re-balance for partition assignment in
> > > > reassignment
> > > > > > > tool.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New
> > +reassignment+partition+logic+for+re-balancing
> > > > > > >
> > > > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1792
> > > > > > >
> > > > > > > While going through the KIP I thought of one thing from the
> JIRA
> > that
> > > > > we
> > > > > > > should change. We should preserve --generate to be existing
> > > > > functionality
> > > > > > > for the next release it is in. If folks want to use
> --re-balance
> > then
> > > > > > > great, it just won't break any upgrade paths, yet.
> > > > > > >
> > > > > > > /*******************************************
> > > > > > > Joe Stein
> > > > > > > Founder, Principal Consultant
> > > > > > > Big Data Open Source Security LLC
> > > > > > > http://www.stealth.ly
> > > > > > > Twitter: @allthingshadoop
> > <http://www.twitter.com/allthingshadoop>
> > > > > > > ********************************************/
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > >
> >
>
--
-- Guozhang
Re: [DISCUSS] KIP-6 - New reassignment partition logic for re-balancing
Posted by Todd Palino <tp...@gmail.com>.
I would not think that partitions moving would cause any orphaned messages
like that. I would be more concerned about what happens when you change the
default on a running cluster from one scheme to another. Would we want to
support some kind of automated reassignment of existing partitions
(personally - no. I want to trigger that manually because it is a very disk
and network intensive process)?
-Todd
On Wed, Mar 4, 2015 at 7:33 PM, Tong Li <li...@us.ibm.com> wrote:
>
>
> Todd,
> I think plugable design is good with solid default. The only issue I
> feel is when you use one and switch to another, will we end up with some
> unread messages hanging around and no one thinks or knows it is their
> responsibility to take care of them?
>
> Thanks.
>
> Tong
>
> Sent from my iPhone
>
> > On Mar 5, 2015, at 10:46 AM, Todd Palino <tp...@gmail.com> wrote:
> >
> > Apologize for the late comment on this...
> >
> > So fair assignment by count (taking into account the current partition
> > count of each broker) is very good. However, it's worth noting that all
> > partitions are not created equal. We have actually been performing more
> > rebalance work based on the partition size on disk, as given equal
> > retention of all topics, the size on disk is a better indicator of the
> > amount of traffic a partition gets, both in terms of storage and network
> > traffic. Overall, this seems to be a better balance.
> >
> > In addition to this, I think there is very much a need to have Kafka be
> > rack-aware. That is, to be able to assure that for a given cluster, you
> > never assign all replicas for a given partition in the same rack. This
> > would allow us to guard against maintenances or power failures that
> affect
> > a full rack of systems (or a given switch).
> >
> > I think it would make sense to implement the reassignment logic as a
> > pluggable component. That way it would be easy to select a scheme when
> > performing a reassignment (count, size, rack aware). Configuring a
> default
> > scheme for a cluster would allow for the brokers to create new topics and
> > partitions in compliance with the requested policy.
> >
> > -Todd
> >
> >
> > On Thu, Jan 22, 2015 at 10:13 PM, Joe Stein <jo...@stealth.ly>
> wrote:
> >
> > > I will go back through the ticket and code and write more up. Should be
> > > able to-do that sometime next week. The intention was to not replace
> > > existing functionality by issue a WARN on use. The following version it
> is
> > > released we could then deprecate it... I will fix the KIP for that too.
> > >
> > > On Fri, Jan 23, 2015 at 12:34 AM, Neha Narkhede <ne...@confluent.io>
> wrote:
> > >
> > > > Hey Joe,
> > > >
> > > > 1. Could you add details to the Public Interface section of the KIP?
> This
> > > > should include the proposed changes to the partition reassignment
> tool.
> > > > Also, maybe the new option can be named --rebalance instead of
> > > > --re-balance?
> > > > 2. It makes sense to list --decommission-broker as part of this KIP.
> > > > Similarly, shouldn't we also have an --add-broker option? The way I
> see
> > > > this is that there are several events when a partition reassignment
> is
> > > > required. Before this functionality is automated on the broker, the
> tool
> > > > will generate an ideal replica placement for each such event. The
> users
> > > > should merely have to specify the nature of the event e.g. adding a
> > > broker
> > > > or decommissioning an existing broker or merely rebalancing.
> > > > 3. If I understand the KIP correctly, the upgrade plan for this
> feature
> > > > includes removing the existing --generate option on the partition
> > > > reassignment tool in 0.8.3 while adding all the new options in the
> same
> > > > release. Is that correct?
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > > On Thu, Jan 22, 2015 at 9:23 PM, Jay Kreps <ja...@gmail.com>
> wrote:
> > > >
> > > > > Ditto on this one. Can you give the algorithm we want to implement?
> > > > >
> > > > > Also I think in terms of scope this is just proposing to change the
> > > logic
> > > > > in ReassignPartitionsCommand? I think we've had the discussion
> various
> > > > > times on the mailing list that what people really want is just for
> > > Kafka
> > > > to
> > > > > do it's best to balance data in an online fashion (for some
> definition
> > > of
> > > > > balance). i.e. if you add a new node partitions would slowly
> migrate to
> > > > it,
> > > > > and if a node dies, partitions slowly migrate off it. This could
> > > > > potentially be more work, but I'm not sure how much more. Has
> anyone
> > > > > thought about how to do it?
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Wed, Jan 21, 2015 at 10:11 PM, Joe Stein <jo...@stealth.ly>
> > > > wrote:
> > > > >
> > > > > > Posted a KIP for --re-balance for partition assignment in
> > > reassignment
> > > > > > tool.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New
> +reassignment+partition+logic+for+re-balancing
> > > > > >
> > > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1792
> > > > > >
> > > > > > While going through the KIP I thought of one thing from the JIRA
> that
> > > > we
> > > > > > should change. We should preserve --generate to be existing
> > > > functionality
> > > > > > for the next release it is in. If folks want to use --re-balance
> then
> > > > > > great, it just won't break any upgrade paths, yet.
> > > > > >
> > > > > > /*******************************************
> > > > > > Joe Stein
> > > > > > Founder, Principal Consultant
> > > > > > Big Data Open Source Security LLC
> > > > > > http://www.stealth.ly
> > > > > > Twitter: @allthingshadoop
> <http://www.twitter.com/allthingshadoop>
> > > > > > ********************************************/
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Neha
> > > >
> > >
>
Re: [DISCUSS] KIP-6 - New reassignment partition logic for re-balancing
Posted by Tong Li <li...@us.ibm.com>.
Todd,
I think plugable design is good with solid default. The only issue I
feel is when you use one and switch to another, will we end up with some
unread messages hanging around and no one thinks or knows it is their
responsibility to take care of them?
Thanks.
Tong
Sent from my iPhone
> On Mar 5, 2015, at 10:46 AM, Todd Palino <tp...@gmail.com> wrote:
>
> Apologize for the late comment on this...
>
> So fair assignment by count (taking into account the current partition
> count of each broker) is very good. However, it's worth noting that all
> partitions are not created equal. We have actually been performing more
> rebalance work based on the partition size on disk, as given equal
> retention of all topics, the size on disk is a better indicator of the
> amount of traffic a partition gets, both in terms of storage and network
> traffic. Overall, this seems to be a better balance.
>
> In addition to this, I think there is very much a need to have Kafka be
> rack-aware. That is, to be able to assure that for a given cluster, you
> never assign all replicas for a given partition in the same rack. This
> would allow us to guard against maintenances or power failures that
affect
> a full rack of systems (or a given switch).
>
> I think it would make sense to implement the reassignment logic as a
> pluggable component. That way it would be easy to select a scheme when
> performing a reassignment (count, size, rack aware). Configuring a
default
> scheme for a cluster would allow for the brokers to create new topics and
> partitions in compliance with the requested policy.
>
> -Todd
>
>
> On Thu, Jan 22, 2015 at 10:13 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > I will go back through the ticket and code and write more up. Should be
> > able to-do that sometime next week. The intention was to not replace
> > existing functionality by issue a WARN on use. The following version it
is
> > released we could then deprecate it... I will fix the KIP for that too.
> >
> > On Fri, Jan 23, 2015 at 12:34 AM, Neha Narkhede <ne...@confluent.io>
wrote:
> >
> > > Hey Joe,
> > >
> > > 1. Could you add details to the Public Interface section of the KIP?
This
> > > should include the proposed changes to the partition reassignment
tool.
> > > Also, maybe the new option can be named --rebalance instead of
> > > --re-balance?
> > > 2. It makes sense to list --decommission-broker as part of this KIP.
> > > Similarly, shouldn't we also have an --add-broker option? The way I
see
> > > this is that there are several events when a partition reassignment
is
> > > required. Before this functionality is automated on the broker, the
tool
> > > will generate an ideal replica placement for each such event. The
users
> > > should merely have to specify the nature of the event e.g. adding a
> > broker
> > > or decommissioning an existing broker or merely rebalancing.
> > > 3. If I understand the KIP correctly, the upgrade plan for this
feature
> > > includes removing the existing --generate option on the partition
> > > reassignment tool in 0.8.3 while adding all the new options in the
same
> > > release. Is that correct?
> > >
> > > Thanks,
> > > Neha
> > >
> > > On Thu, Jan 22, 2015 at 9:23 PM, Jay Kreps <ja...@gmail.com>
wrote:
> > >
> > > > Ditto on this one. Can you give the algorithm we want to implement?
> > > >
> > > > Also I think in terms of scope this is just proposing to change the
> > logic
> > > > in ReassignPartitionsCommand? I think we've had the discussion
various
> > > > times on the mailing list that what people really want is just for
> > Kafka
> > > to
> > > > do it's best to balance data in an online fashion (for some
definition
> > of
> > > > balance). i.e. if you add a new node partitions would slowly
migrate to
> > > it,
> > > > and if a node dies, partitions slowly migrate off it. This could
> > > > potentially be more work, but I'm not sure how much more. Has
anyone
> > > > thought about how to do it?
> > > >
> > > > -Jay
> > > >
> > > > On Wed, Jan 21, 2015 at 10:11 PM, Joe Stein <jo...@stealth.ly>
> > > wrote:
> > > >
> > > > > Posted a KIP for --re-balance for partition assignment in
> > reassignment
> > > > > tool.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New
+reassignment+partition+logic+for+re-balancing
> > > > >
> > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1792
> > > > >
> > > > > While going through the KIP I thought of one thing from the JIRA
that
> > > we
> > > > > should change. We should preserve --generate to be existing
> > > functionality
> > > > > for the next release it is in. If folks want to use --re-balance
then
> > > > > great, it just won't break any upgrade paths, yet.
> > > > >
> > > > > /*******************************************
> > > > > Joe Stein
> > > > > Founder, Principal Consultant
> > > > > Big Data Open Source Security LLC
> > > > > http://www.stealth.ly
> > > > > Twitter: @allthingshadoop
<http://www.twitter.com/allthingshadoop>
> > > > > ********************************************/
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Neha
> > >
> >