Kafka Cluster Upgrade/Migration

[Bryan Baugher <bj...@gmail.com>]

Hi everyone,

I'm starting to look at how you might upgrade your cluster even if a major
upgrade with non-passive API changes and do this all in uptime (i.e. do not
read/write). I realize this may involve knowledge in how Kafka is used, in
our case we own all the reading and writing for now.

Has anyone done this? What tools if any did you use? What was the general
idea?

If Kafka did make non-passive changes can we expect mirror maker or some
migration tool equivalent?

Has anyone tried the confluent REST proxy? Are there any clients for it?

Adding or removing a broker from a cluster seems likely in this case. The
re-assign partitions command seems like it can do it and may support
generating the partition assignments via topics-to-move-json-file and
broker-list options. Has anyone done this? Would this work for
adding/removing brokers from a cluster?

Bryan

Re: reassign a topic partition which has no ISR and leader set to -1

[to...@borked.ca]

When we ran in to this problem we ended up going in to zookeeper and changing the leader to point to one of the replicas, then did a force leader election. This got the partition back online.


  Original Message  
From: Virendra Pratap Singh
Sent: Wednesday, March 4, 2015 2:00 AM
To: Gwen Shapira; users@kafka.apache.org
Reply To: users@kafka.apache.org
Subject: Re: reassign a topic partition which has no ISR and leader set to -1

Thanks Gwen for your info. I brought another kafka server up with assigned broker id of the failed leader and the reassignment went through.
@kafka developers who may be on this distribution list, is there a feature planned in 0.8.2 or can we plan one, where in if the leader and all the replica brokers are lost, the manual assignment of such a partition can be performed. As of current implementation, it looks like the reassignment will not budge till it sees at least one of the leader/replica brokers back again. 
Regards,Virendra From: Gwen Shapira <gs...@cloudera.com>
To: "users@kafka.apache.org" <us...@kafka.apache.org>; Virendra Pratap Singh <vp...@yahoo-inc.com> 
Sent: Tuesday, March 3, 2015 4:29 PM
Subject: Re: reassign a topic partition which has no ISR and leader set to -1

I hate bringing bad news, but...

You can't really reassign replicas if the leader is not available.
Since the leader is gone, the replicas have no where to replicate the data from.

Until you bring the leader back (or one of the replicas with unclean
leader election), you basically lost this partition.

Gwen



On Tue, Mar 3, 2015 at 3:57 PM, Virendra Pratap Singh
<vp...@yahoo-inc.com.invalid> wrote:
> Ran into a situation where both the leader and replica nodes for a few partitions of a given topic went down. So now these partitions have no in-sync replicas and neither any leader (leader set to -1).
> I tried to reassign these partitions to a different set of brokers using partition reassignment tool and then running the preferred replica. However I don't see the partitions getting reassigned and the leader still value still remains "-1" for these.
> What do I need to get this working.
> Kafka cluster is on 0.8.1.1
> Regards,Virendra

Re: reassign a topic partition which has no ISR and leader set to -1

[Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>]

Thanks Gwen for your info. I brought another kafka server up with assigned broker id of the failed leader and the reassignment went through.
@kafka developers who may be on this distribution list, is there a feature planned in 0.8.2 or can we plan one, where in if the leader and all the replica brokers are lost, the manual assignment of such a partition can be performed. As of current implementation, it looks like the reassignment will not budge till it sees at least one of the leader/replica brokers back again. 
Regards,Virendra      From: Gwen Shapira <gs...@cloudera.com>
 To: "users@kafka.apache.org" <us...@kafka.apache.org>; Virendra Pratap Singh <vp...@yahoo-inc.com> 
 Sent: Tuesday, March 3, 2015 4:29 PM
 Subject: Re: reassign a topic partition which has no ISR and leader set to -1
   
I hate bringing bad news, but...

You can't really reassign replicas if the leader is not available.
Since the leader is gone, the replicas have no where to replicate the data from.

Until you bring the leader back (or one of the replicas with unclean
leader election), you basically lost this partition.

Gwen



On Tue, Mar 3, 2015 at 3:57 PM, Virendra Pratap Singh
<vp...@yahoo-inc.com.invalid> wrote:
> Ran into a situation where both the leader and replica nodes for a few partitions of a given topic went down. So now these partitions have no in-sync replicas and neither any leader (leader set to -1).
> I tried to reassign these partitions to a different set of brokers using partition reassignment tool and then running the preferred replica. However I don't see the partitions getting reassigned and the leader still value still remains "-1" for these.
> What do I need to get this working.
> Kafka cluster is on 0.8.1.1
> Regards,Virendra


  

Re: reassign a topic partition which has no ISR and leader set to -1

[Gwen Shapira <gs...@cloudera.com>]

I hate bringing bad news, but...

You can't really reassign replicas if the leader is not available.
Since the leader is gone, the replicas have no where to replicate the data from.

Until you bring the leader back (or one of the replicas with unclean
leader election), you basically lost this partition.

Gwen

On Tue, Mar 3, 2015 at 3:57 PM, Virendra Pratap Singh
<vp...@yahoo-inc.com.invalid> wrote:
> Ran into a situation where both the leader and replica nodes for a few partitions of a given topic went down. So now these partitions have no in-sync replicas and neither any leader (leader set to -1).
> I tried to reassign these partitions to a different set of brokers using partition reassignment tool and then running the preferred replica. However I don't see the partitions getting reassigned and the leader still value still remains "-1" for these.
> What do I need to get this working.
> Kafka cluster is on 0.8.1.1
> Regards,Virendra

reassign a topic partition which has no ISR and leader set to -1

[Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>]

Ran into a situation where both the leader and replica nodes for a few partitions of a given topic went down. So now these partitions have no in-sync replicas and neither any leader (leader set to -1).
I tried to reassign these partitions to a different set of brokers using partition reassignment tool and then running the preferred replica. However I don't see the partitions getting reassigned and the leader still value still remains "-1" for these.
What do I need to get this working.
Kafka cluster is on 0.8.1.1
Regards,Virendra