How do you recover an 0.8 Kafka server

[Eric Sites <Er...@threattrack.com>]

All,

How do you recover from a bad Kafka 0.8 server.

It failed to keep up with the replicated partitions assigned to it about a month ago.

I delete everything in the partitions directories, then created the partition folders assigned to it.
I also recreated the replication-offset-checkpoint file, with topcname part 0.
Then start Kafka back up.

And nothing... No recovery, just a bunch of empty log and index files.
It will not recover.

I have 2 servers like this, thank god I have 6 servers and almost all partitions have 3 replicas.

But I do have a topic that had 1 partition on the servers, now it is showing no leader.

Any help would be great.

Thanks,
Eric Sites

Re: How do you recover an 0.8 Kafka server

[Jun Rao <ju...@gmail.com>]

Actually, you should just need to restart the failed broker and data should
be replicated to that broker. One thing to check is the state-change log.
Do you see any LeaderAndIsr request when the bad broker is restarted?

Thanks,

Jun


On Sun, Aug 4, 2013 at 8:06 PM, Eric Sites <Er...@threattrack.com>wrote:

> All,
>
> How do you recover from a bad Kafka 0.8 server.
>
> It failed to keep up with the replicated partitions assigned to it about a
> month ago.
>
> I delete everything in the partitions directories, then created the
> partition folders assigned to it.
> I also recreated the replication-offset-checkpoint file, with topcname
> part 0.
> Then start Kafka back up.
>
> And nothing... No recovery, just a bunch of empty log and index files.
> It will not recover.
>
> I have 2 servers like this, thank god I have 6 servers and almost all
> partitions have 3 replicas.
>
> But I do have a topic that had 1 partition on the servers, now it is
> showing no leader.
>
> Any help would be great.
>
> Thanks,
> Eric Sites
>
>

Re: How do you recover an 0.8 Kafka server

[Eric Sites <Er...@threattrack.com>]

Fixed it.


An entire cluster restart was required.

Not the best solution, but all 6 servers, topics, partitions, and replicas
are in sync now.
And all topics have leaders.

Not I just need todo a little rebalancing.

Cheers,
Eric Sites

On 8/4/13 11:06 PM, "Eric Sites" <Er...@threattrack.com> wrote:

>All,
>
>How do you recover from a bad Kafka 0.8 server.
>
>It failed to keep up with the replicated partitions assigned to it about
>a month ago.
>
>I delete everything in the partitions directories, then created the
>partition folders assigned to it.
>I also recreated the replication-offset-checkpoint file, with topcname
>part 0.
>Then start Kafka back up.
>
>And nothing... No recovery, just a bunch of empty log and index files.
>It will not recover.
>
>I have 2 servers like this, thank god I have 6 servers and almost all
>partitions have 3 replicas.
>
>But I do have a topic that had 1 partition on the servers, now it is
>showing no leader.
>
>Any help would be great.
>
>Thanks,
>Eric Sites
>