Two Kafka Question

["Sybrandy, Casey" <Ca...@Six3Systems.com>]

Hello,

First, is there a limit to how many Kafka brokers you can have?

Second, if a Kafka broker node fails and I start a new broker on a new node, is it correct to assume that the cluster will copy data to that node to satisfy the replication factor specified for a given topic?  In other words, let's assume that I have a 3 node cluster and a topic with a replication factor of 3.  If one node fails and I start up a new node, will the new node have existing messages replicated to it?

Thanks.

Casey

RE: Two Kafka Question

["Sybrandy, Casey" <Ca...@Six3Systems.com>]

Gwen,

Thanks.

1. I had a feeling about zookeeper being the potential bottleneck, but I wasn't sure.
2. Good to know.

________________________________________
From: Gwen Shapira [gshapira@cloudera.com]
Sent: Monday, November 24, 2014 2:47 PM
To: users@kafka.apache.org
Subject: Re: Two Kafka Question

Hi Casey,

1. There's some limit based on size of zookeeper nodes, not sure exactly
where it is though. We've seen 30 node clusters running in production.

2. For your scenario to work, the new broker will need to have the same
broker id as the old one - or you'll need to manually re-assign partitions.

Gwen

On Mon, Nov 24, 2014 at 11:15 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:

> Hello,
>
> First, is there a limit to how many Kafka brokers you can have?
>
> Second, if a Kafka broker node fails and I start a new broker on a new
> node, is it correct to assume that the cluster will copy data to that node
> to satisfy the replication factor specified for a given topic?  In other
> words, let's assume that I have a 3 node cluster and a topic with a
> replication factor of 3.  If one node fails and I start up a new node, will
> the new node have existing messages replicated to it?
>
> Thanks.
>
> Casey
>

Re: Two Kafka Question

[Gwen Shapira <gs...@cloudera.com>]

Hi Casey,

1. There's some limit based on size of zookeeper nodes, not sure exactly
where it is though. We've seen 30 node clusters running in production.

2. For your scenario to work, the new broker will need to have the same
broker id as the old one - or you'll need to manually re-assign partitions.

Gwen

On Mon, Nov 24, 2014 at 11:15 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:

> Hello,
>
> First, is there a limit to how many Kafka brokers you can have?
>
> Second, if a Kafka broker node fails and I start a new broker on a new
> node, is it correct to assume that the cluster will copy data to that node
> to satisfy the replication factor specified for a given topic?  In other
> words, let's assume that I have a 3 node cluster and a topic with a
> replication factor of 3.  If one node fails and I start up a new node, will
> the new node have existing messages replicated to it?
>
> Thanks.
>
> Casey
>