Question regarding topic creation

[Sreenivasulu Nallapati <sr...@gmail.com>]

Hello,

I am new to Kafka and trying to evaluate for one of our use case. I have a
basic question regarding topic.

Assume we have a 10 broker cluster. If a create a Topic, will it be created
in all 10 brokers?
If not, how it manages internally if we try to send a message to that
perticular topic? assume we created a producer with all brokers

Thanks in advance.


Thanks
Sreeni

Re: Question regarding topic creation

[Jun Rao <ju...@gmail.com>]

In 0.8.x, a topic doesn't necessarily get created on every broker. Each
topic can have one or more partitions. Each partition can have one or more
replicas. In general, the replicas are spread over the brokers in a
cluster. The producer obtains the metadata of a topic to determine which
broker to send a message to.

Thanks,

Jun

On Fri, Oct 3, 2014 at 8:09 AM, Sreenivasulu Nallapati <
sreenu.nallapati@gmail.com> wrote:

> Hello,
>
> I am new to Kafka and trying to evaluate for one of our use case. I have a
> basic question regarding topic.
>
> Assume we have a 10 broker cluster. If a create a Topic, will it be created
> in all 10 brokers?
> If not, how it manages internally if we try to send a message to that
> perticular topic? assume we created a producer with all brokers
>
> Thanks in advance.
>
>
> Thanks
> Sreeni
>

Re: Question regarding topic creation

[Gwen Shapira <gs...@cloudera.com>]

By "created in all 10 brokers" you mean 10 partitions? Or 10 replicas for
each partition?

You can create a topic with either option,  or both. By specifing partition
number and number of replicas when creating the topic or by setting a
default for all future topics.

In my opinion,  10 replicas is a bit excessive,  but definitely possible.

When you send a message to a broker, you need to write it to the leader
replica of the partition you are writing to. The Java producer handles that
for you, even if you don't specify all brokers in the list.

Hope this clarifies things.

Gwen
On Oct 3, 2014 8:09 AM, "Sreenivasulu Nallapati" <sr...@gmail.com>
wrote:

> Hello,
>
> I am new to Kafka and trying to evaluate for one of our use case. I have a
> basic question regarding topic.
>
> Assume we have a 10 broker cluster. If a create a Topic, will it be created
> in all 10 brokers?
> If not, how it manages internally if we try to send a message to that
> perticular topic? assume we created a producer with all brokers
>
> Thanks in advance.
>
>
> Thanks
> Sreeni
>