Partitions - Brokers - Servers

[Julio Camara <ju...@oracle.com>]

I think I understand the concept of partitioning a topic and it's 
benefits. But I can't find any clear explanation of relationship between 
partitions, brokers and physical servers. Is there any document or 
article that discusses the number of partitions for each broker and 
brokers for each server, for example?

In some articles there were examples of a topic with 4 partitions in 4 
brokers. Is this a typical implementation of partitions and brokers, 
i.e. n partitions require n brokers?

I would like to be able to use best practices to size the number of 
servers, brokers and partitions.

Thank you,
-- 
Julio Camara

Re: Partitions - Brokers - Servers

[Julio Camara <ju...@oracle.com>]

Gwen,

that helps. Thank you,

Julio

On 10/13/15 13:57, Gwen Shapira wrote:
> Hi,
>
> We normally run 1 broker per 1 physical server, and up to around 1000
> partitions per broker (although that depends on the specific machine the
> broker is on and specific configuration).
>
> In order to enjoy replication, we recommend a minimum of 3 brokers in the
> cluster, to support 3 replicas per partition.
>
> Hope this helps?
>
> Gwen
>
> On Mon, Oct 12, 2015 at 12:30 PM, Julio Camara <ju...@oracle.com>
> wrote:
>
>> I think I understand the concept of partitioning a topic and it's
>> benefits. But I can't find any clear explanation of relationship between
>> partitions, brokers and physical servers. Is there any document or article
>> that discusses the number of partitions for each broker and brokers for
>> each server, for example?
>>
>> In some articles there were examples of a topic with 4 partitions in 4
>> brokers. Is this a typical implementation of partitions and brokers, i.e. n
>> partitions require n brokers?
>>
>> I would like to be able to use best practices to size the number of
>> servers, brokers and partitions.
>>
>> Thank you,
>> --
>> Julio Camara
>>
>>

Re: Partitions - Brokers - Servers

[Gwen Shapira <gw...@confluent.io>]

Hi,

We normally run 1 broker per 1 physical server, and up to around 1000
partitions per broker (although that depends on the specific machine the
broker is on and specific configuration).

In order to enjoy replication, we recommend a minimum of 3 brokers in the
cluster, to support 3 replicas per partition.

Hope this helps?

Gwen

On Mon, Oct 12, 2015 at 12:30 PM, Julio Camara <ju...@oracle.com>
wrote:

> I think I understand the concept of partitioning a topic and it's
> benefits. But I can't find any clear explanation of relationship between
> partitions, brokers and physical servers. Is there any document or article
> that discusses the number of partitions for each broker and brokers for
> each server, for example?
>
> In some articles there were examples of a topic with 4 partitions in 4
> brokers. Is this a typical implementation of partitions and brokers, i.e. n
> partitions require n brokers?
>
> I would like to be able to use best practices to size the number of
> servers, brokers and partitions.
>
> Thank you,
> --
> Julio Camara
>
>