Partitioning & High availability

[howard chen <ho...@gmail.com>]

Hi,

Some basic questions concerning Partitioning & High availability

Background: Assume I have 3 nodes of Kafka

1. Should I set the num.partitions to 3 or 1?
2. By using the default DefaultPartitioner, it is

hash(key)%num_partitions

What if one of my node down (assume I am using 3 partitions)?

a. All message sent to this node will be dropped?
b. Do I need to wait all the node come up again and able to consume
the message again?

Thanks.

Re: Partitioning & High availability

[Jun Rao <ju...@gmail.com>]

Kafka guarantees FIFO per partition. If a node goes down, clients can't
write/read to partitions on that node. So, data on those partitions are
delayed. Once the broker is up, data will be delivered in order.

In 0.8, availability will be improved with intra-cluster replication.

Thanks,

Jun

On Mon, Oct 22, 2012 at 8:45 AM, howard chen <ho...@gmail.com> wrote:

> Hi,
>
> On Mon, Oct 22, 2012 at 5:58 AM, Jun Rao <ju...@gmail.com> wrote:
> > Howard,
> >
> > In 0.7, partitions are local to each broker. So, if you set # partitions
> to
> > 3 per broker, you will get a total of 9 partitions with 3 brokers. Also,
> > since there is no redundancy across brokers, if a broker goes down, none
> of
> > all partitions on the broker is available. So num_partitions will shrink.
> >
>
> When people use queue, it must support FIFO semantic, so considering
> the "partitions", if any node goes down, the whole queue is basically
> useless then?
>
> THanks.
>

Re: Partitioning & High availability

[howard chen <ho...@gmail.com>]

Hi,

On Mon, Oct 22, 2012 at 5:58 AM, Jun Rao <ju...@gmail.com> wrote:
> Howard,
>
> In 0.7, partitions are local to each broker. So, if you set # partitions to
> 3 per broker, you will get a total of 9 partitions with 3 brokers. Also,
> since there is no redundancy across brokers, if a broker goes down, none of
> all partitions on the broker is available. So num_partitions will shrink.
>

When people use queue, it must support FIFO semantic, so considering
the "partitions", if any node goes down, the whole queue is basically
useless then?

THanks.

Re: Partitioning & High availability

[Jun Rao <ju...@gmail.com>]

Howard,

In 0.7, partitions are local to each broker. So, if you set # partitions to
3 per broker, you will get a total of 9 partitions with 3 brokers. Also,
since there is no redundancy across brokers, if a broker goes down, none of
all partitions on the broker is available. So num_partitions will shrink.

In 0.8, things are different. Partitions are at the cluster level. Number
of partitions is determined at topic creation time and won't change as the
# of brokers changes in the cluster. Also, 0.8 supports intra cluster
replication. So a partition is still available if a single broker is down.

Thanks,

Jun

On Fri, Oct 19, 2012 at 9:43 PM, howard chen <ho...@gmail.com> wrote:

> Hi,
>
> Some basic questions concerning Partitioning & High availability
>
> Background: Assume I have 3 nodes of Kafka
>
> 1. Should I set the num.partitions to 3 or 1?
> 2. By using the default DefaultPartitioner, it is
>
> hash(key)%num_partitions
>
> What if one of my node down (assume I am using 3 partitions)?
>
> a. All message sent to this node will be dropped?
> b. Do I need to wait all the node come up again and able to consume
> the message again?
>
> Thanks.
>