Producing message set

[Piotr Husiatyński <ph...@optiopay.com>]

Hi,

I'm writing new client library for kafka and I cannot find information about
atomicity of message set publishing. I'm implementing protocol described on
wiki [1]. According to documentation, it is possible to send within single
request more than one message - message set for any partition.
In response, there is error code for every partition separately. Does this
mean, that while writing messages to different partitions can fail, writing
multiple messages to one partition as message set is atomic and can either
fail or succeed for all of them?


[1] https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol

Re: Producing message set

[Gwen Shapira <gs...@cloudera.com>]

Looking at the code, it does look like appending a message set to a
partition will succeed or fail as a whole.

You can look at Log.append() for the exact logic if you are interested.

Gwen

On Wed, Feb 11, 2015 at 6:13 AM, Piotr Husiatyński <ph...@optiopay.com> wrote:

> Hi,
>
> I'm writing new client library for kafka and I cannot find information
> about
> atomicity of message set publishing. I'm implementing protocol described on
> wiki [1]. According to documentation, it is possible to send within single
> request more than one message - message set for any partition.
> In response, there is error code for every partition separately. Does this
> mean, that while writing messages to different partitions can fail, writing
> multiple messages to one partition as message set is atomic and can either
> fail or succeed for all of them?
>
>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
>

Re: Producing message set

[Jun Rao <ju...@confluent.io>]

A messageSet does get written to the local log file atomically. However,
the set may not be fetched atomically by the replica fetcher. So, on
switching the leader of a partition, it's possible that only a subset of
the messages in the set show up in the new leader. The exception is that if
the messageSet is compressed, it will show up all or nothing on failover.

Thanks,

Jun

On Wed, Feb 11, 2015 at 6:13 AM, Piotr Husiatyński <ph...@optiopay.com> wrote:

> Hi,
>
> I'm writing new client library for kafka and I cannot find information
> about
> atomicity of message set publishing. I'm implementing protocol described on
> wiki [1]. According to documentation, it is possible to send within single
> request more than one message - message set for any partition.
> In response, there is error code for every partition separately. Does this
> mean, that while writing messages to different partitions can fail, writing
> multiple messages to one partition as message set is atomic and can either
> fail or succeed for all of them?
>
>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
>