Control messages in Kafka

[Chris Riccomini <cr...@linkedin.com.INVALID>]

Hey Guys,

The current transactionality proposal (https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka) and implementation use control messages to handle transactions in Kafka. Kafka traditionally hasn't had control messages in its topics. Transactionality (as it's implemented) introduces this pattern, but appears to do so in a very specific fashion (control messages only for transactions).

It seems to me that a good approach to control messages would be to generalize the control message model in Kafka to support not just transaction control messages, but arbitrary control messages? On the producer side, arbitrary control messages should be allowed to be sent, and on the consumer side, these control messages should be dropped by default.

Just like transactionality, this would let frameworks (e.g. Samza) and other app-specific implementations take advantage of in-topic control messages (as opposed to out of band control messages) without any impact on existing consumers.

Thoughts?

Cheers,
Chris

Re: Control messages in Kafka

[Neha Narkhede <ne...@gmail.com>]

That makes sense. It will be worth starting a discussion and a wiki on a
generally applicable control message format. Would you like to file a JIRA
for that?

Thanks,
Neha

On Wed, Sep 10, 2014 at 1:40 PM, Chris Riccomini <
criccomini@linkedin.com.invalid> wrote:

> Hey Guys,
>
> The current transactionality proposal (
> https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka)
> and implementation use control messages to handle transactions in Kafka.
> Kafka traditionally hasn't had control messages in its topics.
> Transactionality (as it's implemented) introduces this pattern, but appears
> to do so in a very specific fashion (control messages only for
> transactions).
>
> It seems to me that a good approach to control messages would be to
> generalize the control message model in Kafka to support not just
> transaction control messages, but arbitrary control messages? On the
> producer side, arbitrary control messages should be allowed to be sent, and
> on the consumer side, these control messages should be dropped by default.
>
> Just like transactionality, this would let frameworks (e.g. Samza) and
> other app-specific implementations take advantage of in-topic control
> messages (as opposed to out of band control messages) without any impact on
> existing consumers.
>
> Thoughts?
>
> Cheers,
> Chris
>