Storing data in kafka keys

[Ivan Balashov <ib...@gmail.com>]

Hi,

It looks like it is a general practice to avoid storing data in kafka
keys. Some examples of this: Camus, Secor both not using keys. Even
such a swiss-army tool as kafkacat doesn't seem to have the ability to
display key (although I might be wrong). Also, console producer does
not display keys by default, which makes it confusing in initial quick
checks for data.

What's the story behind this? Should one think twice before tying
business data to kafka keys?

Thanks,

Re: Storing data in kafka keys

[Jun Rao <ju...@gmail.com>]

The keyed messages are typically used in two cases (1) you want messages
with the same key to be in the same partition and therefore those messages
with the same key will be consumed by the same consumer instance; (2) you
want to enable the log compaction feature for retention such that the
broker will over time garbage collect all messages with the same key,
except for the most recent message.

Thanks,

Jun

On Wed, Nov 5, 2014 at 10:15 AM, Ivan Balashov <ib...@gmail.com> wrote:

> Hi,
>
> It looks like it is a general practice to avoid storing data in kafka
> keys. Some examples of this: Camus, Secor both not using keys. Even
> such a swiss-army tool as kafkacat doesn't seem to have the ability to
> display key (although I might be wrong). Also, console producer does
> not display keys by default, which makes it confusing in initial quick
> checks for data.
>
> What's the story behind this? Should one think twice before tying
> business data to kafka keys?
>
> Thanks,
>