Kafka null keys - OK or a problem?

[Ali Akhtar <al...@gmail.com>]

A kafka producer written elsewhere that I'm using, which uses the Go kafka
driver, is sending messages where the key is null.

Is this OK - or will this cause issues due to partitioning not happening
correctly?

What would be a good way to generate keys in this case, to ensure even
partition spread?

Thanks.

Re: Kafka null keys - OK or a problem?

[Ali Akhtar <al...@gmail.com>]

It seems to be using a Hash Partitioner here:
https://github.com/Shopify/sarama/blob/master/config.go#L262

and HashPartitioner is documented as:

>  If the message's key is nil then a random partition is chosen

https://godoc.org/github.com/Shopify/sarama#example-Partitioner--Random

So.. it should be okay to have null keys, I'm guessing.

On Mon, Oct 10, 2016 at 11:51 AM, Ali Akhtar <al...@gmail.com> wrote:

> Hey Michael,
>
> We're using this one: https://github.com/Shopify/sarama
>
> Any ideas how that one works?
>
> On Mon, Oct 10, 2016 at 11:48 AM, Michael Noll <mi...@confluent.io>
> wrote:
>
>> FYI: Kafka's new Java producer (which ships with Kafka) the behavior is as
>> follows:  If no partition is explicitly specified (to send the message to)
>> AND the key is null, then the DefaultPartitioner [1] will assign messages
>> to topic partitions in a round-robin fashion.  See the javadoc and also
>> the
>> little bit of code in [1] for details.
>>
>> Not sure which Go client you're using exactly so I can't tell whether your
>> Go client follows the behavior of Kafka's Java producer.
>>
>> -Michael
>>
>>
>>
>>
>> [1]
>> https://github.com/apache/kafka/blob/trunk/clients/src/main/
>> java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java
>>
>>
>> On Mon, Oct 10, 2016 at 7:53 AM, Ali Akhtar <al...@gmail.com> wrote:
>>
>> > If keys are null, what happens in terms of partitioning, is the load
>> spread
>> > evenly..?
>> >
>> > On Mon, Oct 10, 2016 at 7:59 AM, Gwen Shapira <gw...@confluent.io>
>> wrote:
>> >
>> > > Kafka itself supports null keys. I'm not sure about the Go client you
>> > > use, but Confluent's Go client also supports null keys
>> > > (https://github.com/confluentinc/confluent-kafka-go/).
>> > >
>> > > If you decide to generate keys and you want even spread, a random
>> > > number generator is probably your best bet.
>> > >
>> > > Gwen
>> > >
>> > > On Sun, Oct 9, 2016 at 6:05 PM, Ali Akhtar <al...@gmail.com>
>> wrote:
>> > > > A kafka producer written elsewhere that I'm using, which uses the Go
>> > > kafka
>> > > > driver, is sending messages where the key is null.
>> > > >
>> > > > Is this OK - or will this cause issues due to partitioning not
>> > happening
>> > > > correctly?
>> > > >
>> > > > What would be a good way to generate keys in this case, to ensure
>> even
>> > > > partition spread?
>> > > >
>> > > > Thanks.
>> > >
>> > >
>> > >
>> > > --
>> > > Gwen Shapira
>> > > Product Manager | Confluent
>> > > 650.450.2760 | @gwenshap
>> > > Follow us: Twitter | blog
>> > >
>> >
>>
>
>

Re: Kafka null keys - OK or a problem?

[Michael Noll <mi...@confluent.io>]

Depends on which partitioner you are using, see [1] and [2].  From what I
understand the `NewHashPartitioner` comes closest to the behavior of Kafka
Java producer, but instead of going round-robin for null-keyed messages it
picks a partition at random.



[1] https://godoc.org/github.com/Shopify/sarama#Partitioner
[2] https://github.com/Shopify/sarama/blob/master/partitioner.go



On Mon, Oct 10, 2016 at 8:51 AM, Ali Akhtar <al...@gmail.com> wrote:

> Hey Michael,
>
> We're using this one: https://github.com/Shopify/sarama
>
> Any ideas how that one works?
>
> On Mon, Oct 10, 2016 at 11:48 AM, Michael Noll <mi...@confluent.io>
> wrote:
>
> > FYI: Kafka's new Java producer (which ships with Kafka) the behavior is
> as
> > follows:  If no partition is explicitly specified (to send the message
> to)
> > AND the key is null, then the DefaultPartitioner [1] will assign messages
> > to topic partitions in a round-robin fashion.  See the javadoc and also
> the
> > little bit of code in [1] for details.
> >
> > Not sure which Go client you're using exactly so I can't tell whether
> your
> > Go client follows the behavior of Kafka's Java producer.
> >
> > -Michael
> >
> >
> >
> >
> > [1]
> > https://github.com/apache/kafka/blob/trunk/clients/src/
> > main/java/org/apache/kafka/clients/producer/internals/
> > DefaultPartitioner.java
> >
> >
> > On Mon, Oct 10, 2016 at 7:53 AM, Ali Akhtar <al...@gmail.com>
> wrote:
> >
> > > If keys are null, what happens in terms of partitioning, is the load
> > spread
> > > evenly..?
> > >
> > > On Mon, Oct 10, 2016 at 7:59 AM, Gwen Shapira <gw...@confluent.io>
> wrote:
> > >
> > > > Kafka itself supports null keys. I'm not sure about the Go client you
> > > > use, but Confluent's Go client also supports null keys
> > > > (https://github.com/confluentinc/confluent-kafka-go/).
> > > >
> > > > If you decide to generate keys and you want even spread, a random
> > > > number generator is probably your best bet.
> > > >
> > > > Gwen
> > > >
> > > > On Sun, Oct 9, 2016 at 6:05 PM, Ali Akhtar <al...@gmail.com>
> > wrote:
> > > > > A kafka producer written elsewhere that I'm using, which uses the
> Go
> > > > kafka
> > > > > driver, is sending messages where the key is null.
> > > > >
> > > > > Is this OK - or will this cause issues due to partitioning not
> > > happening
> > > > > correctly?
> > > > >
> > > > > What would be a good way to generate keys in this case, to ensure
> > even
> > > > > partition spread?
> > > > >
> > > > > Thanks.
> > > >
> > > >
> > > >
> > > > --
> > > > Gwen Shapira
> > > > Product Manager | Confluent
> > > > 650.450.2760 | @gwenshap
> > > > Follow us: Twitter | blog
> > > >
> > >
> >
>

Re: Kafka null keys - OK or a problem?

[Ali Akhtar <al...@gmail.com>]

Hey Michael,

We're using this one: https://github.com/Shopify/sarama

Any ideas how that one works?

On Mon, Oct 10, 2016 at 11:48 AM, Michael Noll <mi...@confluent.io> wrote:

> FYI: Kafka's new Java producer (which ships with Kafka) the behavior is as
> follows:  If no partition is explicitly specified (to send the message to)
> AND the key is null, then the DefaultPartitioner [1] will assign messages
> to topic partitions in a round-robin fashion.  See the javadoc and also the
> little bit of code in [1] for details.
>
> Not sure which Go client you're using exactly so I can't tell whether your
> Go client follows the behavior of Kafka's Java producer.
>
> -Michael
>
>
>
>
> [1]
> https://github.com/apache/kafka/blob/trunk/clients/src/
> main/java/org/apache/kafka/clients/producer/internals/
> DefaultPartitioner.java
>
>
> On Mon, Oct 10, 2016 at 7:53 AM, Ali Akhtar <al...@gmail.com> wrote:
>
> > If keys are null, what happens in terms of partitioning, is the load
> spread
> > evenly..?
> >
> > On Mon, Oct 10, 2016 at 7:59 AM, Gwen Shapira <gw...@confluent.io> wrote:
> >
> > > Kafka itself supports null keys. I'm not sure about the Go client you
> > > use, but Confluent's Go client also supports null keys
> > > (https://github.com/confluentinc/confluent-kafka-go/).
> > >
> > > If you decide to generate keys and you want even spread, a random
> > > number generator is probably your best bet.
> > >
> > > Gwen
> > >
> > > On Sun, Oct 9, 2016 at 6:05 PM, Ali Akhtar <al...@gmail.com>
> wrote:
> > > > A kafka producer written elsewhere that I'm using, which uses the Go
> > > kafka
> > > > driver, is sending messages where the key is null.
> > > >
> > > > Is this OK - or will this cause issues due to partitioning not
> > happening
> > > > correctly?
> > > >
> > > > What would be a good way to generate keys in this case, to ensure
> even
> > > > partition spread?
> > > >
> > > > Thanks.
> > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter | blog
> > >
> >
>

Re: Kafka null keys - OK or a problem?

[Michael Noll <mi...@confluent.io>]

FYI: Kafka's new Java producer (which ships with Kafka) the behavior is as
follows:  If no partition is explicitly specified (to send the message to)
AND the key is null, then the DefaultPartitioner [1] will assign messages
to topic partitions in a round-robin fashion.  See the javadoc and also the
little bit of code in [1] for details.

Not sure which Go client you're using exactly so I can't tell whether your
Go client follows the behavior of Kafka's Java producer.

-Michael




[1]
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java


On Mon, Oct 10, 2016 at 7:53 AM, Ali Akhtar <al...@gmail.com> wrote:

> If keys are null, what happens in terms of partitioning, is the load spread
> evenly..?
>
> On Mon, Oct 10, 2016 at 7:59 AM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > Kafka itself supports null keys. I'm not sure about the Go client you
> > use, but Confluent's Go client also supports null keys
> > (https://github.com/confluentinc/confluent-kafka-go/).
> >
> > If you decide to generate keys and you want even spread, a random
> > number generator is probably your best bet.
> >
> > Gwen
> >
> > On Sun, Oct 9, 2016 at 6:05 PM, Ali Akhtar <al...@gmail.com> wrote:
> > > A kafka producer written elsewhere that I'm using, which uses the Go
> > kafka
> > > driver, is sending messages where the key is null.
> > >
> > > Is this OK - or will this cause issues due to partitioning not
> happening
> > > correctly?
> > >
> > > What would be a good way to generate keys in this case, to ensure even
> > > partition spread?
> > >
> > > Thanks.
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>

Re: Kafka null keys - OK or a problem?

[Ali Akhtar <al...@gmail.com>]

If keys are null, what happens in terms of partitioning, is the load spread
evenly..?

On Mon, Oct 10, 2016 at 7:59 AM, Gwen Shapira <gw...@confluent.io> wrote:

> Kafka itself supports null keys. I'm not sure about the Go client you
> use, but Confluent's Go client also supports null keys
> (https://github.com/confluentinc/confluent-kafka-go/).
>
> If you decide to generate keys and you want even spread, a random
> number generator is probably your best bet.
>
> Gwen
>
> On Sun, Oct 9, 2016 at 6:05 PM, Ali Akhtar <al...@gmail.com> wrote:
> > A kafka producer written elsewhere that I'm using, which uses the Go
> kafka
> > driver, is sending messages where the key is null.
> >
> > Is this OK - or will this cause issues due to partitioning not happening
> > correctly?
> >
> > What would be a good way to generate keys in this case, to ensure even
> > partition spread?
> >
> > Thanks.
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: Kafka null keys - OK or a problem?

[Gwen Shapira <gw...@confluent.io>]

Kafka itself supports null keys. I'm not sure about the Go client you
use, but Confluent's Go client also supports null keys
(https://github.com/confluentinc/confluent-kafka-go/).

If you decide to generate keys and you want even spread, a random
number generator is probably your best bet.

Gwen

On Sun, Oct 9, 2016 at 6:05 PM, Ali Akhtar <al...@gmail.com> wrote:
> A kafka producer written elsewhere that I'm using, which uses the Go kafka
> driver, is sending messages where the key is null.
>
> Is this OK - or will this cause issues due to partitioning not happening
> correctly?
>
> What would be a good way to generate keys in this case, to ensure even
> partition spread?
>
> Thanks.



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog