You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yury Ruchin <yu...@gmail.com> on 2014/12/05 19:32:47 UTC
Is Kafka documentation regarding null key misleading?
Hello,
I've come across a (seemingly) strange situation when my Kafka producer
gave so uneven distribution across partitions. I found that I used null key
to produce messages, guided by the following clause in the documentation:
"If the key is null, then a random broker partition is picked." However,
after looking at the code, I found that the broker partition is not truly
random for every message - instead, the randomly picked partition number
sticks and only refreshes after the topic.metadata.refresh.ms expires,
which is 10 minutes by default. So, with null key the producer keeps
writing to the same partition for 10 minutes.
Is my understanding of partitioning with null key correct? If yes,
shouldn't the documentation be fixed then to explicitly describe the sticky
pseudo-random partition assignment?
Thanks,
Yury
Re: Is Kafka documentation regarding null key misleading?
Posted by Guozhang Wang <wa...@gmail.com>.
Steven,
You can take a look at kafka.producer.async.DefaultEventHandler, in
getPartition function.
Guozhang
On Thu, Dec 11, 2014 at 9:58 AM, Steven Wu <st...@gmail.com> wrote:
> Guozhang,
>
> can you point me to the code that implements "periodic/sticky" random
> partitioner? I actually like to try it out in our env, even though I assume
> it is NOT ported to 0.8.2 java producer.
>
> Thanks,
> Steven
>
>
> On Mon, Dec 8, 2014 at 1:43 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Yury,
> >
> > Originally the producer behavior under null-key is "random" random, but
> > later changed to this "periodic" random to reduce the number of sockets
> on
> > the server side: imagine if you have n brokers and m producers where m
> >>>
> > n, with random random distribution each server will need to maintain a
> > socket with each of the m producers.
> >
> > We realized that this change IS misleading and we have changed back to
> > random random in the new producer released in 0.8.2.
> >
> >
> > Guozhang
> >
> > On Fri, Dec 5, 2014 at 10:43 AM, Andrew Jorgensen <
> > ajorgensen@twitter.com.invalid> wrote:
> >
> > > If you look under Producer configs you see the following key ‘
> > > topic.metadata.refresh.interval.ms’ with a default of 600 * 1000 (10
> > > minutes). It is not entirely clear but this controls how often a
> producer
> > > will a null key partitioner will switch partitions that it is writing
> to.
> > > In my production app I set this down to 1 minute and haven’t seen any
> ill
> > > effects but it is good to note that the shorter you get *could* cause
> > some
> > > issues and extra overhead. I agree this could probably be a little more
> > > clear in the documentation.
> > > -
> > > Andrew Jorgensen
> > > @ajorgensen
> > >
> > > On December 5, 2014 at 1:34:00 PM, Yury Ruchin (yuri.ruchin@gmail.com)
> > > wrote:
> > >
> > > Hello,
> > >
> > > I've come across a (seemingly) strange situation when my Kafka producer
> > > gave so uneven distribution across partitions. I found that I used null
> > key
> > > to produce messages, guided by the following clause in the
> documentation:
> > > "If the key is null, then a random broker partition is picked."
> However,
> > > after looking at the code, I found that the broker partition is not
> truly
> > > random for every message - instead, the randomly picked partition
> number
> > > sticks and only refreshes after the topic.metadata.refresh.ms expires,
> > > which is 10 minutes by default. So, with null key the producer keeps
> > > writing to the same partition for 10 minutes.
> > >
> > > Is my understanding of partitioning with null key correct? If yes,
> > > shouldn't the documentation be fixed then to explicitly describe the
> > sticky
> > > pseudo-random partition assignment?
> > >
> > > Thanks,
> > > Yury
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Is Kafka documentation regarding null key misleading?
Posted by Steven Wu <st...@gmail.com>.
Guozhang,
can you point me to the code that implements "periodic/sticky" random
partitioner? I actually like to try it out in our env, even though I assume
it is NOT ported to 0.8.2 java producer.
Thanks,
Steven
On Mon, Dec 8, 2014 at 1:43 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Yury,
>
> Originally the producer behavior under null-key is "random" random, but
> later changed to this "periodic" random to reduce the number of sockets on
> the server side: imagine if you have n brokers and m producers where m >>>
> n, with random random distribution each server will need to maintain a
> socket with each of the m producers.
>
> We realized that this change IS misleading and we have changed back to
> random random in the new producer released in 0.8.2.
>
>
> Guozhang
>
> On Fri, Dec 5, 2014 at 10:43 AM, Andrew Jorgensen <
> ajorgensen@twitter.com.invalid> wrote:
>
> > If you look under Producer configs you see the following key ‘
> > topic.metadata.refresh.interval.ms’ with a default of 600 * 1000 (10
> > minutes). It is not entirely clear but this controls how often a producer
> > will a null key partitioner will switch partitions that it is writing to.
> > In my production app I set this down to 1 minute and haven’t seen any ill
> > effects but it is good to note that the shorter you get *could* cause
> some
> > issues and extra overhead. I agree this could probably be a little more
> > clear in the documentation.
> > -
> > Andrew Jorgensen
> > @ajorgensen
> >
> > On December 5, 2014 at 1:34:00 PM, Yury Ruchin (yuri.ruchin@gmail.com)
> > wrote:
> >
> > Hello,
> >
> > I've come across a (seemingly) strange situation when my Kafka producer
> > gave so uneven distribution across partitions. I found that I used null
> key
> > to produce messages, guided by the following clause in the documentation:
> > "If the key is null, then a random broker partition is picked." However,
> > after looking at the code, I found that the broker partition is not truly
> > random for every message - instead, the randomly picked partition number
> > sticks and only refreshes after the topic.metadata.refresh.ms expires,
> > which is 10 minutes by default. So, with null key the producer keeps
> > writing to the same partition for 10 minutes.
> >
> > Is my understanding of partitioning with null key correct? If yes,
> > shouldn't the documentation be fixed then to explicitly describe the
> sticky
> > pseudo-random partition assignment?
> >
> > Thanks,
> > Yury
> >
>
>
>
> --
> -- Guozhang
>
Re: Is Kafka documentation regarding null key misleading?
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Yury,
Originally the producer behavior under null-key is "random" random, but
later changed to this "periodic" random to reduce the number of sockets on
the server side: imagine if you have n brokers and m producers where m >>>
n, with random random distribution each server will need to maintain a
socket with each of the m producers.
We realized that this change IS misleading and we have changed back to
random random in the new producer released in 0.8.2.
Guozhang
On Fri, Dec 5, 2014 at 10:43 AM, Andrew Jorgensen <
ajorgensen@twitter.com.invalid> wrote:
> If you look under Producer configs you see the following key ‘
> topic.metadata.refresh.interval.ms’ with a default of 600 * 1000 (10
> minutes). It is not entirely clear but this controls how often a producer
> will a null key partitioner will switch partitions that it is writing to.
> In my production app I set this down to 1 minute and haven’t seen any ill
> effects but it is good to note that the shorter you get *could* cause some
> issues and extra overhead. I agree this could probably be a little more
> clear in the documentation.
> -
> Andrew Jorgensen
> @ajorgensen
>
> On December 5, 2014 at 1:34:00 PM, Yury Ruchin (yuri.ruchin@gmail.com)
> wrote:
>
> Hello,
>
> I've come across a (seemingly) strange situation when my Kafka producer
> gave so uneven distribution across partitions. I found that I used null key
> to produce messages, guided by the following clause in the documentation:
> "If the key is null, then a random broker partition is picked." However,
> after looking at the code, I found that the broker partition is not truly
> random for every message - instead, the randomly picked partition number
> sticks and only refreshes after the topic.metadata.refresh.ms expires,
> which is 10 minutes by default. So, with null key the producer keeps
> writing to the same partition for 10 minutes.
>
> Is my understanding of partitioning with null key correct? If yes,
> shouldn't the documentation be fixed then to explicitly describe the sticky
> pseudo-random partition assignment?
>
> Thanks,
> Yury
>
--
-- Guozhang
Re: Is Kafka documentation regarding null key misleading?
Posted by Andrew Jorgensen <aj...@twitter.com.INVALID>.
If you look under Producer configs you see the following key ‘topic.metadata.refresh.interval.ms’ with a default of 600 * 1000 (10 minutes). It is not entirely clear but this controls how often a producer will a null key partitioner will switch partitions that it is writing to. In my production app I set this down to 1 minute and haven’t seen any ill effects but it is good to note that the shorter you get *could* cause some issues and extra overhead. I agree this could probably be a little more clear in the documentation.
-
Andrew Jorgensen
@ajorgensen
On December 5, 2014 at 1:34:00 PM, Yury Ruchin (yuri.ruchin@gmail.com) wrote:
Hello,
I've come across a (seemingly) strange situation when my Kafka producer
gave so uneven distribution across partitions. I found that I used null key
to produce messages, guided by the following clause in the documentation:
"If the key is null, then a random broker partition is picked." However,
after looking at the code, I found that the broker partition is not truly
random for every message - instead, the randomly picked partition number
sticks and only refreshes after the topic.metadata.refresh.ms expires,
which is 10 minutes by default. So, with null key the producer keeps
writing to the same partition for 10 minutes.
Is my understanding of partitioning with null key correct? If yes,
shouldn't the documentation be fixed then to explicitly describe the sticky
pseudo-random partition assignment?
Thanks,
Yury
Re: Is Kafka documentation regarding null key misleading?
Posted by Michal Michalski <mi...@boxever.com>.
Yes, it is *very* misleading in my opinion - I've seen so many people
surprised with that behaviour...
Technically it's 100% correct of course: "If the key is null, then the
Producer will assign the message to a random Partition." - that's what
actually happens, because assignment is random. However, everyone I know
(including myself) assumed that "random" means "random for each message",
rather than "picked randomly every topic.metadata.refresh.ms".
So no, you're not the only one who got that wrong ;-)
M.
Kind regards,
Michał Michalski,
michal.michalski@boxever.com
On 5 December 2014 at 18:32, Yury Ruchin <yu...@gmail.com> wrote:
> Hello,
>
> I've come across a (seemingly) strange situation when my Kafka producer
> gave so uneven distribution across partitions. I found that I used null key
> to produce messages, guided by the following clause in the documentation:
> "If the key is null, then a random broker partition is picked." However,
> after looking at the code, I found that the broker partition is not truly
> random for every message - instead, the randomly picked partition number
> sticks and only refreshes after the topic.metadata.refresh.ms expires,
> which is 10 minutes by default. So, with null key the producer keeps
> writing to the same partition for 10 minutes.
>
> Is my understanding of partitioning with null key correct? If yes,
> shouldn't the documentation be fixed then to explicitly describe the sticky
> pseudo-random partition assignment?
>
> Thanks,
> Yury
>