command line clients and request balancing.

[Edward Capriolo <ed...@gmail.com>]

After installing Kafka I did the following

createTopic --partitions 10 -replication 1 --name mytopic

I started producer using bin/kafka-producer-shell

I opened two consumers in the same group using bin/kafka-consumer-shell

In older versions this test used to result in the messages being sent by
the producer round robin-ing between the two consumers. However I noticed
all the messages were going to one consumer until I killed it then all the
messages went to the other.

I opened another producer and still all messages went to the same consumer.

Is this just something new about the shell scripts/ or just a bug in the
0.8.0 version?

Tx

Re: command line clients and request balancing.

[Jun Rao <ju...@gmail.com>]

The reason for this change is described in
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
?

In 0.8.2, we will be releasing the new producer which will have the even
distribution out of the box, but allow the client to implement a sticky
partitioner if needed.

Thanks,

Jun


On Fri, May 2, 2014 at 5:33 AM, Edward Capriolo <ed...@gmail.com>wrote:

> Why was this change made? Does this mean that all applicationa i have will
> need to be converted to use the partitioned send?
>
> This hurts the out of the box experience. Can the console producer simply
> split on first space and use that as the key?
>
> On Thursday, May 1, 2014, Guozhang Wang <wa...@gmail.com> wrote:
> > Hi Edward,
> >
> > What is the current version you are using? Starting in 0.8.1 the default
> > partitioner of the producer would no longer be round-robin at a
> per-message
> > level but would stick to one partition for a period (the metadata refresh
> > period, which is 10 min by default) and then round-robin to another
> > partition in the next period.
> >
> > Guozhang
> >
> > G
> >
> >
> > On Thu, May 1, 2014 at 3:18 PM, Edward Capriolo <edlinuxguru@gmail.com
> >wrote:
> >
> >> After installing Kafka I did the following
> >>
> >> createTopic --partitions 10 -replication 1 --name mytopic
> >>
> >> I started producer using bin/kafka-producer-shell
> >>
> >> I opened two consumers in the same group using bin/kafka-consumer-shell
> >>
> >> In older versions this test used to result in the messages being sent by
> >> the producer round robin-ing between the two consumers. However I
> noticed
> >> all the messages were going to one consumer until I killed it then all
> the
> >> messages went to the other.
> >>
> >> I opened another producer and still all messages went to the same
> consumer.
> >>
> >> Is this just something new about the shell scripts/ or just a bug in the
> >> 0.8.0 version?
> >>
> >> Tx
> >>
> >
> >
> >
> > --
> > -- Guozhang
> >
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>

Re: command line clients and request balancing.

[Edward Capriolo <ed...@gmail.com>]

Why was this change made? Does this mean that all applicationa i have will
need to be converted to use the partitioned send?

This hurts the out of the box experience. Can the console producer simply
split on first space and use that as the key?

On Thursday, May 1, 2014, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Edward,
>
> What is the current version you are using? Starting in 0.8.1 the default
> partitioner of the producer would no longer be round-robin at a
per-message
> level but would stick to one partition for a period (the metadata refresh
> period, which is 10 min by default) and then round-robin to another
> partition in the next period.
>
> Guozhang
>
> G
>
>
> On Thu, May 1, 2014 at 3:18 PM, Edward Capriolo <edlinuxguru@gmail.com
>wrote:
>
>> After installing Kafka I did the following
>>
>> createTopic --partitions 10 -replication 1 --name mytopic
>>
>> I started producer using bin/kafka-producer-shell
>>
>> I opened two consumers in the same group using bin/kafka-consumer-shell
>>
>> In older versions this test used to result in the messages being sent by
>> the producer round robin-ing between the two consumers. However I noticed
>> all the messages were going to one consumer until I killed it then all
the
>> messages went to the other.
>>
>> I opened another producer and still all messages went to the same
consumer.
>>
>> Is this just something new about the shell scripts/ or just a bug in the
>> 0.8.0 version?
>>
>> Tx
>>
>
>
>
> --
> -- Guozhang
>

-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.

Re: command line clients and request balancing.

[Guozhang Wang <wa...@gmail.com>]

Hi Edward,

What is the current version you are using? Starting in 0.8.1 the default
partitioner of the producer would no longer be round-robin at a per-message
level but would stick to one partition for a period (the metadata refresh
period, which is 10 min by default) and then round-robin to another
partition in the next period.

Guozhang

G


On Thu, May 1, 2014 at 3:18 PM, Edward Capriolo <ed...@gmail.com>wrote:

> After installing Kafka I did the following
>
> createTopic --partitions 10 -replication 1 --name mytopic
>
> I started producer using bin/kafka-producer-shell
>
> I opened two consumers in the same group using bin/kafka-consumer-shell
>
> In older versions this test used to result in the messages being sent by
> the producer round robin-ing between the two consumers. However I noticed
> all the messages were going to one consumer until I killed it then all the
> messages went to the other.
>
> I opened another producer and still all messages went to the same consumer.
>
> Is this just something new about the shell scripts/ or just a bug in the
> 0.8.0 version?
>
> Tx
>



-- 
-- Guozhang