Fwd: Batching at the socket layer

[Corey Nolet <cj...@gmail.com>]

I'm curious what type of batching Kafka producers do at the socket layer.
For instance, if I have a partitioner that round robin's n messages to a
different partition, am I guaranteed to get n different messages sent over
the socket or is there some micro-batching going on underneath?

I am trying to understand the semantics of the default partitioner and why
it sticks to partitions for 10 minutes. If I were to lower that interval to
1sec, would I acheive better batching that I would if I was to completely
round-robin each message to a different partition?

Re: Batching at the socket layer

[tao xiao <xi...@gmail.com>]

org.apache.kafka.clients.producer.Producer is the new api producer

On Tue, Mar 10, 2015 at 11:22 PM, Corey Nolet <cj...@gmail.com> wrote:

> Thanks Jiangie! So what version is considered the "new api"? Is that the
> javaapi in version 0.8.2?.
>
> On Mon, Mar 9, 2015 at 2:29 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
> > The stickiness of partition only applies to old producer. In new producer
> > we have the round robin for each message. The batching in new producer is
> > per topic partition, the batch size it is controlled by both max batch
> > size and linger time config.
> >
> > Jiangjie (Becket) Qin
> >
> > On 3/9/15, 10:10 AM, "Corey Nolet" <cj...@gmail.com> wrote:
> >
> > >I'm curious what type of batching Kafka producers do at the socket
> layer.
> > >For instance, if I have a partitioner that round robin's n messages to a
> > >different partition, am I guaranteed to get n different messages sent
> over
> > >the socket or is there some micro-batching going on underneath?
> > >
> > >I am trying to understand the semantics of the default partitioner and
> why
> > >it sticks to partitions for 10 minutes. If I were to lower that interval
> > >to
> > >1sec, would I acheive better batching that I would if I was to
> completely
> > >round-robin each message to a different partition?
> >
> >
>



-- 
Regards,
Tao

Re: Batching at the socket layer

[Corey Nolet <cj...@gmail.com>]

Thanks Jiangie! So what version is considered the "new api"? Is that the
javaapi in version 0.8.2?.

On Mon, Mar 9, 2015 at 2:29 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:

> The stickiness of partition only applies to old producer. In new producer
> we have the round robin for each message. The batching in new producer is
> per topic partition, the batch size it is controlled by both max batch
> size and linger time config.
>
> Jiangjie (Becket) Qin
>
> On 3/9/15, 10:10 AM, "Corey Nolet" <cj...@gmail.com> wrote:
>
> >I'm curious what type of batching Kafka producers do at the socket layer.
> >For instance, if I have a partitioner that round robin's n messages to a
> >different partition, am I guaranteed to get n different messages sent over
> >the socket or is there some micro-batching going on underneath?
> >
> >I am trying to understand the semantics of the default partitioner and why
> >it sticks to partitions for 10 minutes. If I were to lower that interval
> >to
> >1sec, would I acheive better batching that I would if I was to completely
> >round-robin each message to a different partition?
>
>

Re: Batching at the socket layer

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

The stickiness of partition only applies to old producer. In new producer
we have the round robin for each message. The batching in new producer is
per topic partition, the batch size it is controlled by both max batch
size and linger time config.

Jiangjie (Becket) Qin

On 3/9/15, 10:10 AM, "Corey Nolet" <cj...@gmail.com> wrote:

>I'm curious what type of batching Kafka producers do at the socket layer.
>For instance, if I have a partitioner that round robin's n messages to a
>different partition, am I guaranteed to get n different messages sent over
>the socket or is there some micro-batching going on underneath?
>
>I am trying to understand the semantics of the default partitioner and why
>it sticks to partitions for 10 minutes. If I were to lower that interval
>to
>1sec, would I acheive better batching that I would if I was to completely
>round-robin each message to a different partition?