You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yuanjia <yu...@163.com> on 2017/03/01 08:14:43 UTC
when will the messsages be sent to broker
Hi all,
When will the messsages be sent in kafka0.10.0?If I use KafkaProducer.send to send one message, the messsages isn't sent immediately except invoke flush or close.
Thanks.
Yuanjia Li
Re: Re: when will the messsages be sent to broker
Posted by Yuanjia <yu...@163.com>.
Hi Matthias,
I check my testing procedure again and again. I find a problem in that.
I start a producer in debug mode and the breakpoint is suspended to all thread. It maybe block producing thread.
After change to suspend to the main thread, the message is sent out immediately.
Thanks.
Yuanjia Li
From: Matthias J. Sax
Date: 2017-03-02 13:11
To: users
Subject: Re: when will the messsages be sent to broker
That's weird. If batch.size=0 and linger.ms=0 records should get send
out immediately.
buffer.memory is an upper bound for the overall memory. If you write to
multiple partitions, there will be a buffer for each partitions and
buffer.memory is shared over all buffers. Furthermore, if you exceed
buffer.memory your call to Producer#send will block (until it might time
out). I don't think you want to set it to zero.
How do you monitor your Producer to come to the conclusion it's not
sending immediately?
-Matthias
On 3/1/17 6:30 PM, Yuanjia wrote:
> Hi,
> Records are not sent out immediately, even if configing batch.size=0 and linger.ms=0, They are buffered in the memory(buffer.memory).
> I don't want to do sync writes by future.get, which performance is not good.
> Have any configure can increase sending frequency? config buffer.memory=0?
>
> Thanks.
>
> Yuanjia Li
>
> From: Matthias J. Sax
> Date: 2017-03-02 01:42
> To: users
> Subject: Re: when will the messsages be sent to broker
> There is also linger.ms parameter that is an upper bound how long a (not
> yet filled) buffer is hold before sending it even if it's not full.
>
> Furthermore, you can do sync writes and block until producer received
> all acks. But it might have a performance penalty.
>
> http://docs.confluent.io/current/clients/producer.html#synchronous-writes
>
>
>
> -Matthias
>
> On 3/1/17 6:55 AM, Kaufman Ng wrote:
>> By default producer batches records (into a buffer) until batch.size is
>> reached, then records will be sent out. Try lowering batch.size in your
>> producer config.
>>
>> You might want to look at the producer javadoc here:
>> http://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
>>
>> On Wed, Mar 1, 2017 at 3:14 AM, Yuanjia <yu...@163.com> wrote:
>>
>>> Hi all,
>>> When will the messsages be sent in kafka0.10.0?If I use KafkaProducer.send
>>> to send one message, the messsages isn't sent immediately except invoke
>>> flush or close.
>>>
>>> Thanks.
>>>
>>>
>>>
>>> Yuanjia Li
>>>
>>
>>
>>
>
> 【来自网易邮箱的超大附件】
> 邮件带有附件预览链接，若您转发或回复此邮件时不希望对方预览附件，建议您手动删除链接。
>
> signature.asc
> 下载: http://u.163.com/t0/bUAsgYsTI8W0
>
> 预览: http://u.163.com/t0/GHwkOppxnZbj
>
>
【来自网易邮箱的超大附件】
邮件带有附件预览链接，若您转发或回复此邮件时不希望对方预览附件，建议您手动删除链接。
signature.asc
下载: http://u.163.com/t0/Wqpk6rZQV
预览: http://u.163.com/t0/mfTTWdnkW
Re: when will the messsages be sent to broker
Posted by "Matthias J. Sax" <ma...@confluent.io>.
That's weird. If batch.size=0 and linger.ms=0 records should get send
out immediately.
buffer.memory is an upper bound for the overall memory. If you write to
multiple partitions, there will be a buffer for each partitions and
buffer.memory is shared over all buffers. Furthermore, if you exceed
buffer.memory your call to Producer#send will block (until it might time
out). I don't think you want to set it to zero.
How do you monitor your Producer to come to the conclusion it's not
sending immediately?
-Matthias
On 3/1/17 6:30 PM, Yuanjia wrote:
> Hi,
> Records are not sent out immediately, even if configing batch.size=0 and linger.ms=0, They are buffered in the memory(buffer.memory).
> I don't want to do sync writes by future.get, which performance is not good.
> Have any configure can increase sending frequency? config buffer.memory=0?
>
> Thanks.
>
> Yuanjia Li
>
> From: Matthias J. Sax
> Date: 2017-03-02 01:42
> To: users
> Subject: Re: when will the messsages be sent to broker
> There is also linger.ms parameter that is an upper bound how long a (not
> yet filled) buffer is hold before sending it even if it's not full.
>
> Furthermore, you can do sync writes and block until producer received
> all acks. But it might have a performance penalty.
>
> http://docs.confluent.io/current/clients/producer.html#synchronous-writes
>
>
>
> -Matthias
>
> On 3/1/17 6:55 AM, Kaufman Ng wrote:
>> By default producer batches records (into a buffer) until batch.size is
>> reached, then records will be sent out. Try lowering batch.size in your
>> producer config.
>>
>> You might want to look at the producer javadoc here:
>> http://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
>>
>> On Wed, Mar 1, 2017 at 3:14 AM, Yuanjia <yu...@163.com> wrote:
>>
>>> Hi all,
>>> When will the messsages be sent in kafka0.10.0?If I use KafkaProducer.send
>>> to send one message, the messsages isn't sent immediately except invoke
>>> flush or close.
>>>
>>> Thanks.
>>>
>>>
>>>
>>> Yuanjia Li
>>>
>>
>>
>>
>
> 【来自网易邮箱的超大附件】
> 邮件带有附件预览链接，若您转发或回复此邮件时不希望对方预览附件，建议您手动删除链接。
>
> signature.asc
> 下载: http://u.163.com/t0/bUAsgYsTI8W0
>
> 预览: http://u.163.com/t0/GHwkOppxnZbj
>
>
Re: Re: when will the messsages be sent to broker
Posted by Yuanjia <yu...@163.com>.
Hi,
Records are not sent out immediately, even if configing batch.size=0 and linger.ms=0, They are buffered in the memory(buffer.memory).
I don't want to do sync writes by future.get, which performance is not good.
Have any configure can increase sending frequency? config buffer.memory=0?
Thanks.
Yuanjia Li
From: Matthias J. Sax
Date: 2017-03-02 01:42
To: users
Subject: Re: when will the messsages be sent to broker
There is also linger.ms parameter that is an upper bound how long a (not
yet filled) buffer is hold before sending it even if it's not full.
Furthermore, you can do sync writes and block until producer received
all acks. But it might have a performance penalty.
http://docs.confluent.io/current/clients/producer.html#synchronous-writes
-Matthias
On 3/1/17 6:55 AM, Kaufman Ng wrote:
> By default producer batches records (into a buffer) until batch.size is
> reached, then records will be sent out. Try lowering batch.size in your
> producer config.
>
> You might want to look at the producer javadoc here:
> http://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
>
> On Wed, Mar 1, 2017 at 3:14 AM, Yuanjia <yu...@163.com> wrote:
>
>> Hi all,
>> When will the messsages be sent in kafka0.10.0?If I use KafkaProducer.send
>> to send one message, the messsages isn't sent immediately except invoke
>> flush or close.
>>
>> Thanks.
>>
>>
>>
>> Yuanjia Li
>>
>
>
>
【来自网易邮箱的超大附件】
邮件带有附件预览链接，若您转发或回复此邮件时不希望对方预览附件，建议您手动删除链接。
signature.asc
下载: http://u.163.com/t0/bUAsgYsTI8W0
预览: http://u.163.com/t0/GHwkOppxnZbj
Re: when will the messsages be sent to broker
Posted by "Matthias J. Sax" <ma...@confluent.io>.
There is also linger.ms parameter that is an upper bound how long a (not
yet filled) buffer is hold before sending it even if it's not full.
Furthermore, you can do sync writes and block until producer received
all acks. But it might have a performance penalty.
http://docs.confluent.io/current/clients/producer.html#synchronous-writes
-Matthias
On 3/1/17 6:55 AM, Kaufman Ng wrote:
> By default producer batches records (into a buffer) until batch.size is
> reached, then records will be sent out. Try lowering batch.size in your
> producer config.
>
> You might want to look at the producer javadoc here:
> http://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
>
> On Wed, Mar 1, 2017 at 3:14 AM, Yuanjia <yu...@163.com> wrote:
>
>> Hi all,
>> When will the messsages be sent in kafka0.10.0?If I use KafkaProducer.send
>> to send one message, the messsages isn't sent immediately except invoke
>> flush or close.
>>
>> Thanks.
>>
>>
>>
>> Yuanjia Li
>>
>
>
>
Re: when will the messsages be sent to broker
Posted by Kaufman Ng <ka...@confluent.io>.
By default producer batches records (into a buffer) until batch.size is
reached, then records will be sent out. Try lowering batch.size in your
producer config.
You might want to look at the producer javadoc here:
http://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
On Wed, Mar 1, 2017 at 3:14 AM, Yuanjia <yu...@163.com> wrote:
> Hi all,
> When will the messsages be sent in kafka0.10.0?If I use KafkaProducer.send
> to send one message, the messsages isn't sent immediately except invoke
> flush or close.
>
> Thanks.
>
>
>
> Yuanjia Li
>
--
Kaufman Ng
+1 646 961 8063
Solutions Architect | Confluent | www.confluent.io