Re: log.flush.interval.messages setting of Kafka 0.9.0.0

[+8618611626818 <ka...@126.com>]

someone can help explain it？

发自我的超级手机

2016年12月16日 18:17于 Json Tu <ka...@126.com>写道：
>
> Hi all, 
> we have a cluster of 0.9.0.0 with 3 nodes, we have a topic with 3 replicas, and send it with ack -1, our sending latency is avg 7ms. I prepare to optimize performance of cluster through adjusting some params. 
> we find our brokers has set config item as below, 
> log.flush.interval.messages=10000 
> and other relevant parameter is default, and I find the default value of log.flush.interval.messages is LONG.MAX_VALUE, because of setting this config will flush intiative that may affect performace . I wonder can I cancel this config  item’s setting, and use default value. 
>
> I think use default value may have two drawback as below. 
> 1.recovery checkpoint can not be updated，so when load segments，it will scan from begin to end. 
> 2.it may lose data when leader partition’s broker’s vm is restart，but I think 3 replicas can remedy this drawback if the network between them is good. 
>
> any suggestions? thank you 

Re: log.flush.interval.messages setting of Kafka 0.9.0.0

[Colin McCabe <cm...@apache.org>]

Hi,

Based on a quick look at the code, the log.flush.interval.messages
setting affects how frequently we call fsync on the underlying
filesystem.  As you mentioned, by default this is set to Long.MAX_VALUE,
which effectively disables it.

Keep in mind that the underlying operating system will flush data to
disk even without the use of fsync.  There are a few  kernel parameters
that control this on Linux: vm.dirty_bytes, vm.dirty_background_bytes,
vm.dirty_writeback_centisecs, vm.dirty_expire_centisecs, and so forth. 
In other words, the kernel will not buffer an indefinite amount of data,
even if it has a huge amount of memory.  It will eventually be written
to disk.

I agree that having 3x replication can reduce the negative impact of
disabling fsync this way.  However, keep in mind that if all machines
lose power at once, you will still lose the data in memory, even with 3x
replication.  This is one reason why users who require the highest level
of data durability run with battery backup for their clusters.  Another
reason is because some hard drives on the market are not totally
reliable about flushing the data to the platter when they claim to have
done so.

best,
Colin


On Sat, Dec 17, 2016, at 19:06, +8618611626818 wrote:
> someone can help explain it？
> 
> 发自我的超级手机
> 
> 2016年12月16日 18:17于 Json Tu <ka...@126.com>写道：
> >
> > Hi all, 
> > we have a cluster of 0.9.0.0 with 3 nodes, we have a topic with 3 replicas, and send it with ack -1, our sending latency is avg 7ms. I prepare to optimize performance of cluster through adjusting some params. 
> > we find our brokers has set config item as below, 
> > log.flush.interval.messages=10000 
> > and other relevant parameter is default, and I find the default value of log.flush.interval.messages is LONG.MAX_VALUE, because of setting this config will flush intiative that may affect performace . I wonder can I cancel this config  item’s setting, and use default value. 
> >
> > I think use default value may have two drawback as below. 
> > 1.recovery checkpoint can not be updated，so when load segments，it will scan from begin to end. 
> > 2.it may lose data when leader partition’s broker’s vm is restart，but I think 3 replicas can remedy this drawback if the network between them is good. 
> >
> > any suggestions? thank you