Re: Does kafka send the acks response to the producer after flush the messages to the disk or just keep them in the memory

[Jiecxy <25...@qq.com>]

Hi Guozhang,

Thanks for your reply. I’m still confused. I checked the source code.Kafka just uses the class FileChannel.write(buffer) to write the data on the broker, it puts the data in the memory instead of disk. Only you set the FileChannel.force(true), it will flush the data to the disk. I understand the ack mode, but it just requires the isr to receive the data then return the response, in fact, it doesn’t require them to flush the data to the disk. Am I right? Anything wrong?
So my question is, since the flush appears only after called explicitly (or relying on the os), so it is still possible for kafka to loss data (like broker down before the data are flushed to the disk)?

Thanks
Chen 

> 在 2017年2月27日，02:52，Guozhang Wang <wa...@gmail.com> 写道：
> 
> Hello Chen,
> 
> Kafka flushes data to disk (i.e. fsync) asynchronously. Based on the
> ack.mode it will return the response of the produce request to producer
> after it has been replicated (likely in memory) on N partition replicas.
> 
> 
> Guozhang
> 
> On Sun, Feb 26, 2017 at 1:39 AM, Jiecxy <25...@qq.com> wrote:
> 
>> Hi guys,
>> 
>>    Does kafka send the acks response to the producer  after flush the
>> messages to the disk or just keep them in the memory?
>>    How does Kafka flush the messages? By calling the system call, like
>> fsync()?
>> 
>> Thanks
>> Chen
>> 
>> 
> 
> 
> -- 
> -- Guozhang

Re: Does kafka send the acks response to the producer after flush the messages to the disk or just keep them in the memory

[Guozhang Wang <wa...@gmail.com>]

You are right. If all replicas happen to fail at the same time, then even
with ack=everyone your acked sent messages may still be lost. As Kafka
replication documents stated, with N replicas it can tolerate N-1
concurrent failures, hence if you are really unlucky and get N concurrent
failures then replication will not be prevent data loss.

Guozhang

On Tue, Feb 28, 2017 at 5:54 PM, Jiecxy <25...@qq.com> wrote:

> Hi Guozhang,
>
> Thanks for your reply. I’m still confused. I checked the source code.Kafka
> just uses the class FileChannel.write(buffer) to write the data on the
> broker, it puts the data in the memory instead of disk. Only you set the
> FileChannel.force(true), it will flush the data to the disk. I understand
> the ack mode, but it just requires the isr to receive the data then return
> the response, in fact, it doesn’t require them to flush the data to the
> disk. Am I right? Anything wrong?
> So my question is, since the flush appears only after called explicitly
> (or relying on the os), so it is still possible for kafka to loss data
> (like broker down before the data are flushed to the disk)?
>
> Thanks
> Chen
>
> > 在 2017年2月27日，02:52，Guozhang Wang <wa...@gmail.com> 写道：
> >
> > Hello Chen,
> >
> > Kafka flushes data to disk (i.e. fsync) asynchronously. Based on the
> > ack.mode it will return the response of the produce request to producer
> > after it has been replicated (likely in memory) on N partition replicas.
> >
> >
> > Guozhang
> >
> > On Sun, Feb 26, 2017 at 1:39 AM, Jiecxy <25...@qq.com> wrote:
> >
> >> Hi guys,
> >>
> >>    Does kafka send the acks response to the producer  after flush the
> >> messages to the disk or just keep them in the memory?
> >>    How does Kafka flush the messages? By calling the system call, like
> >> fsync()?
> >>
> >> Thanks
> >> Chen
> >>
> >>
> >
> >
> > --
> > -- Guozhang
>
>
>
>


-- 
-- Guozhang