You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kunal Goyal <ku...@cohesity.com> on 2021/08/20 05:01:55 UTC
Ensuring that the message is persisted after acknowledgement
Hello,
We are exploring using Kafka for our application. Our requirement is that
once we write some messages to Kafka, it should be guaranteed that the
messages are persisted to disk.
We found this
<https://www.quora.com/Does-Kafka-sync-data-to-disk-asynchronously-like-Redis-does>
article which says that a Kafka broker acknowledges a record after it has
written the record to the buffer of the I/O device; it does not issue an
explicit fsync operation nor does it wait for the OS to confirm that the
data has been written. Is this statement true for the current
implementation? If so, is there any way in which we can ensure fsync is
called before acknowledgement of messages?
Any help would be appreciated.
--
Thanks & Regards
Kunal Goyal
Re: Ensuring that the message is persisted after acknowledgement
Posted by Eric Azama <ea...@gmail.com>.
Rather than forcing writes to disk after each message, the usual method of
ensuring durability is to configure the topic with a Replication Factor of
3, min.insync.replicas=2 and have producers configure acks=all.
This ensures that the record has been replicated to at least 2 brokers
before an acknowledgement is sent back to the producer.
If you really want to force an fsync after reach record, you could
configure the brokers with log.flush.interval.messages=1
https://kafka.apache.org/documentation/#brokerconfigs_log.flush.interval.messages
If you only want that behavior for certain topics, you could configure the
topic with flush.messages=1
https://kafka.apache.org/documentation/#topicconfigs_flush.messages
Do note that this is explicitly not recommended in the documentation.
On Fri, Aug 20, 2021 at 8:01 AM sunil chaudhari <su...@gmail.com>
wrote:
> Hi Kunal,
> This article may help you.
>
> https://betterprogramming.pub/kafka-acks-explained-c0515b3b707e
>
>
> Cheers,
> Sunil.
>
> On Fri, 20 Aug 2021 at 8:11 PM, Kunal Goyal <ku...@cohesity.com>
> wrote:
>
> > Hello,
> >
> > We are exploring using Kafka for our application. Our requirement is that
> > once we write some messages to Kafka, it should be guaranteed that the
> > messages are persisted to disk.
> > We found this
> > <
> >
> https://www.quora.com/Does-Kafka-sync-data-to-disk-asynchronously-like-Redis-does
> > >
> > article which says that a Kafka broker acknowledges a record after it has
> > written the record to the buffer of the I/O device; it does not issue an
> > explicit fsync operation nor does it wait for the OS to confirm that the
> > data has been written. Is this statement true for the current
> > implementation? If so, is there any way in which we can ensure fsync is
> > called before acknowledgement of messages?
> > Any help would be appreciated.
> >
> > --
> >
> > Thanks & Regards
> >
> > Kunal Goyal
> >
>
Re: Ensuring that the message is persisted after acknowledgement
Posted by Peter Bukowinski <pm...@gmail.com>.
Kunal,
I recommend looking at the broker and topic parameters that include the term “flush” , such as https://kafka.apache.org/documentation/#topicconfigs_flush.messages <https://kafka.apache.org/documentation/#topicconfigs_flush.messages>
Kafka lets you configure how often log messages are flushed to disk, either per topic or globally. The default settings leave the flushing completely to the OS. Kafka was designed to take full advantage of the OS page cache because it significantly improves performance for both producers and consumers, allowing them to write to and read from memory.
If your application requires absolute disk persistence and you are willing to take a significant performance hit, you can set the topic property flush.messages to 1 for any topic that requires this guarantee.
—
Peter
> On Aug 24, 2021, at 10:31 PM, Kunal Goyal <ku...@cohesity.com> wrote:
>
> Hi Sunil
>
> The article that you shared talks about acks. But even if the message is
> received by all in-sync replicas and kafka sends response back to the
> producer, it is possible that none of the replicas did not flush the
> messages to disk. So, if all the replicas crash for some reason, the
> messages would be lost. For our application, we require some way to
> guarantee that the messages are persisted to disk.
>
> Regards,
> Kunal
>
> On Tue, Aug 24, 2021 at 8:40 PM Vairavanathan Emalayan <
> vairavanathan.emalayan@cohesity.com> wrote:
>
>>
>>
>> ---------- Forwarded message ---------
>> From: sunil chaudhari <su...@gmail.com>
>> Date: Fri, Aug 20, 2021 at 8:00 AM
>> Subject: Re: Ensuring that the message is persisted after acknowledgement
>> To: <us...@kafka.apache.org>
>> Cc: Vairavanathan Emalayan <va...@cohesity.com>
>>
>>
>> Hi Kunal,
>> This article may help you.
>>
>> https://betterprogramming.pub/kafka-acks-explained-c0515b3b707e
>>
>>
>> Cheers,
>> Sunil.
>>
>> On Fri, 20 Aug 2021 at 8:11 PM, Kunal Goyal <ku...@cohesity.com>
>> wrote:
>>
>>> Hello,
>>>
>>> We are exploring using Kafka for our application. Our requirement is that
>>> once we write some messages to Kafka, it should be guaranteed that the
>>> messages are persisted to disk.
>>> We found this
>>> <
>>> https://www.quora.com/Does-Kafka-sync-data-to-disk-asynchronously-like-Redis-does
>>>>
>>> article which says that a Kafka broker acknowledges a record after it has
>>> written the record to the buffer of the I/O device; it does not issue an
>>> explicit fsync operation nor does it wait for the OS to confirm that the
>>> data has been written. Is this statement true for the current
>>> implementation? If so, is there any way in which we can ensure fsync is
>>> called before acknowledgement of messages?
>>> Any help would be appreciated.
>>>
>>> --
>>>
>>> Thanks & Regards
>>>
>>> Kunal Goyal
>>>
>>
Re: Ensuring that the message is persisted after acknowledgement
Posted by Kunal Goyal <ku...@cohesity.com>.
Hi Sunil
The article that you shared talks about acks. But even if the message is
received by all in-sync replicas and kafka sends response back to the
producer, it is possible that none of the replicas did not flush the
messages to disk. So, if all the replicas crash for some reason, the
messages would be lost. For our application, we require some way to
guarantee that the messages are persisted to disk.
Regards,
Kunal
On Tue, Aug 24, 2021 at 8:40 PM Vairavanathan Emalayan <
vairavanathan.emalayan@cohesity.com> wrote:
>
>
> ---------- Forwarded message ---------
> From: sunil chaudhari <su...@gmail.com>
> Date: Fri, Aug 20, 2021 at 8:00 AM
> Subject: Re: Ensuring that the message is persisted after acknowledgement
> To: <us...@kafka.apache.org>
> Cc: Vairavanathan Emalayan <va...@cohesity.com>
>
>
> Hi Kunal,
> This article may help you.
>
> https://betterprogramming.pub/kafka-acks-explained-c0515b3b707e
>
>
> Cheers,
> Sunil.
>
> On Fri, 20 Aug 2021 at 8:11 PM, Kunal Goyal <ku...@cohesity.com>
> wrote:
>
>> Hello,
>>
>> We are exploring using Kafka for our application. Our requirement is that
>> once we write some messages to Kafka, it should be guaranteed that the
>> messages are persisted to disk.
>> We found this
>> <
>> https://www.quora.com/Does-Kafka-sync-data-to-disk-asynchronously-like-Redis-does
>> >
>> article which says that a Kafka broker acknowledges a record after it has
>> written the record to the buffer of the I/O device; it does not issue an
>> explicit fsync operation nor does it wait for the OS to confirm that the
>> data has been written. Is this statement true for the current
>> implementation? If so, is there any way in which we can ensure fsync is
>> called before acknowledgement of messages?
>> Any help would be appreciated.
>>
>> --
>>
>> Thanks & Regards
>>
>> Kunal Goyal
>>
>
Re: Ensuring that the message is persisted after acknowledgement
Posted by sunil chaudhari <su...@gmail.com>.
Hi Kunal,
This article may help you.
https://betterprogramming.pub/kafka-acks-explained-c0515b3b707e
Cheers,
Sunil.
On Fri, 20 Aug 2021 at 8:11 PM, Kunal Goyal <ku...@cohesity.com>
wrote:
> Hello,
>
> We are exploring using Kafka for our application. Our requirement is that
> once we write some messages to Kafka, it should be guaranteed that the
> messages are persisted to disk.
> We found this
> <
> https://www.quora.com/Does-Kafka-sync-data-to-disk-asynchronously-like-Redis-does
> >
> article which says that a Kafka broker acknowledges a record after it has
> written the record to the buffer of the I/O device; it does not issue an
> explicit fsync operation nor does it wait for the OS to confirm that the
> data has been written. Is this statement true for the current
> implementation? If so, is there any way in which we can ensure fsync is
> called before acknowledgement of messages?
> Any help would be appreciated.
>
> --
>
> Thanks & Regards
>
> Kunal Goyal
>