You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Manjunath Shivakumar <Ma...@betfair.com> on 2014/08/20 03:00:13 UTC
fsync behaviour
Hi,
I wanted to understand the exact behaviour of kafka ( 0.8.1.1 ) w.r.t fsync. I have read the documentation and some related forum posts, but wanted to confirm the understanding here.
1) If we set the log flush to happen on every message, will it happen synchronously with each produce request, or happen in the background, while the produce request returns immediately?
2) If the fsync happens asynchronous to the produce requests, will the consumers be able to see the messages which are yet to be fsync'ed.
Thanks,
Manjunath
________________________________________________________________________
In order to protect our email recipients, Betfair Group use SkyScan from
MessageLabs to scan all Incoming and Outgoing mail for viruses.
________________________________________________________________________
Re: fsync behaviour
Posted by Jun Rao <ju...@gmail.com>.
That's right. A message is exposed to the consumer as soon as it reaches
all in-sync replicas. Whether it's flushed to disk or not depends on the
configuration of flush interval.
Thanks,
Jun
On Thu, Aug 21, 2014 at 3:37 AM, Manjunath Shivakumar <
Manjunath.Shivakumar@betfair.com> wrote:
> Or to re frame the question when replication is enabled.
>
> Will the message be available to consumers as soon as the message has been
> replicated to an in-sync follower, even before it is flushed to the disk.
>
> Thanks,
> Manju
> ________________________________
> From: Manjunath Shivakumar
> Sent: 21 August 2014 07:01
> To: users@kafka.apache.org
> Subject: Re: fsync behaviour
>
>
> Thanks Jun. If the log flush interval is more than 1, will the unflushed
> messages be visible to the consumers before being written to disk.
>
> Thanks,
> Manju
>
> On 21 Aug 2014 06:03, Jun Rao <ju...@gmail.com> wrote:
> 1) if you configure the log flush interval to be 1 message, every produce
> request will be flushed to disk before the response is sent to the
> producer.
>
> Thanks,
>
> Jun
>
>
> On Tue, Aug 19, 2014 at 6:00 PM, Manjunath Shivakumar <
> Manjunath.Shivakumar@betfair.com> wrote:
>
> > Hi,
> >
> > I wanted to understand the exact behaviour of kafka ( 0.8.1.1 ) w.r.t
> > fsync. I have read the documentation and some related forum posts, but
> > wanted to confirm the understanding here.
> >
> > 1) If we set the log flush to happen on every message, will it happen
> > synchronously with each produce request, or happen in the background,
> while
> > the produce request returns immediately?
> >
> > 2) If the fsync happens asynchronous to the produce requests, will the
> > consumers be able to see the messages which are yet to be fsync'ed.
> >
> > Thanks,
> > Manjunath
> >
> > ________________________________________________________________________
> > In order to protect our email recipients, Betfair Group use SkyScan from
> > MessageLabs to scan all Incoming and Outgoing mail for viruses.
> >
> > ________________________________________________________________________
>
> ________________________________________________________________________
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> ________________________________________________________________________
>
RE: fsync behaviour
Posted by Manjunath Shivakumar <Ma...@betfair.com>.
Or to re frame the question when replication is enabled.
Will the message be available to consumers as soon as the message has been
replicated to an in-sync follower, even before it is flushed to the disk.
Thanks,
Manju
________________________________
From: Manjunath Shivakumar
Sent: 21 August 2014 07:01
To: users@kafka.apache.org
Subject: Re: fsync behaviour
Thanks Jun. If the log flush interval is more than 1, will the unflushed messages be visible to the consumers before being written to disk.
Thanks,
Manju
On 21 Aug 2014 06:03, Jun Rao <ju...@gmail.com> wrote:
1) if you configure the log flush interval to be 1 message, every produce
request will be flushed to disk before the response is sent to the producer.
Thanks,
Jun
On Tue, Aug 19, 2014 at 6:00 PM, Manjunath Shivakumar <
Manjunath.Shivakumar@betfair.com> wrote:
> Hi,
>
> I wanted to understand the exact behaviour of kafka ( 0.8.1.1 ) w.r.t
> fsync. I have read the documentation and some related forum posts, but
> wanted to confirm the understanding here.
>
> 1) If we set the log flush to happen on every message, will it happen
> synchronously with each produce request, or happen in the background, while
> the produce request returns immediately?
>
> 2) If the fsync happens asynchronous to the produce requests, will the
> consumers be able to see the messages which are yet to be fsync'ed.
>
> Thanks,
> Manjunath
>
> ________________________________________________________________________
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> ________________________________________________________________________
________________________________________________________________________
In order to protect our email recipients, Betfair Group use SkyScan from
MessageLabs to scan all Incoming and Outgoing mail for viruses.
________________________________________________________________________
Re: fsync behaviour
Posted by Manjunath Shivakumar <Ma...@betfair.com>.
Thanks Jun. If the log flush interval is more than 1, will the unflushed messages be visible to the consumers before being written to disk.
Thanks,
Manju
On 21 Aug 2014 06:03, Jun Rao <ju...@gmail.com> wrote:
1) if you configure the log flush interval to be 1 message, every produce
request will be flushed to disk before the response is sent to the producer.
Thanks,
Jun
On Tue, Aug 19, 2014 at 6:00 PM, Manjunath Shivakumar <
Manjunath.Shivakumar@betfair.com> wrote:
> Hi,
>
> I wanted to understand the exact behaviour of kafka ( 0.8.1.1 ) w.r.t
> fsync. I have read the documentation and some related forum posts, but
> wanted to confirm the understanding here.
>
> 1) If we set the log flush to happen on every message, will it happen
> synchronously with each produce request, or happen in the background, while
> the produce request returns immediately?
>
> 2) If the fsync happens asynchronous to the produce requests, will the
> consumers be able to see the messages which are yet to be fsync'ed.
>
> Thanks,
> Manjunath
>
> ________________________________________________________________________
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> ________________________________________________________________________
________________________________________________________________________
In order to protect our email recipients, Betfair Group use SkyScan from
MessageLabs to scan all Incoming and Outgoing mail for viruses.
________________________________________________________________________
Re: fsync behaviour
Posted by Jun Rao <ju...@gmail.com>.
1) if you configure the log flush interval to be 1 message, every produce
request will be flushed to disk before the response is sent to the producer.
Thanks,
Jun
On Tue, Aug 19, 2014 at 6:00 PM, Manjunath Shivakumar <
Manjunath.Shivakumar@betfair.com> wrote:
> Hi,
>
> I wanted to understand the exact behaviour of kafka ( 0.8.1.1 ) w.r.t
> fsync. I have read the documentation and some related forum posts, but
> wanted to confirm the understanding here.
>
> 1) If we set the log flush to happen on every message, will it happen
> synchronously with each produce request, or happen in the background, while
> the produce request returns immediately?
>
> 2) If the fsync happens asynchronous to the produce requests, will the
> consumers be able to see the messages which are yet to be fsync'ed.
>
> Thanks,
> Manjunath
>
> ________________________________________________________________________
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> ________________________________________________________________________
FW: fsync behaviour
Posted by Manjunath Shivakumar <Ma...@betfair.com>.
Hi,
Cross posting to devs for elaboration.
Thanks,
Manju
________________________________________
From: Manjunath Shivakumar [Manjunath.Shivakumar@betfair.com]
Sent: 20 August 2014 02:00
To: users@kafka.apache.org
Subject: fsync behaviour
Hi,
I wanted to understand the exact behaviour of kafka ( 0.8.1.1 ) w.r.t fsync. I have read the documentation and some related forum posts, but wanted to confirm the understanding here.
1) If we set the log flush to happen on every message, will it happen synchronously with each produce request, or happen in the background, while the produce request returns immediately?
2) If the fsync happens asynchronous to the produce requests, will the consumers be able to see the messages which are yet to be fsync'ed.
Thanks,
Manjunath
________________________________________________________________________
In order to protect our email recipients, Betfair Group use SkyScan from
MessageLabs to scan all Incoming and Outgoing mail for viruses.
________________________________________________________________________
________________________________________________________________________
In order to protect our email recipients, Betfair Group use SkyScan from
MessageLabs to scan all Incoming and Outgoing mail for viruses.
________________________________________________________________________