You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Eugen Dueck <eu...@tworks.co.jp> on 2020/03/12 02:30:02 UTC
Sequential writes make Kafka fast, or so they say
A question about something that was always in the back of my mind.
According to Jay Kreps
> The first [reason that Kafka is so fast despite writing to disk] is that Kafka does only sequential file I/O.
I wonder how true this statement is, because Kafka uses 3 segments per partition. so even with a single topic and partition per broker and disk, it would not be sequential. Now say we have 1000 partitions per broker/disk, i.e. 3000 files. How can concurrent/interleaved writes to thousands of files on a single disk be considered sequential file I/O?
Isn't the reason Kafka is so fast despite writing to disk the fact that it does not fsync to disk, leaving that to the OS? The OS would, I assume, be smart enough to order the writes when it flushes its caches to disk in a way that minimizes random seeks. But then, wouldn't the manner in which Kafka writes to files be more or less irrelevant? Or put differently: If Kafka was synchronously flushing to disk, wouldn't it have to limit itself to writing all partitions for a broker/disk to a single file, if it wanted to do sequential file I/O?
For reading (historical, non-realtime) data that is not in the OS cache, keeping it in append-only files, the statement makes of course sense.
Re: Sequential writes make Kafka fast, or so they say
Posted by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>.
Hey,
I used the default settings. I think, but I didn't check it, it means it
flushed to disk in every 5 seconds.
I just switched to SSDs (it is kinda amazing to do this online on an AWS
EBS disk :D ) and it works fine now.
Regards,
Peter
On Sat, 14 Mar 2020 at 01:27, Eugen Dueck <eu...@tworks.co.jp> wrote:
> Hi Peter
>
> when you were struggling with HDD, were you using custom flush settings,
> or the "do not flush" default?
>
> I think a system like BookKeeper is essential if you want to fsync
> synchronously, like Pulsar does - especially on HDDs. For SSDs, configuring
> multiple directories, i.e. multiple files being written to concurrently,
> may be necessary to maximize throughput.
>
> Eugen
> ________________________________
> 差出人: Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>
> 送信日時: 2020年3月13日 21:42
> 宛先: users@kafka.apache.org <us...@kafka.apache.org>
> 件名: Re: Sequential writes make Kafka fast, or so they say
>
> Hi,
>
> yes, if you write one partition only, it will be sequential. But that's
> unlikely, so in practice, it won't be sequential overall.
> I used AWS EC2 instances with st1 EBS disks, that is the old HDD type
> rotational disk. It struggled to give any kind of performance to support
> our 6000+ partitions. Switching to gp2 SSD solved that in a second.
>
> There are other emerging messaging systems that for this reason writes to
> one single file... like BookKeeper
>
> Peter
>
> On Thu, 12 Mar 2020 at 03:30, Eugen Dueck <eu...@tworks.co.jp> wrote:
>
> > A question about something that was always in the back of my mind.
> >
> > According to Jay Kreps
> >
> > > The first [reason that Kafka is so fast despite writing to disk] is
> that
> > Kafka does only sequential file I/O.
> >
> > I wonder how true this statement is, because Kafka uses 3 segments per
> > partition. so even with a single topic and partition per broker and disk,
> > it would not be sequential. Now say we have 1000 partitions per
> > broker/disk, i.e. 3000 files. How can concurrent/interleaved writes to
> > thousands of files on a single disk be considered sequential file I/O?
> >
> > Isn't the reason Kafka is so fast despite writing to disk the fact that
> it
> > does not fsync to disk, leaving that to the OS? The OS would, I assume,
> be
> > smart enough to order the writes when it flushes its caches to disk in a
> > way that minimizes random seeks. But then, wouldn't the manner in which
> > Kafka writes to files be more or less irrelevant? Or put differently: If
> > Kafka was synchronously flushing to disk, wouldn't it have to limit
> itself
> > to writing all partitions for a broker/disk to a single file, if it
> wanted
> > to do sequential file I/O?
> >
> > For reading (historical, non-realtime) data that is not in the OS cache,
> > keeping it in append-only files, the statement makes of course sense.
> >
>
Re: Sequential writes make Kafka fast, or so they say
Posted by Eugen Dueck <eu...@tworks.co.jp>.
Hi Peter
when you were struggling with HDD, were you using custom flush settings, or the "do not flush" default?
I think a system like BookKeeper is essential if you want to fsync synchronously, like Pulsar does - especially on HDDs. For SSDs, configuring multiple directories, i.e. multiple files being written to concurrently, may be necessary to maximize throughput.
Eugen
________________________________
�����: P��ter Sin��ros-Szab�� <pe...@transferwise.com.INVALID>
�����Օr: 2020��3��13�� 21:42
����: users@kafka.apache.org <us...@kafka.apache.org>
����: Re: Sequential writes make Kafka fast, or so they say
Hi,
yes, if you write one partition only, it will be sequential. But that's
unlikely, so in practice, it won't be sequential overall.
I used AWS EC2 instances with st1 EBS disks, that is the old HDD type
rotational disk. It struggled to give any kind of performance to support
our 6000+ partitions. Switching to gp2 SSD solved that in a second.
There are other emerging messaging systems that for this reason writes to
one single file... like BookKeeper
Peter
On Thu, 12 Mar 2020 at 03:30, Eugen Dueck <eu...@tworks.co.jp> wrote:
> A question about something that was always in the back of my mind.
>
> According to Jay Kreps
>
> > The first [reason that Kafka is so fast despite writing to disk] is that
> Kafka does only sequential file I/O.
>
> I wonder how true this statement is, because Kafka uses 3 segments per
> partition. so even with a single topic and partition per broker and disk,
> it would not be sequential. Now say we have 1000 partitions per
> broker/disk, i.e. 3000 files. How can concurrent/interleaved writes to
> thousands of files on a single disk be considered sequential file I/O?
>
> Isn't the reason Kafka is so fast despite writing to disk the fact that it
> does not fsync to disk, leaving that to the OS? The OS would, I assume, be
> smart enough to order the writes when it flushes its caches to disk in a
> way that minimizes random seeks. But then, wouldn't the manner in which
> Kafka writes to files be more or less irrelevant? Or put differently: If
> Kafka was synchronously flushing to disk, wouldn't it have to limit itself
> to writing all partitions for a broker/disk to a single file, if it wanted
> to do sequential file I/O?
>
> For reading (historical, non-realtime) data that is not in the OS cache,
> keeping it in append-only files, the statement makes of course sense.
>
Re: Sequential writes make Kafka fast, or so they say
Posted by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>.
Hi,
yes, if you write one partition only, it will be sequential. But that's
unlikely, so in practice, it won't be sequential overall.
I used AWS EC2 instances with st1 EBS disks, that is the old HDD type
rotational disk. It struggled to give any kind of performance to support
our 6000+ partitions. Switching to gp2 SSD solved that in a second.
There are other emerging messaging systems that for this reason writes to
one single file... like BookKeeper
Peter
On Thu, 12 Mar 2020 at 03:30, Eugen Dueck <eu...@tworks.co.jp> wrote:
> A question about something that was always in the back of my mind.
>
> According to Jay Kreps
>
> > The first [reason that Kafka is so fast despite writing to disk] is that
> Kafka does only sequential file I/O.
>
> I wonder how true this statement is, because Kafka uses 3 segments per
> partition. so even with a single topic and partition per broker and disk,
> it would not be sequential. Now say we have 1000 partitions per
> broker/disk, i.e. 3000 files. How can concurrent/interleaved writes to
> thousands of files on a single disk be considered sequential file I/O?
>
> Isn't the reason Kafka is so fast despite writing to disk the fact that it
> does not fsync to disk, leaving that to the OS? The OS would, I assume, be
> smart enough to order the writes when it flushes its caches to disk in a
> way that minimizes random seeks. But then, wouldn't the manner in which
> Kafka writes to files be more or less irrelevant? Or put differently: If
> Kafka was synchronously flushing to disk, wouldn't it have to limit itself
> to writing all partitions for a broker/disk to a single file, if it wanted
> to do sequential file I/O?
>
> For reading (historical, non-realtime) data that is not in the OS cache,
> keeping it in append-only files, the statement makes of course sense.
>