You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by Maurice Barnum <mb...@splunk.com.INVALID> on 2021/12/15 17:35:49 UTC
BP-44: direct io entry logging
In order to improve IO utilization, Splunk developed support for logging
entries bypassing the kernel's buffer cache via O_DIRECT. The code has
been in production for several months, running on Linux, with an
implementation for MacOS also included.
BP-44 proposes to merge the code into the mainstream code base.
https://github.com/apache/bookkeeper/pull/2932 is mostly complete merge of
the code base to the Bookkeeper master branch. Some unit tests fail, most
likely due to merge errors that need to be resolved.
Comments appreciated.
BP-47 Re: BP-44: direct io entry logging
Posted by Maurice Barnum <mb...@splunk.com.INVALID>.
updated BP number.
how is https://bookkeeper.apache.org/community/bookkeeper_proposals/
updated?
On Wed, Dec 15, 2021 at 9:35 AM Maurice Barnum <mb...@splunk.com> wrote:
> In order to improve IO utilization, Splunk developed support for logging
> entries bypassing the kernel's buffer cache via O_DIRECT. The code has
> been in production for several months, running on Linux, with an
> implementation for MacOS also included.
>
> BP-44 proposes to merge the code into the mainstream code base.
>
> https://github.com/apache/bookkeeper/pull/2932 is mostly complete merge
> of the code base to the Bookkeeper master branch. Some unit tests fail,
> most likely due to merge errors that need to be resolved.
>
> Comments appreciated.
>
>
Re: BP-44: direct io entry logging
Posted by Andrey Yegorov <an...@datastax.com>.
Hi Maurice,
This is an interesting addition to the Bookkeeper!
Thank you for contributing it back to the community.
Can you share some performance data/charts to highlight benefits of this
feature?
Also, is your workload write-heavy with reads mostly from the cold data?
I imagine this option will increase latency of tailing reads (short delay,
after the data made it to the disk).
On Wed, Dec 15, 2021 at 9:36 AM Maurice Barnum <mb...@splunk.com.invalid>
wrote:
> In order to improve IO utilization, Splunk developed support for logging
> entries bypassing the kernel's buffer cache via O_DIRECT. The code has
> been in production for several months, running on Linux, with an
> implementation for MacOS also included.
>
> BP-44 proposes to merge the code into the mainstream code base.
>
> https://github.com/apache/bookkeeper/pull/2932 is mostly complete merge of
> the code base to the Bookkeeper master branch. Some unit tests fail, most
> likely due to merge errors that need to be resolved.
>
> Comments appreciated.
>
--
Andrey Yegorov
Re: BP-44: direct io entry logging
Posted by Enrico Olivelli <eo...@gmail.com>.
+1 very interesting.
I have been running with DIRECT_IO in Java for a couple of projects and I
saw good results on some Linux kernels.
looking forward to adopting your implementation
Enrico
Il giorno ven 17 dic 2021 alle ore 05:40 Sijie Guo <gu...@gmail.com> ha
scritto:
> +1 Looks very promising
>
> On Wed, Dec 15, 2021 at 9:36 AM Maurice Barnum <mbarnum@splunk.com.invalid
> >
> wrote:
>
> > In order to improve IO utilization, Splunk developed support for logging
> > entries bypassing the kernel's buffer cache via O_DIRECT. The code has
> > been in production for several months, running on Linux, with an
> > implementation for MacOS also included.
> >
> > BP-44 proposes to merge the code into the mainstream code base.
> >
> > https://github.com/apache/bookkeeper/pull/2932 is mostly complete merge
> of
> > the code base to the Bookkeeper master branch. Some unit tests fail,
> most
> > likely due to merge errors that need to be resolved.
> >
> > Comments appreciated.
> >
>
Re: BP-44: direct io entry logging
Posted by Sijie Guo <gu...@gmail.com>.
+1 Looks very promising
On Wed, Dec 15, 2021 at 9:36 AM Maurice Barnum <mb...@splunk.com.invalid>
wrote:
> In order to improve IO utilization, Splunk developed support for logging
> entries bypassing the kernel's buffer cache via O_DIRECT. The code has
> been in production for several months, running on Linux, with an
> implementation for MacOS also included.
>
> BP-44 proposes to merge the code into the mainstream code base.
>
> https://github.com/apache/bookkeeper/pull/2932 is mostly complete merge of
> the code base to the Bookkeeper master branch. Some unit tests fail, most
> likely due to merge errors that need to be resolved.
>
> Comments appreciated.
>