You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-dev@logging.apache.org by Matt Sicker <bo...@gmail.com> on 2014/04/27 03:48:55 UTC
Would it be useful to implement nio channel-based appenders?
I see that the current FileAppender/FileManager classes (and parents) all
work via streams. Am I missing something, or are there no appenders using
NIO? This could be a neat feature for 2.1.
--
Matt Sicker <bo...@gmail.com>
Re: Would it be useful to implement nio channel-based appenders?
Posted by Matt Sicker <bo...@gmail.com>.
Good to know! The only thing from NIO I've ever used are the buffer APIs
anyway, so I was curious about the channels. At least we can use that as
you suggested, though.
On 27 April 2014 06:43, Remko Popma <re...@gmail.com> wrote:
> AFAIK the only way to do non-blocking file IO is by doing it
> asynchronously via a non-blocking queue like the disruptor.
>
> For networking IO the channel/selector stuff is interesting when you need
> to scale up to tens of thousands of connections and you get beyond the
> place where you can have a dedicated thread per connection. As Ralph
> mentioned, many experts in this area warn that this is much, much harder to
> get right and only gives you better _scalability_, not better throughput
> performance.
>
> That said, what I find interesting about NIO is the ByteBuffer API.
> It has a more powerful API than streams, but gives a single API that still
> abstracts over whether you are writing to a slab of heap memory, off-heap
> memory, a memory-mapped file, or even directly to a device. I like the
> fact that it combines a stream-like API (non-indexed get/put methods) with
> a random-access (the indexed get/put methods) API, and the various other
> goodies like bulk get/put, views, compacting, slicing etc.
> The price you pay is slightly higher complexity. When someone passes you a
> ByteBuffer and you want to read from it, the question is always, has this
> buffer already been flipped(), or do I need to flip() it myself, and this
> can be a source of confusion and bugs.
>
> A ByteBuffer API may not be applicable everywhere, but when used
> judiciously it can help avoid unnecessary copying and improve performance.
> I'm thinking it may be possible to have zero copy by using a ByteBuffer API
> for things like binary loggers (
> https://issues.apache.org/jira/browse/LOG4J2-506 ), in combination with
> the memory-mapped file appender.
>
> Perhaps a bit of a narrow field though... :-)
>
> Remko
>
>
>
> On Sun, Apr 27, 2014 at 7:20 PM, Ralph Goers <rg...@apache.org> wrote:
>
>> OK, but again, why? The articles I've sen say it is harder to implement
>> and slower. I might be able to see using it for sockets but I have a hard
>> time seeing what benefit there would be for file appenders.
>>
>> Ralph
>>
>> On Apr 27, 2014, at 12:16 AM, Matt Sicker <bo...@gmail.com> wrote:
>>
>> The neat feature is non-blocking IO. We could use the Channel API for
>> non-blocking IO in appenders.
>>
>>
>> On 26 April 2014 23:35, Ralph Goers <ra...@dslextreme.com> wrote:
>>
>>> 1. Why would it be a neat feature?
>>> 2. The FileManager uses NIO to lock the file if it was requested.
>>> 3. Many of the Appenders use Streams. I happen to like that as it allows
>>> them to share common code.
>>>
>>> Ralph
>>>
>>> On Apr 26, 2014, at 6:48 PM, Matt Sicker <bo...@gmail.com> wrote:
>>>
>>> I see that the current FileAppender/FileManager classes (and parents)
>>> all work via streams. Am I missing something, or are there no appenders
>>> using NIO? This could be a neat feature for 2.1.
>>>
>>> --
>>> Matt Sicker <bo...@gmail.com>
>>>
>>>
>>>
>>
>>
>> --
>> Matt Sicker <bo...@gmail.com>
>>
>>
>
--
Matt Sicker <bo...@gmail.com>
Re: Would it be useful to implement nio channel-based appenders?
Posted by Remko Popma <re...@gmail.com>.
AFAIK the only way to do non-blocking file IO is by doing it asynchronously
via a non-blocking queue like the disruptor.
For networking IO the channel/selector stuff is interesting when you need
to scale up to tens of thousands of connections and you get beyond the
place where you can have a dedicated thread per connection. As Ralph
mentioned, many experts in this area warn that this is much, much harder to
get right and only gives you better _scalability_, not better throughput
performance.
That said, what I find interesting about NIO is the ByteBuffer API.
It has a more powerful API than streams, but gives a single API that still
abstracts over whether you are writing to a slab of heap memory, off-heap
memory, a memory-mapped file, or even directly to a device. I like the
fact that it combines a stream-like API (non-indexed get/put methods) with
a random-access (the indexed get/put methods) API, and the various other
goodies like bulk get/put, views, compacting, slicing etc.
The price you pay is slightly higher complexity. When someone passes you a
ByteBuffer and you want to read from it, the question is always, has this
buffer already been flipped(), or do I need to flip() it myself, and this
can be a source of confusion and bugs.
A ByteBuffer API may not be applicable everywhere, but when used
judiciously it can help avoid unnecessary copying and improve performance.
I'm thinking it may be possible to have zero copy by using a ByteBuffer API
for things like binary loggers (
https://issues.apache.org/jira/browse/LOG4J2-506 ), in combination with the
memory-mapped file appender.
Perhaps a bit of a narrow field though... :-)
Remko
On Sun, Apr 27, 2014 at 7:20 PM, Ralph Goers <rg...@apache.org> wrote:
> OK, but again, why? The articles I've sen say it is harder to implement
> and slower. I might be able to see using it for sockets but I have a hard
> time seeing what benefit there would be for file appenders.
>
> Ralph
>
> On Apr 27, 2014, at 12:16 AM, Matt Sicker <bo...@gmail.com> wrote:
>
> The neat feature is non-blocking IO. We could use the Channel API for
> non-blocking IO in appenders.
>
>
> On 26 April 2014 23:35, Ralph Goers <ra...@dslextreme.com> wrote:
>
>> 1. Why would it be a neat feature?
>> 2. The FileManager uses NIO to lock the file if it was requested.
>> 3. Many of the Appenders use Streams. I happen to like that as it allows
>> them to share common code.
>>
>> Ralph
>>
>> On Apr 26, 2014, at 6:48 PM, Matt Sicker <bo...@gmail.com> wrote:
>>
>> I see that the current FileAppender/FileManager classes (and parents) all
>> work via streams. Am I missing something, or are there no appenders using
>> NIO? This could be a neat feature for 2.1.
>>
>> --
>> Matt Sicker <bo...@gmail.com>
>>
>>
>>
>
>
> --
> Matt Sicker <bo...@gmail.com>
>
>
Re: Would it be useful to implement nio channel-based appenders?
Posted by Ralph Goers <rg...@apache.org>.
OK, but again, why? The articles I've sen say it is harder to implement and slower. I might be able to see using it for sockets but I have a hard time seeing what benefit there would be for file appenders.
Ralph
> On Apr 27, 2014, at 12:16 AM, Matt Sicker <bo...@gmail.com> wrote:
>
> The neat feature is non-blocking IO. We could use the Channel API for non-blocking IO in appenders.
>
>
>> On 26 April 2014 23:35, Ralph Goers <ra...@dslextreme.com> wrote:
>> 1. Why would it be a neat feature?
>> 2. The FileManager uses NIO to lock the file if it was requested.
>> 3. Many of the Appenders use Streams. I happen to like that as it allows them to share common code.
>>
>> Ralph
>>
>>> On Apr 26, 2014, at 6:48 PM, Matt Sicker <bo...@gmail.com> wrote:
>>>
>>> I see that the current FileAppender/FileManager classes (and parents) all work via streams. Am I missing something, or are there no appenders using NIO? This could be a neat feature for 2.1.
>>>
>>> --
>>> Matt Sicker <bo...@gmail.com>
>
>
>
> --
> Matt Sicker <bo...@gmail.com>
Re: Would it be useful to implement nio channel-based appenders?
Posted by Matt Sicker <bo...@gmail.com>.
The neat feature is non-blocking IO. We could use the Channel API for
non-blocking IO in appenders.
On 26 April 2014 23:35, Ralph Goers <ra...@dslextreme.com> wrote:
> 1. Why would it be a neat feature?
> 2. The FileManager uses NIO to lock the file if it was requested.
> 3. Many of the Appenders use Streams. I happen to like that as it allows
> them to share common code.
>
> Ralph
>
> On Apr 26, 2014, at 6:48 PM, Matt Sicker <bo...@gmail.com> wrote:
>
> I see that the current FileAppender/FileManager classes (and parents) all
> work via streams. Am I missing something, or are there no appenders using
> NIO? This could be a neat feature for 2.1.
>
> --
> Matt Sicker <bo...@gmail.com>
>
>
>
--
Matt Sicker <bo...@gmail.com>
Re: Would it be useful to implement nio channel-based appenders?
Posted by Ralph Goers <ra...@dslextreme.com>.
1. Why would it be a neat feature?
2. The FileManager uses NIO to lock the file if it was requested.
3. Many of the Appenders use Streams. I happen to like that as it allows them to share common code.
Ralph
On Apr 26, 2014, at 6:48 PM, Matt Sicker <bo...@gmail.com> wrote:
> I see that the current FileAppender/FileManager classes (and parents) all work via streams. Am I missing something, or are there no appenders using NIO? This could be a neat feature for 2.1.
>
> --
> Matt Sicker <bo...@gmail.com>