[jira] [Updated] (LOG4J2-1197) Improve RollingFileAppender to avoid having stale log events in the buffer

["Remko Popma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LOG4J2-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Remko Popma updated LOG4J2-1197:
--------------------------------
    Description: 
With bufferedIO=true the buffer contains log events that I want to see, but the buffer is only flushed to disk when it is full (which is not frequent enough).

Typical scenario is when the server is not in production, so almost no traffic except for the request manually triggered by the operator, usually in order to debug something.

>From Remko:
{quote}
AsyncAppender or Async Loggers use [smart batching|http://mechanical-sympathy.blogspot.jp/2011/10/smart-batching.html], where the buffer will be flushed when the queue is empty. This results in efficient IO (lumping together IO operations where possible), and at the same time your log file always reflects the most recent message.

For synchronous buffered logging, a smarter flushing mechanism would be useful.

Question is, what would trigger the check (whether the buffer needs to be flushed)?
* Easiest is to check with each log event, but if no new event arrives you still end up with a stale buffer. Better than nothing, but not great.
* An alternative is to check periodically from a background thread and flush if the buffer is non-empty. This probably gives the best results but is fairly complex to implement.
{quote}

----
Note 1: The FlumeAppender uses a maxDelayMillis to address this.
Note 2: Could all synchronous appenders benefit from this improvement?

  was:
With bufferedIO=true the buffer contains log events that I want to see, but the buffer is only flushed to disk when it is full (which is not frequent enough).

Typical scenario is when the server is not in production, so almost no traffic except for the request manually triggered by the operator, usually in order to debug something.

>From Remko:
=====
If you use AsyncAppender or Async Loggers, the buffer will be flushed when the queue is empty. This results in efficient IO (lumping together IO operations where possible), but at the same time your log file always reflects the most recent message.

For synchronous buffered logging, I can see how a smarter flushing mechanism would be useful.

Question is, what would trigger the check (whether the buffer needs to be flushed)?

Easiest is to check with each log event, but if no new event arrives you still end up with a stale buffer. Better than nothing, but not great.

An alternative is to check periodically from a background thread and flush if the buffer is non-empty. This probably gives the best results but is fairly complex to implement.

(...)
Now, with Async Appenders/Loggers, we can use what Martin Thompson calls "smart batching
<http://mechanical-sympathy.blogspot.jp/2011/10/smart-batching.html>": when the background thread processes a log event, it checks if the queue is now empty (no further log events are in the pipeline to be processed) and if it is, it will also call Appender.flush() in addition to the above.
=====

Note 1: The FlumeAppender uses a maxDelayMillis to address this.
Note 2: Could all synchronous appenders benefit from this improvement?


> Improve RollingFileAppender to avoid having stale log events in the buffer
> --------------------------------------------------------------------------
>
>                 Key: LOG4J2-1197
>                 URL: https://issues.apache.org/jira/browse/LOG4J2-1197
>             Project: Log4j 2
>          Issue Type: Improvement
>            Reporter: Benjamin Jaton
>              Labels: appender
>
> With bufferedIO=true the buffer contains log events that I want to see, but the buffer is only flushed to disk when it is full (which is not frequent enough).
> Typical scenario is when the server is not in production, so almost no traffic except for the request manually triggered by the operator, usually in order to debug something.
> From Remko:
> {quote}
> AsyncAppender or Async Loggers use [smart batching|http://mechanical-sympathy.blogspot.jp/2011/10/smart-batching.html], where the buffer will be flushed when the queue is empty. This results in efficient IO (lumping together IO operations where possible), and at the same time your log file always reflects the most recent message.
> For synchronous buffered logging, a smarter flushing mechanism would be useful.
> Question is, what would trigger the check (whether the buffer needs to be flushed)?
> * Easiest is to check with each log event, but if no new event arrives you still end up with a stale buffer. Better than nothing, but not great.
> * An alternative is to check periodically from a background thread and flush if the buffer is non-empty. This probably gives the best results but is fairly complex to implement.
> {quote}
> ----
> Note 1: The FlumeAppender uses a maxDelayMillis to address this.
> Note 2: Could all synchronous appenders benefit from this improvement?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org