You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-dev@logging.apache.org by "Remko Popma (JIRA)" <ji...@apache.org> on 2014/09/18 06:42:33 UTC
[jira] [Comment Edited] (LOG4J2-431) Create
MemoryMappedFileAppender
[ https://issues.apache.org/jira/browse/LOG4J2-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138418#comment-14138418 ]
Remko Popma edited comment on LOG4J2-431 at 9/18/14 4:41 AM:
-------------------------------------------------------------
Update: user manual documentation is complete except for one outstanding change to rewrite the initial sentence and remove the "Beta" label. Further feedback welcome.
Next item would be a performance test report comparing this appender to RandomAccessFile and File appender, but that is not a showstopper and may be included in a subsequent release.
Java is supposed to be platform-independent so perhaps I worry too much, but since memory-mapped files may have some platform-specific idiosyncrasies, I tested on a number of platforms.
JUnit tests pass on
* 32-bit Windows XP (32 bit Oracle JVM 1.7.0_55)
* 64-bit Windows 7 (64 bit Oracle JVM 1.8.0_05, 64 bit Oracle JVM 1.7.0_55 and 64 bit Oracle JVM 1.6.0_45)
* 64-bit Solaris 10 (64 bit Oracle JVM 1.7.0_06-b24)
* 64-bit RHEL 5.5 (Linux 2.6.18-194.el5) with 64 bit Oracle JDK1.7.0_05-b06
* 64-bit RHEL 6.5 (Linux 2.6.32-431.el6.x86_64) with 64 bit Oracle JDK1.7.0_05-b06 and 64 bit OpenJDK1.7.0_45 (rhel-2.4.3.3.el6-x86_64 u45-b15)
Test scenarios:
* Create new file with default region length, log less than file size. Ensure the log file is shrunk to actually used size, and all log statements are present in the file.
* Append to an existing file. Extend by default region length, log less than file length. Ensure that previous data is not overwritten, new data is appended, and the log file is shrunk to the actually used size when released.
* Create a new file with very small region length, repeatedly log to exceed region length. Ensure all data is correctly logged and the log file is shrunk to the actually used size when released.
More test scenarios welcome.
If no objections I will merge this into master tonight (8-12 hours from now).
was (Author: remkop@yahoo.com):
Update: user manual documentation is complete except for one outstanding change to rewrite the initial sentence and remove the "Beta" label. Further feedback welcome.
Next item would be a performance test report comparing this appender to RandomAccessFile and File appender, but that is not a showstopper and may be included in a subsequent release.
Java is supposed to be platform-independent so perhaps I worry too much, but since memory-mapped files may have some platform-specific idiosyncrasies, I tested on a number of platforms.
JUnit tests pass on
* 32-bit Windows XP (32 bit Oracle JVM 1.7.0_55)
* 64-bit Windows 7 (64 bit Oracle JVM 1.8.0_05 - and a few others, I'll update the exact JVM versions later)
* 64-bit Solaris 10 (64 bit Oracle JVM 1.7.0_06-b24)
* 64-bit RHEL 5.5 (Linux 2.6.18-194.el5) with 64 bit Oracle JDK1.7.0_05-b06
* 64-bit RHEL 6.5 (Linux 2.6.32-431.el6.x86_64) with 64 bit Oracle JDK1.7.0_05-b06 and 64 bit OpenJDK1.7.0_45 (rhel-2.4.3.3.el6-x86_64 u45-b15)
Test scenarios:
* Create new file with default region length, log less than file size. Ensure the log file is shrunk to actually used size, and all log statements are present in the file.
* Append to an existing file. Extend by default region length, log less than file length. Ensure that previous data is not overwritten, new data is appended, and the log file is shrunk to the actually used size when released.
* Create a new file with very small region length, repeatedly log to exceed region length. Ensure all data is correctly logged and the log file is shrunk to the actually used size when released.
More test scenarios welcome.
If no objections I will merge this into master tonight (8-12 hours from now).
> Create MemoryMappedFileAppender
> -------------------------------
>
> Key: LOG4J2-431
> URL: https://issues.apache.org/jira/browse/LOG4J2-431
> Project: Log4j 2
> Issue Type: New Feature
> Components: Appenders
> Reporter: Remko Popma
> Assignee: Remko Popma
> Priority: Minor
> Fix For: 2.1
>
> Attachments: MemoryMappedFileAppender.java, MemoryMappedFileAppenderTest.java, MemoryMappedFileAppenderTest.xml, MemoryMappedFileManager.java, MemoryMappedFileManagerTest.java
>
>
> A memory-mapped file appender may have better performance than the ByteBuffer + RandomAccessFile combination used by the RandomAccessFileAppender.
> *Drawbacks*
> * The drawback is that the file needs to be pre-allocated and only up to the file size can be mapped into memory. When the end of the file is reached the appender would need to extend the file and re-map.
> * Remapping is expensive (I think single-digit millisecond-range, need to check). For low-latency apps this kind of spike may be unacceptable so careful tuning is required.
> * Memory usage: If re-mapping happens too often you lose the performance benefits, so the memory-mapped buffer needs to be fairly large, which uses up memory.
> * At roll-over and shutdown the file should be truncated to immediately after the last written data (otherwise the user is left with a log file that ends in garbage).
> *Advantages*
> Measuring on a Solaris box, the difference between flushing to disk (with {{RandomAccessFile.write(bytes[])}}) and putting data in a MappedByteBuffer is about 20x: around 600ns for a ByteBuffer put and around 12-15 microseconds for a RandomAccessFile.write.
> (Of course different hardware and OS may give different results...)
> *Use cases*
> The difference may be most visible if {{immediateFlush}} is set to {{true}}, which is only recommended if async loggers/appenders are not used. If {{immediateFlush=false}}, the large buffer used by RandomAccessFileAppender means you won't need to touch disk very often.
> So a MemoryMappedFileAppender is most useful in _synchronous_ logging scenarios, where you get the speed of writing to memory but the data is available on disk almost immediately. (MMap writes directly to the OS disk buffer.)
> In case of a application crash, the OS ensures that all data in the buffer will be written to disk. In case of an OS crash the data that was most recently added to the buffer may not be written to disk.
> Because by nature this appender would occupy a fair amount of memory, it is most suitable for applications running on server-class hardware with lots of memory available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org