You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by "Alexander Lambert (HE/HIM)" <la...@microsoft.com.INVALID> on 2022/05/04 21:44:10 UTC
RollingFileAppender enhancement for file compression
Hi all,
We have a local modification to logging-log4j2/log4j-core that we are hoping to contribute back to open source. We modify the RollingFileAppender, adding a config option to compress the file as it is originally written instead of on rollover. The value of this is in reducing the volume of data required to be written to disk, improving performance.
The summary of this change is as follows:
* A new Boolean config option "compressOnWrite" for the RollingFileAppender.
* If true, the log is written to e.g. a GZIPOutputStream instead of a FileOutputStream.
* Any rollover compression action implied by the filePattern extension is skipped as the file is already compressed, and so only a rename is done.
Right now our change only applies to the ".gz" compression type, but it would be possible to extend this functionality to the other compression methods - the file extension from the fileName could be used to detect this. The GZIP format is however especially appropriate for this feature:
* No need to enforce an OnStartupTriggeringPolicy for the rollover, as it is valid to append new GZIP data to a closed GZIP file.
Is there any reason that this enhancement would not be accepted into log4j?
Many thanks,
Alex
Re: RollingFileAppender enhancement for file compression
Posted by Ralph Goers <ra...@dslextreme.com>.
How is that different than what it already does? Today we simply inspect the
file extension and compress using that method after rollover.
Ralph
> On May 5, 2022, at 9:47 AM, Priyanka Pardesi Ramachander <ra...@gmail.com> wrote:
>
> Hi All,
>
> How about this approach instead?..
> -> A new Boolean config option "fileCompression" for the
> RollingFileAppender.
> -> If true, the log is written to FileOutputStream (as usual).
> -> And at the time of rollover, compress the file (as .gz, .zip, etc) and
> delete the original file.
> In case of any abnormal termination of the app, once it is up again (if
> filePattern extension is still the same) it will continue to log to the
> same file.
>
>
> Sincerely,
>
> Priyanka
>
>
> On Thu, May 5, 2022 at 3:27 PM Ralph Goers <ra...@dslextreme.com>
> wrote:
>
>> While useful I have concerns as to what the state of the file would be if
>> the application dies. Also, is the file readable as it is being written?
>> The RollingAppenders are already overly complicated so I am reluctant to
>> add new features without doing something to simplify them.
>>
>> Ralph
>>
>>> On May 5, 2022, at 12:34 AM, Matt Sicker <bo...@gmail.com> wrote:
>>>
>>> That sounds pretty useful to me. Using the existing compression
>>> actions could make it so you can more easily support other compression
>>> formats.
>>>
>>> On Wed, May 4, 2022 at 4:44 PM Alexander Lambert (HE/HIM)
>>> <la...@microsoft.com.invalid> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> We have a local modification to logging-log4j2/log4j-core that we are
>> hoping to contribute back to open source. We modify the
>> RollingFileAppender, adding a config option to compress the file as it is
>> originally written instead of on rollover. The value of this is in reducing
>> the volume of data required to be written to disk, improving performance.
>>>>
>>>> The summary of this change is as follows:
>>>>
>>>> * A new Boolean config option "compressOnWrite" for the
>> RollingFileAppender.
>>>> * If true, the log is written to e.g. a GZIPOutputStream instead of
>> a FileOutputStream.
>>>> * Any rollover compression action implied by the filePattern
>> extension is skipped as the file is already compressed, and so only a
>> rename is done.
>>>>
>>>> Right now our change only applies to the ".gz" compression type, but it
>> would be possible to extend this functionality to the other compression
>> methods - the file extension from the fileName could be used to detect
>> this. The GZIP format is however especially appropriate for this feature:
>>>>
>>>> * No need to enforce an OnStartupTriggeringPolicy for the rollover,
>> as it is valid to append new GZIP data to a closed GZIP file.
>>>>
>>>> Is there any reason that this enhancement would not be accepted into
>> log4j?
>>>>
>>>> Many thanks,
>>>> Alex
>>
>>
Re: RollingFileAppender enhancement for file compression
Posted by Priyanka Pardesi Ramachander <ra...@gmail.com>.
Hi All,
How about this approach instead?..
-> A new Boolean config option "fileCompression" for the
RollingFileAppender.
-> If true, the log is written to FileOutputStream (as usual).
-> And at the time of rollover, compress the file (as .gz, .zip, etc) and
delete the original file.
In case of any abnormal termination of the app, once it is up again (if
filePattern extension is still the same) it will continue to log to the
same file.
Sincerely,
Priyanka
On Thu, May 5, 2022 at 3:27 PM Ralph Goers <ra...@dslextreme.com>
wrote:
> While useful I have concerns as to what the state of the file would be if
> the application dies. Also, is the file readable as it is being written?
> The RollingAppenders are already overly complicated so I am reluctant to
> add new features without doing something to simplify them.
>
> Ralph
>
> > On May 5, 2022, at 12:34 AM, Matt Sicker <bo...@gmail.com> wrote:
> >
> > That sounds pretty useful to me. Using the existing compression
> > actions could make it so you can more easily support other compression
> > formats.
> >
> > On Wed, May 4, 2022 at 4:44 PM Alexander Lambert (HE/HIM)
> > <la...@microsoft.com.invalid> wrote:
> >>
> >> Hi all,
> >>
> >> We have a local modification to logging-log4j2/log4j-core that we are
> hoping to contribute back to open source. We modify the
> RollingFileAppender, adding a config option to compress the file as it is
> originally written instead of on rollover. The value of this is in reducing
> the volume of data required to be written to disk, improving performance.
> >>
> >> The summary of this change is as follows:
> >>
> >> * A new Boolean config option "compressOnWrite" for the
> RollingFileAppender.
> >> * If true, the log is written to e.g. a GZIPOutputStream instead of
> a FileOutputStream.
> >> * Any rollover compression action implied by the filePattern
> extension is skipped as the file is already compressed, and so only a
> rename is done.
> >>
> >> Right now our change only applies to the ".gz" compression type, but it
> would be possible to extend this functionality to the other compression
> methods - the file extension from the fileName could be used to detect
> this. The GZIP format is however especially appropriate for this feature:
> >>
> >> * No need to enforce an OnStartupTriggeringPolicy for the rollover,
> as it is valid to append new GZIP data to a closed GZIP file.
> >>
> >> Is there any reason that this enhancement would not be accepted into
> log4j?
> >>
> >> Many thanks,
> >> Alex
>
>
RE: RollingFileAppender enhancement for file compression
Posted by "Alexander Lambert (HE/HIM)" <la...@microsoft.com.INVALID>.
Thanks all!
WRT Ralph's comments:
The compressed active log can be read as it is being written with no problems using e.g. zcat/zless. This can read up to the end of the data despite the GZIP not being closed.
There is a concern about the application dying and not closing the GZIP compression. If we start logging again to a file damaged in this way then we will not be able to read the new data even with zless, as it stops reading at the unexpected new GZIP header. This could however be alleviated with an onStartupTriggeringPolicy for rollover - the worst that could happen in this situation is that one of the rolled over logs would be missing a GZIP trailer and therefore would not work with gunzip, but would still have all its data readable through zless. No log data would be lost.
WRT Volkan's comment:
This enhancement required changes to several areas of logging-log4j2 (e.g. compression actions, file managers) - I'm not sure if it could easily be split off into its own appender.
-----Original Message-----
From: Volkan Yazıcı <vo...@yazi.ci>
Sent: Thursday, May 5, 2022 3:11 PM
To: Apache Logging Developers List <de...@logging.apache.org>
Subject: Re: RollingFileAppender enhancement for file compression
I agree with Ralph here. My suggestion would be to publish this custom appender as a separate F/OSS project first. It can certainly be contributed back to Log4j at a later stage depending on its reception by the community.
On Thu, May 5, 2022 at 9:27 AM Ralph Goers <ra...@dslextreme.com>
wrote:
> While useful I have concerns as to what the state of the file would be
> if the application dies. Also, is the file readable as it is being written?
> The RollingAppenders are already overly complicated so I am reluctant
> to add new features without doing something to simplify them.
>
> Ralph
>
> > On May 5, 2022, at 12:34 AM, Matt Sicker <bo...@gmail.com> wrote:
> >
> > That sounds pretty useful to me. Using the existing compression
> > actions could make it so you can more easily support other
> > compression formats.
> >
> > On Wed, May 4, 2022 at 4:44 PM Alexander Lambert (HE/HIM)
> > <la...@microsoft.com.invalid> wrote:
> >>
> >> Hi all,
> >>
> >> We have a local modification to logging-log4j2/log4j-core that we
> >> are
> hoping to contribute back to open source. We modify the
> RollingFileAppender, adding a config option to compress the file as it
> is originally written instead of on rollover. The value of this is in
> reducing the volume of data required to be written to disk, improving performance.
> >>
> >> The summary of this change is as follows:
> >>
> >> * A new Boolean config option "compressOnWrite" for the
> RollingFileAppender.
> >> * If true, the log is written to e.g. a GZIPOutputStream instead of
> a FileOutputStream.
> >> * Any rollover compression action implied by the filePattern
> extension is skipped as the file is already compressed, and so only a
> rename is done.
> >>
> >> Right now our change only applies to the ".gz" compression type,
> >> but it
> would be possible to extend this functionality to the other
> compression methods - the file extension from the fileName could be
> used to detect this. The GZIP format is however especially appropriate for this feature:
> >>
> >> * No need to enforce an OnStartupTriggeringPolicy for the rollover,
> as it is valid to append new GZIP data to a closed GZIP file.
> >>
> >> Is there any reason that this enhancement would not be accepted
> >> into
> log4j?
> >>
> >> Many thanks,
> >> Alex
>
>
Re: RollingFileAppender enhancement for file compression
Posted by Volkan Yazıcı <vo...@yazi.ci>.
I agree with Ralph here. My suggestion would be to publish this custom
appender as a separate F/OSS project first. It can certainly be contributed
back to Log4j at a later stage depending on its reception by the community.
On Thu, May 5, 2022 at 9:27 AM Ralph Goers <ra...@dslextreme.com>
wrote:
> While useful I have concerns as to what the state of the file would be if
> the application dies. Also, is the file readable as it is being written?
> The RollingAppenders are already overly complicated so I am reluctant to
> add new features without doing something to simplify them.
>
> Ralph
>
> > On May 5, 2022, at 12:34 AM, Matt Sicker <bo...@gmail.com> wrote:
> >
> > That sounds pretty useful to me. Using the existing compression
> > actions could make it so you can more easily support other compression
> > formats.
> >
> > On Wed, May 4, 2022 at 4:44 PM Alexander Lambert (HE/HIM)
> > <la...@microsoft.com.invalid> wrote:
> >>
> >> Hi all,
> >>
> >> We have a local modification to logging-log4j2/log4j-core that we are
> hoping to contribute back to open source. We modify the
> RollingFileAppender, adding a config option to compress the file as it is
> originally written instead of on rollover. The value of this is in reducing
> the volume of data required to be written to disk, improving performance.
> >>
> >> The summary of this change is as follows:
> >>
> >> * A new Boolean config option "compressOnWrite" for the
> RollingFileAppender.
> >> * If true, the log is written to e.g. a GZIPOutputStream instead of
> a FileOutputStream.
> >> * Any rollover compression action implied by the filePattern
> extension is skipped as the file is already compressed, and so only a
> rename is done.
> >>
> >> Right now our change only applies to the ".gz" compression type, but it
> would be possible to extend this functionality to the other compression
> methods - the file extension from the fileName could be used to detect
> this. The GZIP format is however especially appropriate for this feature:
> >>
> >> * No need to enforce an OnStartupTriggeringPolicy for the rollover,
> as it is valid to append new GZIP data to a closed GZIP file.
> >>
> >> Is there any reason that this enhancement would not be accepted into
> log4j?
> >>
> >> Many thanks,
> >> Alex
>
>
Re: RollingFileAppender enhancement for file compression
Posted by Ralph Goers <ra...@dslextreme.com>.
While useful I have concerns as to what the state of the file would be if the application dies. Also, is the file readable as it is being written? The RollingAppenders are already overly complicated so I am reluctant to add new features without doing something to simplify them.
Ralph
> On May 5, 2022, at 12:34 AM, Matt Sicker <bo...@gmail.com> wrote:
>
> That sounds pretty useful to me. Using the existing compression
> actions could make it so you can more easily support other compression
> formats.
>
> On Wed, May 4, 2022 at 4:44 PM Alexander Lambert (HE/HIM)
> <la...@microsoft.com.invalid> wrote:
>>
>> Hi all,
>>
>> We have a local modification to logging-log4j2/log4j-core that we are hoping to contribute back to open source. We modify the RollingFileAppender, adding a config option to compress the file as it is originally written instead of on rollover. The value of this is in reducing the volume of data required to be written to disk, improving performance.
>>
>> The summary of this change is as follows:
>>
>> * A new Boolean config option "compressOnWrite" for the RollingFileAppender.
>> * If true, the log is written to e.g. a GZIPOutputStream instead of a FileOutputStream.
>> * Any rollover compression action implied by the filePattern extension is skipped as the file is already compressed, and so only a rename is done.
>>
>> Right now our change only applies to the ".gz" compression type, but it would be possible to extend this functionality to the other compression methods - the file extension from the fileName could be used to detect this. The GZIP format is however especially appropriate for this feature:
>>
>> * No need to enforce an OnStartupTriggeringPolicy for the rollover, as it is valid to append new GZIP data to a closed GZIP file.
>>
>> Is there any reason that this enhancement would not be accepted into log4j?
>>
>> Many thanks,
>> Alex
Re: RollingFileAppender enhancement for file compression
Posted by Matt Sicker <bo...@gmail.com>.
That sounds pretty useful to me. Using the existing compression
actions could make it so you can more easily support other compression
formats.
On Wed, May 4, 2022 at 4:44 PM Alexander Lambert (HE/HIM)
<la...@microsoft.com.invalid> wrote:
>
> Hi all,
>
> We have a local modification to logging-log4j2/log4j-core that we are hoping to contribute back to open source. We modify the RollingFileAppender, adding a config option to compress the file as it is originally written instead of on rollover. The value of this is in reducing the volume of data required to be written to disk, improving performance.
>
> The summary of this change is as follows:
>
> * A new Boolean config option "compressOnWrite" for the RollingFileAppender.
> * If true, the log is written to e.g. a GZIPOutputStream instead of a FileOutputStream.
> * Any rollover compression action implied by the filePattern extension is skipped as the file is already compressed, and so only a rename is done.
>
> Right now our change only applies to the ".gz" compression type, but it would be possible to extend this functionality to the other compression methods - the file extension from the fileName could be used to detect this. The GZIP format is however especially appropriate for this feature:
>
> * No need to enforce an OnStartupTriggeringPolicy for the rollover, as it is valid to append new GZIP data to a closed GZIP file.
>
> Is there any reason that this enhancement would not be accepted into log4j?
>
> Many thanks,
> Alex