You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by Chandra <ch...@rwth-aachen.de> on 2017/11/16 05:41:05 UTC
Compression on rolling leads to OOM
Guys,
I need some input on how this handle situation:
we are on HP/LL setting where the incoming requests are processed and logged ( buffered, of course with “async logging”). There are some situations where due to spikes in the volume of requests, the compression on rolling creates memory starvation.
Now, a straight forward fix would to remove it from the “context” of jvm and use a script to monitor and compress it timely. as low-hanging the solution may be, I am skeptical of this solution as the script _may_ fail leading to disk starvation(!)
I am looking for an alternate and _full-proof_ solution for this situation. Any thoughts, suggestions would be useful and appreciated.
thanks!
Chandra
On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <ra...@dslextreme.com>, wrote:
> Unless someone objects I plan to start the release process for Log4j 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but the problem only occurs in rare situations and I am not sure how to fix it yet. There are a lot of other issues that deserve looking at but nothing I can see that warrants waiting on.
>
> Ralph
Re: Compression on rolling leads to OOM
Posted by Ralph Goers <ra...@dslextreme.com>.
What do you mean “It is the same as running it out of the jvm”? ProcessBuilder creates processes that execute commands. They don’t have to be Java. That said, it certainly could be a Java environment that we control that can only run the asynchronous actions Log4j supports.
Ralph
> On Nov 16, 2017, at 11:37 AM, Chandra <ch...@rwth-aachen.de> wrote:
>
> well, ProcessBuilder.start() and Runtime.exec is one way to go but it’s the same as running it out of jvm.
>
> On 17 Nov 2017, 12:05 AM +0530, Matt Sicker <bo...@gmail.com>, wrote:
>> I don't think Java supports process forking (not even sure if Windows
>> does), but you can execute and control processes at least.
>>
>> On 16 November 2017 at 12:28, Ralph Goers <ralph.goers@dslextreme.com
>> wrote:
>>
>>> Sure, why not?
>>>
>>> Ralph
>>>
>>>> On Nov 16, 2017, at 11:24 AM, Chandra <chandra.tungathurthi@rwth-
>>> aachen.de> wrote:
>>>>
>>>> with-in log4j?
>>>>
>>>> On 16 Nov 2017, 11:46 PM +0530, Ralph Goers <ra...@dslextreme.com>,
>>> wrote:
>>>>> I could see forking a process instead of spawning a thread to perform
>>> the compression as a viable approach.
>>>>>
>>>>> Ralph
>>>>>
>>>>>> On Nov 16, 2017, at 10:01 AM, Chandra <chandra.tungathurthi@rwth-
>>> aachen.de> wrote:
>>>>>>
>>>>>> thanks for info matt, was planning to use this on some other project
>>> in mind ( might pick your brain on that later ;) )
>>>>>>
>>>>>> separating it out the compression to external process isn’t
>>> necessarily a bad idea, but having non-reliable scripts is. As it so
>>> happened many times before. I’d rather depend on my app than an external
>>> process.
>>>>>> which is why I was looking for an “agent” .
>>>>>>
>>>>>> thanks!
>>>>>> chandra
>>>>>>
>>>>>>
>>>>>> On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
>>>>>>> I brought up Snappy only because I used their off-heap API recently.
>>> Snappy
>>>>>>> is more about real time compression rather than size (I think snappy
>>> files
>>>>>>> tend to be larger than gzip files, but take less resources to
>>> compress and
>>>>>>> decompress). The idea here is to offer support via libraries using
>>> native
>>>>>>> implementations that can work with direct byte buffers, mmap'd files,
>>> or
>>>>>>> even just a file name.
>>>>>>>
>>>>>>> With that in mind, is it so bad to offer the ability to execute an
>>> external
>>>>>>> process to compress the file?
>>>>>>>
>>>>>>> On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-
>>> aachen.de
>>>>>>> wrote:
>>>>>>>
>>>>>>>>> What if compression worked off-heap
>>>>>>>> off-heap compression sounds interesting. Let me check if I can find
>>> any.
>>>>>>>>
>>>>>>>> Snappy’s compression is a different altogether, I am not necessarily
>>>>>>>> looking for a different compression formats, as I’d have add support
>>> for it
>>>>>>>> in downstream. Standard bz2 , gzip would work in-fact.
>>>>>>>>
>>>>>>>>
>>>>>>>> Ideally a reliable agent something like `FileBeat` would be great
>>> for this
>>>>>>>> situation. :-/
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Chandra
>>>>>>>>
>>>>>>>>
>>>>>>>> On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>,
>>> wrote:
>>>>>>>>> What if compression worked off-heap like with some of the native
>>>>>>>>> implementations of codecs? I'm thinking of this one <
>>>>>>>>> https://github.com/xerial/snappy-java> for example.
>>>>>>>>>
>>>>>>>>> On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
>>>>>>>> aachen.de
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Guys,
>>>>>>>>>>
>>>>>>>>>> I need some input on how this handle situation:
>>>>>>>>>>
>>>>>>>>>> we are on HP/LL setting where the incoming requests are processed
>>> and
>>>>>>>>>> logged ( buffered, of course with “async logging”). There are some
>>>>>>>>>> situations where due to spikes in the volume of requests, the
>>>>>>>> compression
>>>>>>>>>> on rolling creates memory starvation.
>>>>>>>>>>
>>>>>>>>>> Now, a straight forward fix would to remove it from the “context”
>>> of
>>>>>>>> jvm
>>>>>>>>>> and use a script to monitor and compress it timely. as low-hanging
>>> the
>>>>>>>>>> solution may be, I am skeptical of this solution as the script
>>> _may_
>>>>>>>> fail
>>>>>>>>>> leading to disk starvation(!)
>>>>>>>>>>
>>>>>>>>>> I am looking for an alternate and _full-proof_ solution for this
>>>>>>>>>> situation. Any thoughts, suggestions would be useful and
>>> appreciated.
>>>>>>>>>>
>>>>>>>>>> thanks!
>>>>>>>>>> Chandra
>>>>>>>>>>
>>>>>>>>>> On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
>>>>>>>> ralph.goers@dslextreme.com>,
>>>>>>>>>> wrote:
>>>>>>>>>>> Unless someone objects I plan to start the release process for
>>> Log4j
>>>>>>>>>> 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but
>>> the
>>>>>>>>>> problem only occurs in rare situations and I am not sure how to
>>> fix it
>>>>>>>> yet.
>>>>>>>>>> There are a lot of other issues that deserve looking at but
>>> nothing I
>>>>>>>> can
>>>>>>>>>> see that warrants waiting on.
>>>>>>>>>>>
>>>>>>>>>>> Ralph
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Matt Sicker <boards@gmail.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Matt Sicker <boards@gmail.com
>>>>>
>>>>>
>>>
>>>
>>>
>>
>>
>> --
>> Matt Sicker <boards@gmail.com
Re: Compression on rolling leads to OOM
Posted by Chandra <ch...@rwth-aachen.de>.
well, ProcessBuilder.start() and Runtime.exec is one way to go but it’s the same as running it out of jvm.
On 17 Nov 2017, 12:05 AM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> I don't think Java supports process forking (not even sure if Windows
> does), but you can execute and control processes at least.
>
> On 16 November 2017 at 12:28, Ralph Goers <ralph.goers@dslextreme.com
> wrote:
>
> > Sure, why not?
> >
> > Ralph
> >
> > > On Nov 16, 2017, at 11:24 AM, Chandra <chandra.tungathurthi@rwth-
> > aachen.de> wrote:
> > >
> > > with-in log4j?
> > >
> > > On 16 Nov 2017, 11:46 PM +0530, Ralph Goers <ra...@dslextreme.com>,
> > wrote:
> > > > I could see forking a process instead of spawning a thread to perform
> > the compression as a viable approach.
> > > >
> > > > Ralph
> > > >
> > > > > On Nov 16, 2017, at 10:01 AM, Chandra <chandra.tungathurthi@rwth-
> > aachen.de> wrote:
> > > > >
> > > > > thanks for info matt, was planning to use this on some other project
> > in mind ( might pick your brain on that later ;) )
> > > > >
> > > > > separating it out the compression to external process isn’t
> > necessarily a bad idea, but having non-reliable scripts is. As it so
> > happened many times before. I’d rather depend on my app than an external
> > process.
> > > > > which is why I was looking for an “agent” .
> > > > >
> > > > > thanks!
> > > > > chandra
> > > > >
> > > > >
> > > > > On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> > > > > > I brought up Snappy only because I used their off-heap API recently.
> > Snappy
> > > > > > is more about real time compression rather than size (I think snappy
> > files
> > > > > > tend to be larger than gzip files, but take less resources to
> > compress and
> > > > > > decompress). The idea here is to offer support via libraries using
> > native
> > > > > > implementations that can work with direct byte buffers, mmap'd files,
> > or
> > > > > > even just a file name.
> > > > > >
> > > > > > With that in mind, is it so bad to offer the ability to execute an
> > external
> > > > > > process to compress the file?
> > > > > >
> > > > > > On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-
> > aachen.de
> > > > > > wrote:
> > > > > >
> > > > > > > > What if compression worked off-heap
> > > > > > > off-heap compression sounds interesting. Let me check if I can find
> > any.
> > > > > > >
> > > > > > > Snappy’s compression is a different altogether, I am not necessarily
> > > > > > > looking for a different compression formats, as I’d have add support
> > for it
> > > > > > > in downstream. Standard bz2 , gzip would work in-fact.
> > > > > > >
> > > > > > >
> > > > > > > Ideally a reliable agent something like `FileBeat` would be great
> > for this
> > > > > > > situation. :-/
> > > > > > >
> > > > > > > Best,
> > > > > > > Chandra
> > > > > > >
> > > > > > >
> > > > > > > On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>,
> > wrote:
> > > > > > > > What if compression worked off-heap like with some of the native
> > > > > > > > implementations of codecs? I'm thinking of this one <
> > > > > > > > https://github.com/xerial/snappy-java> for example.
> > > > > > > >
> > > > > > > > On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
> > > > > > > aachen.de
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Guys,
> > > > > > > > >
> > > > > > > > > I need some input on how this handle situation:
> > > > > > > > >
> > > > > > > > > we are on HP/LL setting where the incoming requests are processed
> > and
> > > > > > > > > logged ( buffered, of course with “async logging”). There are some
> > > > > > > > > situations where due to spikes in the volume of requests, the
> > > > > > > compression
> > > > > > > > > on rolling creates memory starvation.
> > > > > > > > >
> > > > > > > > > Now, a straight forward fix would to remove it from the “context”
> > of
> > > > > > > jvm
> > > > > > > > > and use a script to monitor and compress it timely. as low-hanging
> > the
> > > > > > > > > solution may be, I am skeptical of this solution as the script
> > _may_
> > > > > > > fail
> > > > > > > > > leading to disk starvation(!)
> > > > > > > > >
> > > > > > > > > I am looking for an alternate and _full-proof_ solution for this
> > > > > > > > > situation. Any thoughts, suggestions would be useful and
> > appreciated.
> > > > > > > > >
> > > > > > > > > thanks!
> > > > > > > > > Chandra
> > > > > > > > >
> > > > > > > > > On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
> > > > > > > ralph.goers@dslextreme.com>,
> > > > > > > > > wrote:
> > > > > > > > > > Unless someone objects I plan to start the release process for
> > Log4j
> > > > > > > > > 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but
> > the
> > > > > > > > > problem only occurs in rare situations and I am not sure how to
> > fix it
> > > > > > > yet.
> > > > > > > > > There are a lot of other issues that deserve looking at but
> > nothing I
> > > > > > > can
> > > > > > > > > see that warrants waiting on.
> > > > > > > > > >
> > > > > > > > > > Ralph
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Matt Sicker <boards@gmail.com
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Matt Sicker <boards@gmail.com
> > > >
> > > >
> >
> >
> >
>
>
> --
> Matt Sicker <boards@gmail.com
Re: Compression on rolling leads to OOM
Posted by Ralph Goers <ra...@dslextreme.com>.
I should have used “spawn” instead of “fork”.
Ralph
> On Nov 16, 2017, at 11:34 AM, Matt Sicker <bo...@gmail.com> wrote:
>
> I don't think Java supports process forking (not even sure if Windows
> does), but you can execute and control processes at least.
>
> On 16 November 2017 at 12:28, Ralph Goers <ra...@dslextreme.com>
> wrote:
>
>> Sure, why not?
>>
>> Ralph
>>
>>> On Nov 16, 2017, at 11:24 AM, Chandra <chandra.tungathurthi@rwth-
>> aachen.de> wrote:
>>>
>>> with-in log4j?
>>>
>>> On 16 Nov 2017, 11:46 PM +0530, Ralph Goers <ra...@dslextreme.com>,
>> wrote:
>>>> I could see forking a process instead of spawning a thread to perform
>> the compression as a viable approach.
>>>>
>>>> Ralph
>>>>
>>>>> On Nov 16, 2017, at 10:01 AM, Chandra <chandra.tungathurthi@rwth-
>> aachen.de> wrote:
>>>>>
>>>>> thanks for info matt, was planning to use this on some other project
>> in mind ( might pick your brain on that later ;) )
>>>>>
>>>>> separating it out the compression to external process isn’t
>> necessarily a bad idea, but having non-reliable scripts is. As it so
>> happened many times before. I’d rather depend on my app than an external
>> process.
>>>>> which is why I was looking for an “agent” .
>>>>>
>>>>> thanks!
>>>>> chandra
>>>>>
>>>>>
>>>>> On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
>>>>>> I brought up Snappy only because I used their off-heap API recently.
>> Snappy
>>>>>> is more about real time compression rather than size (I think snappy
>> files
>>>>>> tend to be larger than gzip files, but take less resources to
>> compress and
>>>>>> decompress). The idea here is to offer support via libraries using
>> native
>>>>>> implementations that can work with direct byte buffers, mmap'd files,
>> or
>>>>>> even just a file name.
>>>>>>
>>>>>> With that in mind, is it so bad to offer the ability to execute an
>> external
>>>>>> process to compress the file?
>>>>>>
>>>>>> On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-
>> aachen.de
>>>>>> wrote:
>>>>>>
>>>>>>>> What if compression worked off-heap
>>>>>>> off-heap compression sounds interesting. Let me check if I can find
>> any.
>>>>>>>
>>>>>>> Snappy’s compression is a different altogether, I am not necessarily
>>>>>>> looking for a different compression formats, as I’d have add support
>> for it
>>>>>>> in downstream. Standard bz2 , gzip would work in-fact.
>>>>>>>
>>>>>>>
>>>>>>> Ideally a reliable agent something like `FileBeat` would be great
>> for this
>>>>>>> situation. :-/
>>>>>>>
>>>>>>> Best,
>>>>>>> Chandra
>>>>>>>
>>>>>>>
>>>>>>> On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>,
>> wrote:
>>>>>>>> What if compression worked off-heap like with some of the native
>>>>>>>> implementations of codecs? I'm thinking of this one <
>>>>>>>> https://github.com/xerial/snappy-java> for example.
>>>>>>>>
>>>>>>>> On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
>>>>>>> aachen.de
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Guys,
>>>>>>>>>
>>>>>>>>> I need some input on how this handle situation:
>>>>>>>>>
>>>>>>>>> we are on HP/LL setting where the incoming requests are processed
>> and
>>>>>>>>> logged ( buffered, of course with “async logging”). There are some
>>>>>>>>> situations where due to spikes in the volume of requests, the
>>>>>>> compression
>>>>>>>>> on rolling creates memory starvation.
>>>>>>>>>
>>>>>>>>> Now, a straight forward fix would to remove it from the “context”
>> of
>>>>>>> jvm
>>>>>>>>> and use a script to monitor and compress it timely. as low-hanging
>> the
>>>>>>>>> solution may be, I am skeptical of this solution as the script
>> _may_
>>>>>>> fail
>>>>>>>>> leading to disk starvation(!)
>>>>>>>>>
>>>>>>>>> I am looking for an alternate and _full-proof_ solution for this
>>>>>>>>> situation. Any thoughts, suggestions would be useful and
>> appreciated.
>>>>>>>>>
>>>>>>>>> thanks!
>>>>>>>>> Chandra
>>>>>>>>>
>>>>>>>>> On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
>>>>>>> ralph.goers@dslextreme.com>,
>>>>>>>>> wrote:
>>>>>>>>>> Unless someone objects I plan to start the release process for
>> Log4j
>>>>>>>>> 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but
>> the
>>>>>>>>> problem only occurs in rare situations and I am not sure how to
>> fix it
>>>>>>> yet.
>>>>>>>>> There are a lot of other issues that deserve looking at but
>> nothing I
>>>>>>> can
>>>>>>>>> see that warrants waiting on.
>>>>>>>>>>
>>>>>>>>>> Ralph
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Matt Sicker <boards@gmail.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Matt Sicker <boards@gmail.com
>>>>
>>>>
>>
>>
>>
>
>
> --
> Matt Sicker <bo...@gmail.com>
Re: Compression on rolling leads to OOM
Posted by Chandra <ch...@rwth-aachen.de>.
well that’s the idea. have log4j rotate files, keep it in a separate directory and a script or an agent take care of compression and shipping.
On 17 Nov 2017, 12:07 AM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> As for an agent, that'd be a separate daemon to watch your log files and
> compress them. Due to file handle issues with Java, I don't think said
> agent could handle file rotations and such without some convoluted staging
> area.
>
> On 16 November 2017 at 12:34, Matt Sicker <bo...@gmail.com> wrote:
>
> > I don't think Java supports process forking (not even sure if Windows
> > does), but you can execute and control processes at least.
> >
> > On 16 November 2017 at 12:28, Ralph Goers <ralph.goers@dslextreme.com
> > wrote:
> >
> > > Sure, why not?
> > >
> > > Ralph
> > >
> > > > On Nov 16, 2017, at 11:24 AM, Chandra <chandra.tungathurthi@rwth-aac
> > > hen.de> wrote:
> > > >
> > > > with-in log4j?
> > > >
> > > > On 16 Nov 2017, 11:46 PM +0530, Ralph Goers <ra...@dslextreme.com>,
> > > wrote:
> > > > > I could see forking a process instead of spawning a thread to perform
> > > the compression as a viable approach.
> > > > >
> > > > > Ralph
> > > > >
> > > > > > On Nov 16, 2017, at 10:01 AM, Chandra <chandra.tungathurthi@rwth-aac
> > > hen.de> wrote:
> > > > > >
> > > > > > thanks for info matt, was planning to use this on some other project
> > > in mind ( might pick your brain on that later ;) )
> > > > > >
> > > > > > separating it out the compression to external process isn’t
> > > necessarily a bad idea, but having non-reliable scripts is. As it so
> > > happened many times before. I’d rather depend on my app than an external
> > > process.
> > > > > > which is why I was looking for an “agent” .
> > > > > >
> > > > > > thanks!
> > > > > > chandra
> > > > > >
> > > > > >
> > > > > > On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> > > > > > > I brought up Snappy only because I used their off-heap API recently.
> > > Snappy
> > > > > > > is more about real time compression rather than size (I think snappy
> > > files
> > > > > > > tend to be larger than gzip files, but take less resources to
> > > compress and
> > > > > > > decompress). The idea here is to offer support via libraries using
> > > native
> > > > > > > implementations that can work with direct byte buffers, mmap'd
> > > files, or
> > > > > > > even just a file name.
> > > > > > >
> > > > > > > With that in mind, is it so bad to offer the ability to execute an
> > > external
> > > > > > > process to compress the file?
> > > > > > >
> > > > > > > On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-aac
> > > hen.de
> > > > > > > wrote:
> > > > > > >
> > > > > > > > > What if compression worked off-heap
> > > > > > > > off-heap compression sounds interesting. Let me check if I can find
> > > any.
> > > > > > > >
> > > > > > > > Snappy’s compression is a different altogether, I am not necessarily
> > > > > > > > looking for a different compression formats, as I’d have add
> > > support for it
> > > > > > > > in downstream. Standard bz2 , gzip would work in-fact.
> > > > > > > >
> > > > > > > >
> > > > > > > > Ideally a reliable agent something like `FileBeat` would be great
> > > for this
> > > > > > > > situation. :-/
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Chandra
> > > > > > > >
> > > > > > > >
> > > > > > > > On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>,
> > > wrote:
> > > > > > > > > What if compression worked off-heap like with some of the native
> > > > > > > > > implementations of codecs? I'm thinking of this one <
> > > > > > > > > https://github.com/xerial/snappy-java> for example.
> > > > > > > > >
> > > > > > > > > On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
> > > > > > > > aachen.de
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Guys,
> > > > > > > > > >
> > > > > > > > > > I need some input on how this handle situation:
> > > > > > > > > >
> > > > > > > > > > we are on HP/LL setting where the incoming requests are processed
> > > and
> > > > > > > > > > logged ( buffered, of course with “async logging”). There are some
> > > > > > > > > > situations where due to spikes in the volume of requests, the
> > > > > > > > compression
> > > > > > > > > > on rolling creates memory starvation.
> > > > > > > > > >
> > > > > > > > > > Now, a straight forward fix would to remove it from the “context”
> > > of
> > > > > > > > jvm
> > > > > > > > > > and use a script to monitor and compress it timely. as
> > > low-hanging the
> > > > > > > > > > solution may be, I am skeptical of this solution as the script
> > > _may_
> > > > > > > > fail
> > > > > > > > > > leading to disk starvation(!)
> > > > > > > > > >
> > > > > > > > > > I am looking for an alternate and _full-proof_ solution for this
> > > > > > > > > > situation. Any thoughts, suggestions would be useful and
> > > appreciated.
> > > > > > > > > >
> > > > > > > > > > thanks!
> > > > > > > > > > Chandra
> > > > > > > > > >
> > > > > > > > > > On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
> > > > > > > > ralph.goers@dslextreme.com>,
> > > > > > > > > > wrote:
> > > > > > > > > > > Unless someone objects I plan to start the release process for
> > > Log4j
> > > > > > > > > > 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but
> > > the
> > > > > > > > > > problem only occurs in rare situations and I am not sure how to
> > > fix it
> > > > > > > > yet.
> > > > > > > > > > There are a lot of other issues that deserve looking at but
> > > nothing I
> > > > > > > > can
> > > > > > > > > > see that warrants waiting on.
> > > > > > > > > > >
> > > > > > > > > > > Ralph
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Matt Sicker <boards@gmail.com
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Matt Sicker <boards@gmail.com
> > > > >
> > > > >
> > >
> > >
> > >
> >
> >
> > --
> > Matt Sicker <boards@gmail.com
> >
>
>
>
> --
> Matt Sicker <boards@gmail.com
Re: Compression on rolling leads to OOM
Posted by Matt Sicker <bo...@gmail.com>.
As for an agent, that'd be a separate daemon to watch your log files and
compress them. Due to file handle issues with Java, I don't think said
agent could handle file rotations and such without some convoluted staging
area.
On 16 November 2017 at 12:34, Matt Sicker <bo...@gmail.com> wrote:
> I don't think Java supports process forking (not even sure if Windows
> does), but you can execute and control processes at least.
>
> On 16 November 2017 at 12:28, Ralph Goers <ra...@dslextreme.com>
> wrote:
>
>> Sure, why not?
>>
>> Ralph
>>
>> > On Nov 16, 2017, at 11:24 AM, Chandra <chandra.tungathurthi@rwth-aac
>> hen.de> wrote:
>> >
>> > with-in log4j?
>> >
>> > On 16 Nov 2017, 11:46 PM +0530, Ralph Goers <ra...@dslextreme.com>,
>> wrote:
>> >> I could see forking a process instead of spawning a thread to perform
>> the compression as a viable approach.
>> >>
>> >> Ralph
>> >>
>> >>> On Nov 16, 2017, at 10:01 AM, Chandra <chandra.tungathurthi@rwth-aac
>> hen.de> wrote:
>> >>>
>> >>> thanks for info matt, was planning to use this on some other project
>> in mind ( might pick your brain on that later ;) )
>> >>>
>> >>> separating it out the compression to external process isn’t
>> necessarily a bad idea, but having non-reliable scripts is. As it so
>> happened many times before. I’d rather depend on my app than an external
>> process.
>> >>> which is why I was looking for an “agent” .
>> >>>
>> >>> thanks!
>> >>> chandra
>> >>>
>> >>>
>> >>> On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
>> >>>> I brought up Snappy only because I used their off-heap API recently.
>> Snappy
>> >>>> is more about real time compression rather than size (I think snappy
>> files
>> >>>> tend to be larger than gzip files, but take less resources to
>> compress and
>> >>>> decompress). The idea here is to offer support via libraries using
>> native
>> >>>> implementations that can work with direct byte buffers, mmap'd
>> files, or
>> >>>> even just a file name.
>> >>>>
>> >>>> With that in mind, is it so bad to offer the ability to execute an
>> external
>> >>>> process to compress the file?
>> >>>>
>> >>>> On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-aac
>> hen.de
>> >>>> wrote:
>> >>>>
>> >>>>>> What if compression worked off-heap
>> >>>>> off-heap compression sounds interesting. Let me check if I can find
>> any.
>> >>>>>
>> >>>>> Snappy’s compression is a different altogether, I am not necessarily
>> >>>>> looking for a different compression formats, as I’d have add
>> support for it
>> >>>>> in downstream. Standard bz2 , gzip would work in-fact.
>> >>>>>
>> >>>>>
>> >>>>> Ideally a reliable agent something like `FileBeat` would be great
>> for this
>> >>>>> situation. :-/
>> >>>>>
>> >>>>> Best,
>> >>>>> Chandra
>> >>>>>
>> >>>>>
>> >>>>> On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>,
>> wrote:
>> >>>>>> What if compression worked off-heap like with some of the native
>> >>>>>> implementations of codecs? I'm thinking of this one <
>> >>>>>> https://github.com/xerial/snappy-java> for example.
>> >>>>>>
>> >>>>>> On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
>> >>>>> aachen.de
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Guys,
>> >>>>>>>
>> >>>>>>> I need some input on how this handle situation:
>> >>>>>>>
>> >>>>>>> we are on HP/LL setting where the incoming requests are processed
>> and
>> >>>>>>> logged ( buffered, of course with “async logging”). There are some
>> >>>>>>> situations where due to spikes in the volume of requests, the
>> >>>>> compression
>> >>>>>>> on rolling creates memory starvation.
>> >>>>>>>
>> >>>>>>> Now, a straight forward fix would to remove it from the “context”
>> of
>> >>>>> jvm
>> >>>>>>> and use a script to monitor and compress it timely. as
>> low-hanging the
>> >>>>>>> solution may be, I am skeptical of this solution as the script
>> _may_
>> >>>>> fail
>> >>>>>>> leading to disk starvation(!)
>> >>>>>>>
>> >>>>>>> I am looking for an alternate and _full-proof_ solution for this
>> >>>>>>> situation. Any thoughts, suggestions would be useful and
>> appreciated.
>> >>>>>>>
>> >>>>>>> thanks!
>> >>>>>>> Chandra
>> >>>>>>>
>> >>>>>>> On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
>> >>>>> ralph.goers@dslextreme.com>,
>> >>>>>>> wrote:
>> >>>>>>>> Unless someone objects I plan to start the release process for
>> Log4j
>> >>>>>>> 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but
>> the
>> >>>>>>> problem only occurs in rare situations and I am not sure how to
>> fix it
>> >>>>> yet.
>> >>>>>>> There are a lot of other issues that deserve looking at but
>> nothing I
>> >>>>> can
>> >>>>>>> see that warrants waiting on.
>> >>>>>>>>
>> >>>>>>>> Ralph
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Matt Sicker <boards@gmail.com
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Matt Sicker <boards@gmail.com
>> >>
>> >>
>>
>>
>>
>
>
> --
> Matt Sicker <bo...@gmail.com>
>
--
Matt Sicker <bo...@gmail.com>
Re: Compression on rolling leads to OOM
Posted by Matt Sicker <bo...@gmail.com>.
I don't think Java supports process forking (not even sure if Windows
does), but you can execute and control processes at least.
On 16 November 2017 at 12:28, Ralph Goers <ra...@dslextreme.com>
wrote:
> Sure, why not?
>
> Ralph
>
> > On Nov 16, 2017, at 11:24 AM, Chandra <chandra.tungathurthi@rwth-
> aachen.de> wrote:
> >
> > with-in log4j?
> >
> > On 16 Nov 2017, 11:46 PM +0530, Ralph Goers <ra...@dslextreme.com>,
> wrote:
> >> I could see forking a process instead of spawning a thread to perform
> the compression as a viable approach.
> >>
> >> Ralph
> >>
> >>> On Nov 16, 2017, at 10:01 AM, Chandra <chandra.tungathurthi@rwth-
> aachen.de> wrote:
> >>>
> >>> thanks for info matt, was planning to use this on some other project
> in mind ( might pick your brain on that later ;) )
> >>>
> >>> separating it out the compression to external process isn’t
> necessarily a bad idea, but having non-reliable scripts is. As it so
> happened many times before. I’d rather depend on my app than an external
> process.
> >>> which is why I was looking for an “agent” .
> >>>
> >>> thanks!
> >>> chandra
> >>>
> >>>
> >>> On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> >>>> I brought up Snappy only because I used their off-heap API recently.
> Snappy
> >>>> is more about real time compression rather than size (I think snappy
> files
> >>>> tend to be larger than gzip files, but take less resources to
> compress and
> >>>> decompress). The idea here is to offer support via libraries using
> native
> >>>> implementations that can work with direct byte buffers, mmap'd files,
> or
> >>>> even just a file name.
> >>>>
> >>>> With that in mind, is it so bad to offer the ability to execute an
> external
> >>>> process to compress the file?
> >>>>
> >>>> On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-
> aachen.de
> >>>> wrote:
> >>>>
> >>>>>> What if compression worked off-heap
> >>>>> off-heap compression sounds interesting. Let me check if I can find
> any.
> >>>>>
> >>>>> Snappy’s compression is a different altogether, I am not necessarily
> >>>>> looking for a different compression formats, as I’d have add support
> for it
> >>>>> in downstream. Standard bz2 , gzip would work in-fact.
> >>>>>
> >>>>>
> >>>>> Ideally a reliable agent something like `FileBeat` would be great
> for this
> >>>>> situation. :-/
> >>>>>
> >>>>> Best,
> >>>>> Chandra
> >>>>>
> >>>>>
> >>>>> On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>,
> wrote:
> >>>>>> What if compression worked off-heap like with some of the native
> >>>>>> implementations of codecs? I'm thinking of this one <
> >>>>>> https://github.com/xerial/snappy-java> for example.
> >>>>>>
> >>>>>> On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
> >>>>> aachen.de
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Guys,
> >>>>>>>
> >>>>>>> I need some input on how this handle situation:
> >>>>>>>
> >>>>>>> we are on HP/LL setting where the incoming requests are processed
> and
> >>>>>>> logged ( buffered, of course with “async logging”). There are some
> >>>>>>> situations where due to spikes in the volume of requests, the
> >>>>> compression
> >>>>>>> on rolling creates memory starvation.
> >>>>>>>
> >>>>>>> Now, a straight forward fix would to remove it from the “context”
> of
> >>>>> jvm
> >>>>>>> and use a script to monitor and compress it timely. as low-hanging
> the
> >>>>>>> solution may be, I am skeptical of this solution as the script
> _may_
> >>>>> fail
> >>>>>>> leading to disk starvation(!)
> >>>>>>>
> >>>>>>> I am looking for an alternate and _full-proof_ solution for this
> >>>>>>> situation. Any thoughts, suggestions would be useful and
> appreciated.
> >>>>>>>
> >>>>>>> thanks!
> >>>>>>> Chandra
> >>>>>>>
> >>>>>>> On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
> >>>>> ralph.goers@dslextreme.com>,
> >>>>>>> wrote:
> >>>>>>>> Unless someone objects I plan to start the release process for
> Log4j
> >>>>>>> 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but
> the
> >>>>>>> problem only occurs in rare situations and I am not sure how to
> fix it
> >>>>> yet.
> >>>>>>> There are a lot of other issues that deserve looking at but
> nothing I
> >>>>> can
> >>>>>>> see that warrants waiting on.
> >>>>>>>>
> >>>>>>>> Ralph
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Matt Sicker <boards@gmail.com
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Matt Sicker <boards@gmail.com
> >>
> >>
>
>
>
--
Matt Sicker <bo...@gmail.com>
Re: Compression on rolling leads to OOM
Posted by Ralph Goers <ra...@dslextreme.com>.
Sure, why not?
Ralph
> On Nov 16, 2017, at 11:24 AM, Chandra <ch...@rwth-aachen.de> wrote:
>
> with-in log4j?
>
> On 16 Nov 2017, 11:46 PM +0530, Ralph Goers <ra...@dslextreme.com>, wrote:
>> I could see forking a process instead of spawning a thread to perform the compression as a viable approach.
>>
>> Ralph
>>
>>> On Nov 16, 2017, at 10:01 AM, Chandra <ch...@rwth-aachen.de> wrote:
>>>
>>> thanks for info matt, was planning to use this on some other project in mind ( might pick your brain on that later ;) )
>>>
>>> separating it out the compression to external process isn’t necessarily a bad idea, but having non-reliable scripts is. As it so happened many times before. I’d rather depend on my app than an external process.
>>> which is why I was looking for an “agent” .
>>>
>>> thanks!
>>> chandra
>>>
>>>
>>> On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
>>>> I brought up Snappy only because I used their off-heap API recently. Snappy
>>>> is more about real time compression rather than size (I think snappy files
>>>> tend to be larger than gzip files, but take less resources to compress and
>>>> decompress). The idea here is to offer support via libraries using native
>>>> implementations that can work with direct byte buffers, mmap'd files, or
>>>> even just a file name.
>>>>
>>>> With that in mind, is it so bad to offer the ability to execute an external
>>>> process to compress the file?
>>>>
>>>> On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-aachen.de
>>>> wrote:
>>>>
>>>>>> What if compression worked off-heap
>>>>> off-heap compression sounds interesting. Let me check if I can find any.
>>>>>
>>>>> Snappy’s compression is a different altogether, I am not necessarily
>>>>> looking for a different compression formats, as I’d have add support for it
>>>>> in downstream. Standard bz2 , gzip would work in-fact.
>>>>>
>>>>>
>>>>> Ideally a reliable agent something like `FileBeat` would be great for this
>>>>> situation. :-/
>>>>>
>>>>> Best,
>>>>> Chandra
>>>>>
>>>>>
>>>>> On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
>>>>>> What if compression worked off-heap like with some of the native
>>>>>> implementations of codecs? I'm thinking of this one <
>>>>>> https://github.com/xerial/snappy-java> for example.
>>>>>>
>>>>>> On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
>>>>> aachen.de
>>>>>> wrote:
>>>>>>
>>>>>>> Guys,
>>>>>>>
>>>>>>> I need some input on how this handle situation:
>>>>>>>
>>>>>>> we are on HP/LL setting where the incoming requests are processed and
>>>>>>> logged ( buffered, of course with “async logging”). There are some
>>>>>>> situations where due to spikes in the volume of requests, the
>>>>> compression
>>>>>>> on rolling creates memory starvation.
>>>>>>>
>>>>>>> Now, a straight forward fix would to remove it from the “context” of
>>>>> jvm
>>>>>>> and use a script to monitor and compress it timely. as low-hanging the
>>>>>>> solution may be, I am skeptical of this solution as the script _may_
>>>>> fail
>>>>>>> leading to disk starvation(!)
>>>>>>>
>>>>>>> I am looking for an alternate and _full-proof_ solution for this
>>>>>>> situation. Any thoughts, suggestions would be useful and appreciated.
>>>>>>>
>>>>>>> thanks!
>>>>>>> Chandra
>>>>>>>
>>>>>>> On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
>>>>> ralph.goers@dslextreme.com>,
>>>>>>> wrote:
>>>>>>>> Unless someone objects I plan to start the release process for Log4j
>>>>>>> 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but the
>>>>>>> problem only occurs in rare situations and I am not sure how to fix it
>>>>> yet.
>>>>>>> There are a lot of other issues that deserve looking at but nothing I
>>>>> can
>>>>>>> see that warrants waiting on.
>>>>>>>>
>>>>>>>> Ralph
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Matt Sicker <boards@gmail.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Matt Sicker <boards@gmail.com
>>
>>
Re: Compression on rolling leads to OOM
Posted by Chandra <ch...@rwth-aachen.de>.
with-in log4j?
On 16 Nov 2017, 11:46 PM +0530, Ralph Goers <ra...@dslextreme.com>, wrote:
> I could see forking a process instead of spawning a thread to perform the compression as a viable approach.
>
> Ralph
>
> > On Nov 16, 2017, at 10:01 AM, Chandra <ch...@rwth-aachen.de> wrote:
> >
> > thanks for info matt, was planning to use this on some other project in mind ( might pick your brain on that later ;) )
> >
> > separating it out the compression to external process isn’t necessarily a bad idea, but having non-reliable scripts is. As it so happened many times before. I’d rather depend on my app than an external process.
> > which is why I was looking for an “agent” .
> >
> > thanks!
> > chandra
> >
> >
> > On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> > > I brought up Snappy only because I used their off-heap API recently. Snappy
> > > is more about real time compression rather than size (I think snappy files
> > > tend to be larger than gzip files, but take less resources to compress and
> > > decompress). The idea here is to offer support via libraries using native
> > > implementations that can work with direct byte buffers, mmap'd files, or
> > > even just a file name.
> > >
> > > With that in mind, is it so bad to offer the ability to execute an external
> > > process to compress the file?
> > >
> > > On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-aachen.de
> > > wrote:
> > >
> > > > > What if compression worked off-heap
> > > > off-heap compression sounds interesting. Let me check if I can find any.
> > > >
> > > > Snappy’s compression is a different altogether, I am not necessarily
> > > > looking for a different compression formats, as I’d have add support for it
> > > > in downstream. Standard bz2 , gzip would work in-fact.
> > > >
> > > >
> > > > Ideally a reliable agent something like `FileBeat` would be great for this
> > > > situation. :-/
> > > >
> > > > Best,
> > > > Chandra
> > > >
> > > >
> > > > On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> > > > > What if compression worked off-heap like with some of the native
> > > > > implementations of codecs? I'm thinking of this one <
> > > > > https://github.com/xerial/snappy-java> for example.
> > > > >
> > > > > On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
> > > > aachen.de
> > > > > wrote:
> > > > >
> > > > > > Guys,
> > > > > >
> > > > > > I need some input on how this handle situation:
> > > > > >
> > > > > > we are on HP/LL setting where the incoming requests are processed and
> > > > > > logged ( buffered, of course with “async logging”). There are some
> > > > > > situations where due to spikes in the volume of requests, the
> > > > compression
> > > > > > on rolling creates memory starvation.
> > > > > >
> > > > > > Now, a straight forward fix would to remove it from the “context” of
> > > > jvm
> > > > > > and use a script to monitor and compress it timely. as low-hanging the
> > > > > > solution may be, I am skeptical of this solution as the script _may_
> > > > fail
> > > > > > leading to disk starvation(!)
> > > > > >
> > > > > > I am looking for an alternate and _full-proof_ solution for this
> > > > > > situation. Any thoughts, suggestions would be useful and appreciated.
> > > > > >
> > > > > > thanks!
> > > > > > Chandra
> > > > > >
> > > > > > On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
> > > > ralph.goers@dslextreme.com>,
> > > > > > wrote:
> > > > > > > Unless someone objects I plan to start the release process for Log4j
> > > > > > 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but the
> > > > > > problem only occurs in rare situations and I am not sure how to fix it
> > > > yet.
> > > > > > There are a lot of other issues that deserve looking at but nothing I
> > > > can
> > > > > > see that warrants waiting on.
> > > > > > >
> > > > > > > Ralph
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Matt Sicker <boards@gmail.com
> > > >
> > >
> > >
> > >
> > > --
> > > Matt Sicker <boards@gmail.com
>
>
Re: Compression on rolling leads to OOM
Posted by Ralph Goers <ra...@dslextreme.com>.
I could see forking a process instead of spawning a thread to perform the compression as a viable approach.
Ralph
> On Nov 16, 2017, at 10:01 AM, Chandra <ch...@rwth-aachen.de> wrote:
>
> thanks for info matt, was planning to use this on some other project in mind ( might pick your brain on that later ;) )
>
> separating it out the compression to external process isn’t necessarily a bad idea, but having non-reliable scripts is. As it so happened many times before. I’d rather depend on my app than an external process.
> which is why I was looking for an “agent” .
>
> thanks!
> chandra
>
>
> On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
>> I brought up Snappy only because I used their off-heap API recently. Snappy
>> is more about real time compression rather than size (I think snappy files
>> tend to be larger than gzip files, but take less resources to compress and
>> decompress). The idea here is to offer support via libraries using native
>> implementations that can work with direct byte buffers, mmap'd files, or
>> even just a file name.
>>
>> With that in mind, is it so bad to offer the ability to execute an external
>> process to compress the file?
>>
>> On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-aachen.de
>> wrote:
>>
>>>> What if compression worked off-heap
>>> off-heap compression sounds interesting. Let me check if I can find any.
>>>
>>> Snappy’s compression is a different altogether, I am not necessarily
>>> looking for a different compression formats, as I’d have add support for it
>>> in downstream. Standard bz2 , gzip would work in-fact.
>>>
>>>
>>> Ideally a reliable agent something like `FileBeat` would be great for this
>>> situation. :-/
>>>
>>> Best,
>>> Chandra
>>>
>>>
>>> On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
>>>> What if compression worked off-heap like with some of the native
>>>> implementations of codecs? I'm thinking of this one <
>>>> https://github.com/xerial/snappy-java> for example.
>>>>
>>>> On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
>>> aachen.de
>>>> wrote:
>>>>
>>>>> Guys,
>>>>>
>>>>> I need some input on how this handle situation:
>>>>>
>>>>> we are on HP/LL setting where the incoming requests are processed and
>>>>> logged ( buffered, of course with “async logging”). There are some
>>>>> situations where due to spikes in the volume of requests, the
>>> compression
>>>>> on rolling creates memory starvation.
>>>>>
>>>>> Now, a straight forward fix would to remove it from the “context” of
>>> jvm
>>>>> and use a script to monitor and compress it timely. as low-hanging the
>>>>> solution may be, I am skeptical of this solution as the script _may_
>>> fail
>>>>> leading to disk starvation(!)
>>>>>
>>>>> I am looking for an alternate and _full-proof_ solution for this
>>>>> situation. Any thoughts, suggestions would be useful and appreciated.
>>>>>
>>>>> thanks!
>>>>> Chandra
>>>>>
>>>>> On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
>>> ralph.goers@dslextreme.com>,
>>>>> wrote:
>>>>>> Unless someone objects I plan to start the release process for Log4j
>>>>> 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but the
>>>>> problem only occurs in rare situations and I am not sure how to fix it
>>> yet.
>>>>> There are a lot of other issues that deserve looking at but nothing I
>>> can
>>>>> see that warrants waiting on.
>>>>>>
>>>>>> Ralph
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Matt Sicker <boards@gmail.com
>>>
>>
>>
>>
>> --
>> Matt Sicker <boards@gmail.com
Re: Compression on rolling leads to OOM
Posted by Chandra <ch...@rwth-aachen.de>.
thanks for info matt, was planning to use this on some other project in mind ( might pick your brain on that later ;) )
separating it out the compression to external process isn’t necessarily a bad idea, but having non-reliable scripts is. As it so happened many times before. I’d rather depend on my app than an external process.
which is why I was looking for an “agent” .
thanks!
chandra
On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> I brought up Snappy only because I used their off-heap API recently. Snappy
> is more about real time compression rather than size (I think snappy files
> tend to be larger than gzip files, but take less resources to compress and
> decompress). The idea here is to offer support via libraries using native
> implementations that can work with direct byte buffers, mmap'd files, or
> even just a file name.
>
> With that in mind, is it so bad to offer the ability to execute an external
> process to compress the file?
>
> On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-aachen.de
> wrote:
>
> > > What if compression worked off-heap
> > off-heap compression sounds interesting. Let me check if I can find any.
> >
> > Snappy’s compression is a different altogether, I am not necessarily
> > looking for a different compression formats, as I’d have add support for it
> > in downstream. Standard bz2 , gzip would work in-fact.
> >
> >
> > Ideally a reliable agent something like `FileBeat` would be great for this
> > situation. :-/
> >
> > Best,
> > Chandra
> >
> >
> > On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> > > What if compression worked off-heap like with some of the native
> > > implementations of codecs? I'm thinking of this one <
> > > https://github.com/xerial/snappy-java> for example.
> > >
> > > On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
> > aachen.de
> > > wrote:
> > >
> > > > Guys,
> > > >
> > > > I need some input on how this handle situation:
> > > >
> > > > we are on HP/LL setting where the incoming requests are processed and
> > > > logged ( buffered, of course with “async logging”). There are some
> > > > situations where due to spikes in the volume of requests, the
> > compression
> > > > on rolling creates memory starvation.
> > > >
> > > > Now, a straight forward fix would to remove it from the “context” of
> > jvm
> > > > and use a script to monitor and compress it timely. as low-hanging the
> > > > solution may be, I am skeptical of this solution as the script _may_
> > fail
> > > > leading to disk starvation(!)
> > > >
> > > > I am looking for an alternate and _full-proof_ solution for this
> > > > situation. Any thoughts, suggestions would be useful and appreciated.
> > > >
> > > > thanks!
> > > > Chandra
> > > >
> > > > On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
> > ralph.goers@dslextreme.com>,
> > > > wrote:
> > > > > Unless someone objects I plan to start the release process for Log4j
> > > > 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but the
> > > > problem only occurs in rare situations and I am not sure how to fix it
> > yet.
> > > > There are a lot of other issues that deserve looking at but nothing I
> > can
> > > > see that warrants waiting on.
> > > > >
> > > > > Ralph
> > > >
> > >
> > >
> > >
> > > --
> > > Matt Sicker <boards@gmail.com
> >
>
>
>
> --
> Matt Sicker <boards@gmail.com
Re: Compression on rolling leads to OOM
Posted by Matt Sicker <bo...@gmail.com>.
I brought up Snappy only because I used their off-heap API recently. Snappy
is more about real time compression rather than size (I think snappy files
tend to be larger than gzip files, but take less resources to compress and
decompress). The idea here is to offer support via libraries using native
implementations that can work with direct byte buffers, mmap'd files, or
even just a file name.
With that in mind, is it so bad to offer the ability to execute an external
process to compress the file?
On 16 November 2017 at 09:59, Chandra <ch...@rwth-aachen.de>
wrote:
> > What if compression worked off-heap
> off-heap compression sounds interesting. Let me check if I can find any.
>
> Snappy’s compression is a different altogether, I am not necessarily
> looking for a different compression formats, as I’d have add support for it
> in downstream. Standard bz2 , gzip would work in-fact.
>
>
> Ideally a reliable agent something like `FileBeat` would be great for this
> situation. :-/
>
> Best,
> Chandra
>
>
> On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> > What if compression worked off-heap like with some of the native
> > implementations of codecs? I'm thinking of this one <
> > https://github.com/xerial/snappy-java> for example.
> >
> > On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-
> aachen.de
> > wrote:
> >
> > > Guys,
> > >
> > > I need some input on how this handle situation:
> > >
> > > we are on HP/LL setting where the incoming requests are processed and
> > > logged ( buffered, of course with “async logging”). There are some
> > > situations where due to spikes in the volume of requests, the
> compression
> > > on rolling creates memory starvation.
> > >
> > > Now, a straight forward fix would to remove it from the “context” of
> jvm
> > > and use a script to monitor and compress it timely. as low-hanging the
> > > solution may be, I am skeptical of this solution as the script _may_
> fail
> > > leading to disk starvation(!)
> > >
> > > I am looking for an alternate and _full-proof_ solution for this
> > > situation. Any thoughts, suggestions would be useful and appreciated.
> > >
> > > thanks!
> > > Chandra
> > >
> > > On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <
> ralph.goers@dslextreme.com>,
> > > wrote:
> > > > Unless someone objects I plan to start the release process for Log4j
> > > 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but the
> > > problem only occurs in rare situations and I am not sure how to fix it
> yet.
> > > There are a lot of other issues that deserve looking at but nothing I
> can
> > > see that warrants waiting on.
> > > >
> > > > Ralph
> > >
> >
> >
> >
> > --
> > Matt Sicker <boards@gmail.com
>
--
Matt Sicker <bo...@gmail.com>
Re: Compression on rolling leads to OOM
Posted by Chandra <ch...@rwth-aachen.de>.
> What if compression worked off-heap
off-heap compression sounds interesting. Let me check if I can find any.
Snappy’s compression is a different altogether, I am not necessarily looking for a different compression formats, as I’d have add support for it in downstream. Standard bz2 , gzip would work in-fact.
Ideally a reliable agent something like `FileBeat` would be great for this situation. :-/
Best,
Chandra
On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <bo...@gmail.com>, wrote:
> What if compression worked off-heap like with some of the native
> implementations of codecs? I'm thinking of this one <
> https://github.com/xerial/snappy-java> for example.
>
> On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth-aachen.de
> wrote:
>
> > Guys,
> >
> > I need some input on how this handle situation:
> >
> > we are on HP/LL setting where the incoming requests are processed and
> > logged ( buffered, of course with “async logging”). There are some
> > situations where due to spikes in the volume of requests, the compression
> > on rolling creates memory starvation.
> >
> > Now, a straight forward fix would to remove it from the “context” of jvm
> > and use a script to monitor and compress it timely. as low-hanging the
> > solution may be, I am skeptical of this solution as the script _may_ fail
> > leading to disk starvation(!)
> >
> > I am looking for an alternate and _full-proof_ solution for this
> > situation. Any thoughts, suggestions would be useful and appreciated.
> >
> > thanks!
> > Chandra
> >
> > On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <ra...@dslextreme.com>,
> > wrote:
> > > Unless someone objects I plan to start the release process for Log4j
> > 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but the
> > problem only occurs in rare situations and I am not sure how to fix it yet.
> > There are a lot of other issues that deserve looking at but nothing I can
> > see that warrants waiting on.
> > >
> > > Ralph
> >
>
>
>
> --
> Matt Sicker <boards@gmail.com
Re: Compression on rolling leads to OOM
Posted by Matt Sicker <bo...@gmail.com>.
What if compression worked off-heap like with some of the native
implementations of codecs? I'm thinking of this one <
https://github.com/xerial/snappy-java> for example.
On 15 November 2017 at 23:41, Chandra <ch...@rwth-aachen.de>
wrote:
> Guys,
>
> I need some input on how this handle situation:
>
> we are on HP/LL setting where the incoming requests are processed and
> logged ( buffered, of course with “async logging”). There are some
> situations where due to spikes in the volume of requests, the compression
> on rolling creates memory starvation.
>
> Now, a straight forward fix would to remove it from the “context” of jvm
> and use a script to monitor and compress it timely. as low-hanging the
> solution may be, I am skeptical of this solution as the script _may_ fail
> leading to disk starvation(!)
>
> I am looking for an alternate and _full-proof_ solution for this
> situation. Any thoughts, suggestions would be useful and appreciated.
>
> thanks!
> Chandra
>
> On 16 Nov 2017, 10:01 AM +0530, Ralph Goers <ra...@dslextreme.com>,
> wrote:
> > Unless someone objects I plan to start the release process for Log4j
> 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but the
> problem only occurs in rare situations and I am not sure how to fix it yet.
> There are a lot of other issues that deserve looking at but nothing I can
> see that warrants waiting on.
> >
> > Ralph
>
--
Matt Sicker <bo...@gmail.com>