You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Nick Gearls <ni...@gmail.com> on 2011/01/07 14:26:35 UTC
Huge memory leak in mod_substitute
Hello,
I saw servers completely freezing (the whole system, not only httpd) due
to a memory leak problem in mod_substitute.
This is a major issue as it can completely block your systems, sometimes
even the fail-over as the server is blocked, but not completely down.
Even the management interfaces are blocked.
We reported the problem
(https://issues.apache.org/bugzilla/show_bug.cgi?id=50559), with a
(scaled-down) reproducible process to trigger it.
Did somebody already begin to work on mod_substitute memory management?
Regards,
Nick
Re: Huge memory leak in mod_substitute
Posted by Jim Jagielski <ji...@jaguNET.com>.
If what you have substituted is a pattern for a subsequent
match, then you need to worry about flattening.
For example, if you s/cat/dog/ and then later on have
a s/doghouse/shed/ then you need to worry if you had
some places where you went from cathouse -> doghouse due
to the 1st replacement because that 2nd substitution won't
happen for *those* instances.
eg:
I put my cat in the cathouse
-> I put my dog in the doghouse [1st sub]
With flattening you will get:
-> I put my dog in the shed [2nd sub]
But without it you would get:
-> I put my dog in the doghouse
because the 'doghouse' acts like 'dog''house' where
there is this imaginary "break" between dog
and house (since there are in different buckets inside
Apache).
On Jan 7, 2011, at 11:31 AM, Nick Gearls wrote:
> When setting the q flag, I have no more memory leaks.
> And it substitutes all instances on the line; however, it seems we cannot assume it will always do so :-(
>
> Btw, I never understood when multiple subst will work with the q flag; does somebody have an answer?
> For instance,
> Substitute s/aa/b/q
> works on "aaaaaaaaaaaaaa", even on a 10 KB long with "a".
>
> Thanks,
>
> Nick
>
> On 7/1/2011 15:49, Nick Kew wrote:
>> On Fri, 07 Jan 2011 14:26:35 +0100
>> Nick Gearls<ni...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I saw servers completely freezing (the whole system, not only httpd) due
>>> to a memory leak problem in mod_substitute.
>>> This is a major issue as it can completely block your systems, sometimes
>>> even the fail-over as the server is blocked, but not completely down.
>>> Even the management interfaces are blocked.
>>> We reported the problem
>>> (https://issues.apache.org/bugzilla/show_bug.cgi?id=50559), with a
>>> (scaled-down) reproducible process to trigger it.
>>>
>>> Did somebody already begin to work on mod_substitute memory management?
>>
>> The obvious question would be whether the "q" flag affects your
>> observations. But the detailed bug report suggests that's unlikely.
>>
>> So the second obvious question: have you tried equivalent observations
>> with mod_sed and/or mod_line_edit?
>>
>
Re: Huge memory leak in mod_substitute
Posted by Nick Gearls <ni...@gmail.com>.
When setting the q flag, I have no more memory leaks.
And it substitutes all instances on the line; however, it seems we
cannot assume it will always do so :-(
Btw, I never understood when multiple subst will work with the q flag;
does somebody have an answer?
For instance,
Substitute s/aa/b/q
works on "aaaaaaaaaaaaaa", even on a 10 KB long with "a".
Thanks,
Nick
On 7/1/2011 15:49, Nick Kew wrote:
> On Fri, 07 Jan 2011 14:26:35 +0100
> Nick Gearls<ni...@gmail.com> wrote:
>
>> Hello,
>>
>> I saw servers completely freezing (the whole system, not only httpd) due
>> to a memory leak problem in mod_substitute.
>> This is a major issue as it can completely block your systems, sometimes
>> even the fail-over as the server is blocked, but not completely down.
>> Even the management interfaces are blocked.
>> We reported the problem
>> (https://issues.apache.org/bugzilla/show_bug.cgi?id=50559), with a
>> (scaled-down) reproducible process to trigger it.
>>
>> Did somebody already begin to work on mod_substitute memory management?
>
> The obvious question would be whether the "q" flag affects your
> observations. But the detailed bug report suggests that's unlikely.
>
> So the second obvious question: have you tried equivalent observations
> with mod_sed and/or mod_line_edit?
>
Re: Huge memory leak in mod_substitute
Posted by Eric Covener <co...@gmail.com>.
On Fri, Jan 7, 2011 at 11:05 AM, Nick Gearls <ni...@gmail.com> wrote:
>> we need to flatten and that takes time and space... lots o' space
> OK, but why more space at each request?
> The memory should be released at the end of the request.
http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxmemfree ?
Re: Huge memory leak in mod_substitute
Posted by Nick Gearls <ni...@gmail.com>.
> we need to flatten and that takes time and space... lots o' space
OK, but why more space at each request?
The memory should be released at the end of the request.
Nick
On 7/1/2011 16:56, Jim Jagielski wrote:
>
> On Jan 7, 2011, at 10:46 AM, Nick Kew wrote:
>
>> On Fri, 7 Jan 2011 15:59:26 +0100
>> "Plüm, Rüdiger, VF-Group"<ru...@vodafone.com> wrote:
>>
>>>> The obvious question would be whether the "q" flag affects your
>>>> observations. But the detailed bug report suggests that's unlikely.
>>>
>>> Isn't it the case that "q" is the default for one substitution rule
>>> and for more than one it is no longer?
>>> That would explain the different behaviour.
>>
>> Bah. I'd misread Test 2 in the report - saw just the one rule.
>>
>> I guess it just boils down to the logic of when folding does or
>> doesn't get optimised out.
>>
>
> I think it's just the nature of the beast... at least,
> the nature of the design where if we need to be able to
> scan between buckets, we need to flatten and that takes
> time and space... lots o' space.
>
>
Re: Huge memory leak in mod_substitute
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Jan 7, 2011, at 3:10 PM, William A. Rowe Jr. wrote:
> On 1/7/2011 1:23 PM, Jim Jagielski wrote:
>>
>> /a.*b/
>>
>> How big will that be?
>
> Oh it's certainly possible to create some very impractical patterns...
> /<html>.*</html>/ comes to mind. The pattern space in these cases is
> infinite, or rather, the size of the remaining document once an /a/ is
> sited in your example above.
>
Yeah, that's the rub. For regular patterns, then we can (and should)
be smarter, since we *know* the max size we are looking for...
for regexes, it's hard to get "much" better ;)
Re: Huge memory leak in mod_substitute
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 1/7/2011 1:23 PM, Jim Jagielski wrote:
>
> /a.*b/
>
> How big will that be?
Oh it's certainly possible to create some very impractical patterns...
/<html>.*</html>/ comes to mind. The pattern space in these cases is
infinite, or rather, the size of the remaining document once an /a/ is
sited in your example above.
Re: Huge memory leak in mod_substitute
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Jan 7, 2011, at 2:04 PM, William A. Rowe Jr. wrote:
> On 1/7/2011 9:58 AM, "Plüm, Rüdiger, VF-Group" wrote:
>>
>>> -----Original Message-----
>>> From: Jim Jagielski
>>> Sent: Freitag, 7. Januar 2011 16:57
>>>
>>> I think it's just the nature of the beast... at least,
>>> the nature of the design where if we need to be able to
>>> scan between buckets, we need to flatten and that takes
>>> time and space... lots o' space.
>>
>> Agreed.
>
> Nonsense. It simply needs to be read into a pattern match buffer of
> the largest pattern space *2, and shifted out by halves. Flatten
> is was a quick and dirty API that should never have left alpha :)
>
/a.*b/
How big will that be?
Re: Huge memory leak in mod_substitute
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 1/7/2011 9:58 AM, "Plüm, Rüdiger, VF-Group" wrote:
>
>> -----Original Message-----
>> From: Jim Jagielski
>> Sent: Freitag, 7. Januar 2011 16:57
>>
>> I think it's just the nature of the beast... at least,
>> the nature of the design where if we need to be able to
>> scan between buckets, we need to flatten and that takes
>> time and space... lots o' space.
>
> Agreed.
Nonsense. It simply needs to be read into a pattern match buffer of
the largest pattern space *2, and shifted out by halves. Flatten
is was a quick and dirty API that should never have left alpha :)
RE: Huge memory leak in mod_substitute
Posted by "Plüm, Rüdiger, VF-Group" <ru...@vodafone.com>.
> -----Original Message-----
> From: Jim Jagielski
> Sent: Freitag, 7. Januar 2011 16:57
> To: dev@httpd.apache.org
> Subject: Re: Huge memory leak in mod_substitute
>
>
> On Jan 7, 2011, at 10:46 AM, Nick Kew wrote:
>
> > On Fri, 7 Jan 2011 15:59:26 +0100
> > "Plüm, Rüdiger, VF-Group" <ru...@vodafone.com> wrote:
> >
> >>> The obvious question would be whether the "q" flag affects your
> >>> observations. But the detailed bug report suggests
> that's unlikely.
> >>
> >> Isn't it the case that "q" is the default for one substitution rule
> >> and for more than one it is no longer?
> >> That would explain the different behaviour.
> >
> > Bah. I'd misread Test 2 in the report - saw just the one rule.
> >
> > I guess it just boils down to the logic of when folding does or
> > doesn't get optimised out.
> >
>
> I think it's just the nature of the beast... at least,
> the nature of the design where if we need to be able to
> scan between buckets, we need to flatten and that takes
> time and space... lots o' space.
>
>
Agreed.
Regards
Rüdiger
Re: Huge memory leak in mod_substitute
Posted by Jim Jagielski <ji...@apache.org>.
On Jan 7, 2011, at 10:46 AM, Nick Kew wrote:
> On Fri, 7 Jan 2011 15:59:26 +0100
> "Plüm, Rüdiger, VF-Group" <ru...@vodafone.com> wrote:
>
>>> The obvious question would be whether the "q" flag affects your
>>> observations. But the detailed bug report suggests that's unlikely.
>>
>> Isn't it the case that "q" is the default for one substitution rule
>> and for more than one it is no longer?
>> That would explain the different behaviour.
>
> Bah. I'd misread Test 2 in the report - saw just the one rule.
>
> I guess it just boils down to the logic of when folding does or
> doesn't get optimised out.
>
I think it's just the nature of the beast... at least,
the nature of the design where if we need to be able to
scan between buckets, we need to flatten and that takes
time and space... lots o' space.
Re: Huge memory leak in mod_substitute
Posted by Nick Kew <ni...@webthing.com>.
On Fri, 7 Jan 2011 15:59:26 +0100
"Plüm, Rüdiger, VF-Group" <ru...@vodafone.com> wrote:
> > The obvious question would be whether the "q" flag affects your
> > observations. But the detailed bug report suggests that's unlikely.
>
> Isn't it the case that "q" is the default for one substitution rule
> and for more than one it is no longer?
> That would explain the different behaviour.
Bah. I'd misread Test 2 in the report - saw just the one rule.
I guess it just boils down to the logic of when folding does or
doesn't get optimised out.
--
Nick Kew
RE: Huge memory leak in mod_substitute
Posted by "Plüm, Rüdiger, VF-Group" <ru...@vodafone.com>.
> -----Original Message-----
> From: Nick Kew [mailto:nick@webthing.com]
> Sent: Freitag, 7. Januar 2011 15:49
> To: dev@httpd.apache.org
> Subject: Re: Huge memory leak in mod_substitute
>
> On Fri, 07 Jan 2011 14:26:35 +0100
> Nick Gearls <ni...@gmail.com> wrote:
>
> > Hello,
> >
> > I saw servers completely freezing (the whole system, not
> only httpd) due
> > to a memory leak problem in mod_substitute.
> > This is a major issue as it can completely block your
> systems, sometimes
> > even the fail-over as the server is blocked, but not
> completely down.
> > Even the management interfaces are blocked.
> > We reported the problem
> > (https://issues.apache.org/bugzilla/show_bug.cgi?id=50559), with a
> > (scaled-down) reproducible process to trigger it.
> >
> > Did somebody already begin to work on mod_substitute memory
> management?
>
> The obvious question would be whether the "q" flag affects your
> observations. But the detailed bug report suggests that's unlikely.
Isn't it the case that "q" is the default for one substitution rule
and for more than one it is no longer?
That would explain the different behaviour.
Regards
Rüdiger
Re: Huge memory leak in mod_substitute
Posted by Nick Kew <ni...@webthing.com>.
On Fri, 07 Jan 2011 14:26:35 +0100
Nick Gearls <ni...@gmail.com> wrote:
> Hello,
>
> I saw servers completely freezing (the whole system, not only httpd) due
> to a memory leak problem in mod_substitute.
> This is a major issue as it can completely block your systems, sometimes
> even the fail-over as the server is blocked, but not completely down.
> Even the management interfaces are blocked.
> We reported the problem
> (https://issues.apache.org/bugzilla/show_bug.cgi?id=50559), with a
> (scaled-down) reproducible process to trigger it.
>
> Did somebody already begin to work on mod_substitute memory management?
The obvious question would be whether the "q" flag affects your
observations. But the detailed bug report suggests that's unlikely.
So the second obvious question: have you tried equivalent observations
with mod_sed and/or mod_line_edit?
--
Nick Kew