You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Nick Gearls <ni...@gmail.com> on 2009/06/10 13:11:16 UTC

mod_substitute memory problem

Hello,

There seems to be a memory problem when substituing something on very 
long lines (several hundreds KB). this problem is different from bug 
44948 (I applied this patch).

When using something like "Substitute s/string1/string2/" on a 300 KB 
line with 30 times "string1" on the line, there are 2 problems:

1. It uses a huge amount of memory, as - I think - it reallocates a new 
buffer (300 KB) before each substitute without freeing the previous one

2. The memory is not freeed at the end of the HTTP request.
Maybe is it due to Keep-alive?

If using the "q" switch to not flatten the buckets, it uses almost no 
memory. Btw, it correctly handles more 60 substitutions on the same line 
(some shorter, some longer) without the q flag !?! When exactly is this 
flag needed?


In macro SEDSCAT, we have
	s2 = apr_pstrmemdup(pool, buff, blen);
         s1 = apr_pstrcat(pool, s1, s2, NULL);
Shouldn't we free the previous s1 buffer?

Anyway, I do not understand why the memory is not released, as the pool 
is supposed to be destroyed.

Regards,

Nick

Re: mod_substitute memory problem

Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
Nick Gearls wrote:
> 
> If using the "q" switch to not flatten the buckets, it uses almost no
> memory. Btw, it correctly handles more 60 substitutions on the same line
> (some shorter, some longer) without the q flag !?! When exactly is this
> flag needed?

The "q" switch causes the pattern to fail every time the match pattern
spans two filter input buffers.  So it's very hit-or-miss and you have
no assurance that it can process this all.

Since the filter is built for text, and text is generally arranged with
lines short than 300KB, I wouldn't spend a bunch of time on this.  Take
a look at httpd trunk's mod_sed and see if the real thing is kinder with
respect to memory utilization; it truly is a stream editor.

Re: mod_substitute memory problem

Posted by Ruediger Pluem <rp...@apache.org>.

On 06/11/2009 09:11 AM, Nick Gearls wrote:
> Yes, that's what I meant.
> 
> The kind of thing that (surprisingly) works is
>  s/(pattern)/---toremove---$1/q
>  s/---toremove---//q
> 
> The second one should obviously use the n flag, but even with a regex it
> works. I also tried with the second string longer or shorter, and it
> always works (at least on my test page).
> Can I really consider this as safe?


I guess this special case yes, but not in general.

Regards

Rüdiger


Re: mod_substitute memory problem

Posted by Nick Gearls <ni...@gmail.com>.
Yes, that's what I meant.

The kind of thing that (surprisingly) works is
  s/(pattern)/---toremove---$1/q
  s/---toremove---//q

The second one should obviously use the n flag, but even with a regex it 
works. I also tried with the second string longer or shorter, and it 
always works (at least on my test page).
Can I really consider this as safe?

Thanks,

Nick


Ruediger Pluem wrote:
> 
> On 06/10/2009 06:18 PM, Nick Gearls wrote:
>> Hi Nick,
>>
>> Do you mean that mod_sed should be used instead of mod_substitute?
>> Because it is more complete, or more mature? I only need the substitution.
>>
>> About the flattening: is the q flag only needed if you want to use two
>> subst on the same line with overlapping between them?
> 
> It's the other way round: If you have two subst with overlappings using the
> q flag might result in missing substitutions. As this does not seem to be the
> case for your problem it is safe (and good) to use it as it solves your memory
> problem.
> 
> Regards
> 
> Rüdiger
> 
> 

Re: mod_substitute memory problem

Posted by Ruediger Pluem <rp...@apache.org>.

On 06/10/2009 06:18 PM, Nick Gearls wrote:
> Hi Nick,
> 
> Do you mean that mod_sed should be used instead of mod_substitute?
> Because it is more complete, or more mature? I only need the substitution.
> 
> About the flattening: is the q flag only needed if you want to use two
> subst on the same line with overlapping between them?

It's the other way round: If you have two subst with overlappings using the
q flag might result in missing substitutions. As this does not seem to be the
case for your problem it is safe (and good) to use it as it solves your memory
problem.

Regards

Rüdiger


Re: mod_substitute memory problem

Posted by Nick Gearls <ni...@gmail.com>.
Hi Nick,

Do you mean that mod_sed should be used instead of mod_substitute?
Because it is more complete, or more mature? I only need the substitution.

About the flattening: is the q flag only needed if you want to use two 
subst on the same line with overlapping between them?

Thanks,

Nick


Nick Kew wrote:
> On Wed, 10 Jun 2009 13:11:16 +0200
> Nick Gearls <ni...@gmail.com> wrote:
> 
>> 2. The memory is not freeed at the end of the HTTP request.
>> Maybe is it due to Keep-alive?
> 
> It's recycled within the server.
> 
>> If using the "q" switch to not flatten the buckets, it uses almost no 
>> memory. Btw, it correctly handles more 60 substitutions on the same
>> line (some shorter, some longer) without the q flag !?! When exactly
>> is this flag needed?
> 
> Rarely, IMO.  When I wrote mod_line_edit - which does essentially the
> same as mod_substitute - I didn't provide that option, and instead
> documented it as not working with overlapping substitutions.
> 
> Another "slow by default" flag in mod_substitute is that it
> uses regexps by default.  Turn them off unless you definitely
> need them!
> 
>> Anyway, I do not understand why the memory is not released, as the
>> pool is supposed to be destroyed.
> 
> It's recycled within the server.
> 
> mod_sed is now the state-of-the-art.  Perhaps a visit to
> http://httpd.apache.org/docs/2.3/mod/mod_sed.html
> would be in order.
> 

Re: mod_substitute memory problem

Posted by Nick Kew <ni...@webthing.com>.
On Wed, 10 Jun 2009 13:11:16 +0200
Nick Gearls <ni...@gmail.com> wrote:

> 2. The memory is not freeed at the end of the HTTP request.
> Maybe is it due to Keep-alive?

It's recycled within the server.

> If using the "q" switch to not flatten the buckets, it uses almost no 
> memory. Btw, it correctly handles more 60 substitutions on the same
> line (some shorter, some longer) without the q flag !?! When exactly
> is this flag needed?

Rarely, IMO.  When I wrote mod_line_edit - which does essentially the
same as mod_substitute - I didn't provide that option, and instead
documented it as not working with overlapping substitutions.

Another "slow by default" flag in mod_substitute is that it
uses regexps by default.  Turn them off unless you definitely
need them!

> Anyway, I do not understand why the memory is not released, as the
> pool is supposed to be destroyed.

It's recycled within the server.

mod_sed is now the state-of-the-art.  Perhaps a visit to
http://httpd.apache.org/docs/2.3/mod/mod_sed.html
would be in order.

-- 
Nick Kew

Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/

Re: mod_substitute memory problem

Posted by Nick Gearls <ni...@gmail.com>.
The problem is real.
I have an application which generates one line of about 300 KB :-(
In this case, Apache eats up all the memory (about 1 GB), and never 
gives it back to the OS - game over - reboot !

How could this be solved?

Thanks,

Nick


Dan Poirier wrote:
> Nick Gearls <ni...@gmail.com> writes:
> 
>> There seems to be a memory problem when substituing something on very
>> long lines (several hundreds KB). this problem is different from bug
>> 44948 (I applied this patch).
>>
>> When using something like "Substitute s/string1/string2/" on a 300 KB
>> line with 30 times "string1" on the line, there are 2 problems:
>>
>> 1. It uses a huge amount of memory, as - I think - it reallocates a
>> new buffer (300 KB) before each substitute without freeing the
>> previous one
> 
> I think you're right - most of the working memory is not released until
> the request is over.  This is kind of a worst-case for mod_substitute's
> temporary memory usage.
> 
> Is this a problem in real cases?  Or just in artificial testing?
> If these are real requests, maybe if you tell a little more about them,
> we can find a better solution.
> 
>> 2. The memory is not freeed at the end of the HTTP request.
>> Maybe is it due to Keep-alive?
> 
> If by "the memory is not freed", you mean the memory usage for the
> http process as reported by the OS doesn't go down, that's right.
> But at the end of the request, the memory should be released back
> to Apache to re-use.  You can test this by repeating your test several
> times - after the first time, the process's memory usage should not keep
> going up the way it does the first time.
> 
> If that's a problem in your case, you can set MaxMemFree to ask Apache
> to release memory by calling free() above a certain threshold.  Even
> then, whether the process releases the memory back to the OS depends on
> the OS, so it might not gain you anything.
> 
> But if this is going to happen very often, you might as well let Apache
> hold on to the memory; it's just going to grab it back again the next
> time this happens.
> 
>> If using the "q" switch to not flatten the buckets, it uses almost no
>> memory. Btw, it correctly handles more 60 substitutions on the same
>> line (some shorter, some longer) without the q flag !?! When exactly
>> is this flag needed?
> 
> Don't know about this one.
> 
>> In macro SEDSCAT, we have
>> 	s2 = apr_pstrmemdup(pool, buff, blen);
>>         s1 = apr_pstrcat(pool, s1, s2, NULL);
>> Shouldn't we free the previous s1 buffer?
> 
> Pools don't provide a way of freeing individual allocations; you have
> to destroy the whole pool.
> 
>> Anyway, I do not understand why the memory is not released, as the
>> pool is supposed to be destroyed.
> 
> See above.
> 

Re: mod_substitute memory problem

Posted by Dan Poirier <po...@pobox.com>.
Nick Gearls <ni...@gmail.com> writes:

> There seems to be a memory problem when substituing something on very
> long lines (several hundreds KB). this problem is different from bug
> 44948 (I applied this patch).
>
> When using something like "Substitute s/string1/string2/" on a 300 KB
> line with 30 times "string1" on the line, there are 2 problems:
>
> 1. It uses a huge amount of memory, as - I think - it reallocates a
> new buffer (300 KB) before each substitute without freeing the
> previous one

I think you're right - most of the working memory is not released until
the request is over.  This is kind of a worst-case for mod_substitute's
temporary memory usage.

Is this a problem in real cases?  Or just in artificial testing?
If these are real requests, maybe if you tell a little more about them,
we can find a better solution.

> 2. The memory is not freeed at the end of the HTTP request.
> Maybe is it due to Keep-alive?

If by "the memory is not freed", you mean the memory usage for the
http process as reported by the OS doesn't go down, that's right.
But at the end of the request, the memory should be released back
to Apache to re-use.  You can test this by repeating your test several
times - after the first time, the process's memory usage should not keep
going up the way it does the first time.

If that's a problem in your case, you can set MaxMemFree to ask Apache
to release memory by calling free() above a certain threshold.  Even
then, whether the process releases the memory back to the OS depends on
the OS, so it might not gain you anything.

But if this is going to happen very often, you might as well let Apache
hold on to the memory; it's just going to grab it back again the next
time this happens.

> If using the "q" switch to not flatten the buckets, it uses almost no
> memory. Btw, it correctly handles more 60 substitutions on the same
> line (some shorter, some longer) without the q flag !?! When exactly
> is this flag needed?

Don't know about this one.

> In macro SEDSCAT, we have
> 	s2 = apr_pstrmemdup(pool, buff, blen);
>         s1 = apr_pstrcat(pool, s1, s2, NULL);
> Shouldn't we free the previous s1 buffer?

Pools don't provide a way of freeing individual allocations; you have
to destroy the whole pool.

> Anyway, I do not understand why the memory is not released, as the
> pool is supposed to be destroyed.

See above.

-- 
Dan Poirier <po...@pobox.com>