You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2009/10/19 20:50:00 UTC
DO NOT REPLY [Bug 48024] New: mod_sed consumes too much memory when
used inside reverse proxy and line sizes are very long e.g. 1-5M
https://issues.apache.org/bugzilla/show_bug.cgi?id=48024
Summary: mod_sed consumes too much memory when used inside
reverse proxy and line sizes are very long e.g. 1-5M
Product: Apache httpd-2
Version: 2.3-HEAD
Platform: PC
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Core
AssignedTo: bugs@httpd.apache.org
ReportedBy: basant.kukreja@sun.com
This has been reported to me by Bram <mo...@mail.wizbit.be>
When Apache is used as a reverse proxy and mod_sed is used to filter
to content and the output contains lines which are really long e.g
1M-4M chracters in single line, mod_sed consumes too much memory.
Here is the snips of the mail from Bram :
- 1 mb file with only a newline at the end
$ perl -we '$\="\x0D\x0A"; print "a" x (1 * 2 ** 20);' > /var/www/a.1.bin
When the file is downloaded apache uses 70 mb of memory
- 2 mb file with only a newline at the end:
$ perl -we '$\="\x0D\x0A"; print "a" x (2 * 2 ** 20);' > /var/www/a.2.bin
When the file is downloaded apache uses 277 mb of memory
- 3 mb file with only a newline at the end:
$ perl -we '$\="\x0D\x0A"; print "a" x (3 * 2 ** 20);' > /var/www/a.3.bin
When the file is downloaded apache uses 608 mb of memory
- 4 mb file with only a newline at the end:
$ perl -we '$\="\x0D\x0A"; print "a" x (4 * 2 ** 20);' > /var/www/a.2.bin
When the file is downloaded apache uses 1070 mb of memory
------------------------------------------------
I am able to reproduce the issue with following configuration :
LoadModule sed_module modules/mod_sed.so
LoadModule headers_module modules/mod_headers.so
LoadModule substitute_module modules/mod_substitute.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
ProxyPass / http://hostname:8080/
<Location />
Header unset Content-Length
SetOutputFilter Sed
OutputSed s/\(https:\/\/192.168.173.114\)\//\1:444\//g
</Location>
------------------------------------------------
The above doesn't have any impact if line sizes are small which is the typical
case with html or ascii text.
* Using core category because mod_sed component is not available.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48024] mod_sed consumes too much memory when used
inside reverse proxy and line sizes are very long e.g. 1-5M
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48024
--- Comment #2 from Basant Kumar Kukreja <ba...@sun.com> 2009-10-19 12:06:15 UTC ---
I need to correct the grammer :
==============================================
Fix for this is that grow_buffer should grow the sizes at least twice each time
it is invoked. This reduced the the reallocation to just 9 times for 2M line
size.
==============================================
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48024] mod_sed consumes too much memory when used
inside reverse proxy and line sizes are very long e.g. 1-5M
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48024
Basant Kumar Kukreja <ba...@sun.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Keywords| |PatchAvailable
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48024] mod_sed consumes too much memory when used
inside reverse proxy and line sizes are very long e.g. 1-5M
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48024
--- Comment #1 from Basant Kumar Kukreja <ba...@sun.com> 2009-10-19 11:54:06 UTC ---
Created an attachment (id=24397)
--> (https://issues.apache.org/bugzilla/attachment.cgi?id=24397)
Patch
sed needs to buffer the line until it finds the end of the line.
At present it increments the line buffer by 4KB size or based on new sizes.
When mod_sed is used inside apache with regular files, number of time
reallocation happens are less because single brigade contains the entire
file.
But when mod_sed is used as a reverse proxy then the number of times
sed_response is called is called can be huge which means buffer will be
reallocated many many times compared to filtering text directly. I noticed
that for 2M characters in single line then mod_sed reallocated stuff in 275
times.
Fix for this is that grow_buffer should grow the sizes at least each time it
is invoked. This reduced the the reallocation to just 9 times for 2M line
size.
Fix is attached.
Bram confirmed that fix worked for him. Here is the comments from Bram.
----------------------------------
For a 2 MB, 3 MB and 4 MB file it seems to be below 26/27 MB of memory.
For a 20 MB file it is around 98 MB of memory
----------------------------------
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48024] mod_sed consumes too much memory when used
inside reverse proxy and line sizes are very long e.g. 1-5M
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48024
Basant Kumar Kukreja <ba...@sun.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #24397|0 |1
is obsolete| |
--- Comment #3 from Basant Kumar Kukreja <ba...@sun.com> 2009-10-19 12:18:46 UTC ---
Created an attachment (id=24398)
--> (https://issues.apache.org/bugzilla/attachment.cgi?id=24398)
Patch against trunk
Patched against trunk (Previous patch was generated by mistake by some other
repository.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48024] mod_sed consumes too much memory when used
inside reverse proxy and line sizes are very long e.g. 1-5M
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48024
Ruediger Pluem <rp...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Keywords| |FixedInTrunk
Component|Core |mod_sed
--- Comment #4 from Ruediger Pluem <rp...@apache.org> 2009-10-19 14:25:24 CEST ---
Thanks for the patch. Fixed in trunk as r826772. Also I added mod_sed as
component. Thanks for noting.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48024] mod_sed consumes too much memory when used
inside reverse proxy and line sizes are very long e.g. 1-5M
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48024
Stefan Fritsch <sf...@sfritsch.de> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
--- Comment #5 from Stefan Fritsch <sf...@sfritsch.de> 2011-06-25 12:29:17 UTC ---
Closing since mod_sed is only in trunk
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org