You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2009/11/04 22:00:41 UTC
DO NOT REPLY [Bug 48130] New: DAV operations on large filesets
consume all the server memory
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
Summary: DAV operations on large filesets consume all the
server memory
Product: Apache httpd-2
Version: 2.2.14
Platform: Other
OS/Version: AIX
Status: NEW
Severity: normal
Priority: P2
Component: mod_dav
AssignedTo: bugs@httpd.apache.org
ReportedBy: ville@jussi.la
Compiled apache with --enable-dav with gcc version 4.2.0. AIX OS level is
5300-08-04-0844.
Enabled DAV for NetApp vFiler filesystem (=network mounted) including up to 57
directories each holding 3000 to 20000 files. Filesystem is holding
approximately 70 Gigabytes of data.
Apache2 installation and DavLockDB located on SAN-volume.
Listing directories via a browser (without DAV) does not make httpd to increase
memory allocation when browsing through directories. Everything works nicely.
Listing directories via Windows XP SP3 DAV-client makes httpd to allocate more
and more memory when opening directory after directory. Everythings seems to
work, but more and more memory is allocated and nothings seems to be freed. In
the end, all the AIX memory is allocated by the httpd and AIX starts to kill
processes.
DAV enabled with :
<Directory "/netapp/...">
Dav On
Options Indexes
Order Allow,Deny
Allow from all
AuthType Basic
AuthBasicProvider file
AuthUserFile /home/.../httpd/conf/passwords
AuthGroupFile /home/.../httpd/conf/groups
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
--- Comment #3 from Diego Santa Cruz <Di...@spinetix.com> 2012-01-30 15:17:28 UTC ---
We have also come into this problem in our product. As in our case httpd runs
on an embedded device memory is a bit constrained, but some users still create
directories with more than a thousand files per directory which uses up a lot
of memory.
To fix the situation we have done a few patches to mod_dav and mod_dav_fs to
remove as much O(N) dependencies on the number of files per directory. We have
had them in production since long with no issues, so I am pushing them to be
considered for inclusion in httpd. Note that the patches are probably not
perfect but still they do improve the situation significantly. The only
remaining O(N) source I see is the lock DB, but that is O(N) in the number of
locks, so I think that is much less of an issue.
Patches against httpd 2.2.21 follow.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
--- Comment #4 from Diego Santa Cruz <Di...@spinetix.com> 2012-01-30 15:19:58 UTC ---
Created attachment 28232
--> https://issues.apache.org/bugzilla/attachment.cgi?id=28232
mod_dav / mod_dav_fs memory reduction patch 1/5
Use a subpool while doing iteration on dir entries in dav_fs_walker to reduce
memory usage, however this does not help on *nix since apr_stat does not use
the pool in those platforms.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
--- Comment #5 from Diego Santa Cruz <Di...@spinetix.com> 2012-01-30 15:23:25 UTC ---
Created attachment 28233
--> https://issues.apache.org/bugzilla/attachment.cgi?id=28233
mod_dav / mod_dav_fs memory reduction patch 2/5
Use subpool when allocating a propdb and introduce function variants that
allow destroying the propdb pool when the propdb of a resource is closed
(dav_close_propdb2() and dav_get_props2()).
Then use these new functions to destroy the propdb pool after use in
dav_propfind_walker() reducing memory usage to O(1) instead of O(N) with the
number of resources in a collection and greatly helps PROPFIND.
So far these new functions are not exported from the mod_dav module, that might
be considered when an mod_dav API addition is welcome.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
mariogruber@schleuniger.ch changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |mariogruber@schleuniger.ch
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
--- Comment #6 from Diego Santa Cruz <Di...@spinetix.com> 2012-01-30 15:24:20 UTC ---
Created attachment 28234
--> https://issues.apache.org/bugzilla/attachment.cgi?id=28234
mod_dav / mod_dav_fs memory reduction patch 3/5
Use a scratch pool when walking the tree for DAV copy and move requests so
that memory usage is O(1) instead of O(N) with the number of resources
per directory; note however that it is still O(N) with the number of
directories in the walked hierarchy.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 48130] DAV operations on large filesets consume all the OS
memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
Christophe JAILLET <ch...@wanadoo.fr> changed:
What |Removed |Added
----------------------------------------------------------------------------
Keywords| |PatchAvailable
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
Ville Jussila <vi...@jussi.la> changed:
What |Removed |Added
----------------------------------------------------------------------------
Summary|DAV operations on large |DAV operations on large
|filesets consume all the |filesets consume all the OS
|server memory |memory
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
Ville Jussila <vi...@jussi.la> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |ville@jussi.la
Severity|normal |major
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
Stefan Fritsch <sf...@sfritsch.de> changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|2.2.14 |2.3-HEAD
OS/Version|AIX |All
--- Comment #2 from Stefan Fritsch <sf...@sfritsch.de> 2010-03-07 13:11:00 UTC ---
Memory management in mod_dav is broken. A PROPFIND of a dir with 10000 files
needs around 140MB of memory on my system.
Some initiative to overhaul pool usage was described here:
http://mail-archives.apache.org/mod_mbox/httpd-dev/200305.mbox/<86smqzgcuf.fsf%40kepler.ch.collab.net>
Unfortunately this was never completed.
Probably the pool guidelines from subversion should be used:
http://subversion.apache.org/docs/community-guide/conventions.html#apr-pools
This would mean that most mod_dav functions would need to tak a scratch pool
and a result pool as parameters. And dav_push_error needs some magic to ensure
correct lifetime of the error stack.
In addition there is the problem that the response is streamed to the client
while locks are held (PR 36636).
This looks like a large amount of work. Probably a project for Apache 3.0 and
not for 2.4.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
--- Comment #7 from Diego Santa Cruz <Di...@spinetix.com> 2012-01-30 15:25:37 UTC ---
Created attachment 28235
--> https://issues.apache.org/bugzilla/attachment.cgi?id=28235
mod_dav / mod_dav_fs memory reduction patch 4/5
When doing a PROPFIND mod_dav creates a subrequest for some live properties
(e.g., getcontenttype and getcontentlanguage) but does not destroy the
subrequest, thus the associated memory is not freed until the main request
is done. This patch destroys the subrequest when a propdb is closed,
significantly reducing the amount of memory when doing a PROPFIND for all
properties or a set of properties requiring subrequests.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
--- Comment #8 from Diego Santa Cruz <Di...@spinetix.com> 2012-01-30 15:26:20 UTC ---
Created attachment 28236
--> https://issues.apache.org/bugzilla/attachment.cgi?id=28236
mod_dav / mod_dav_fs memory reduction patch 5/5
mod_dav_fs' dav_fs_walker() always uses the same pool during the tree walk,
this patch changes that to using subpools for child resources to avoid
having an O(N) memory consumption on the number of resources per dir.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
Diego Santa Cruz <Di...@spinetix.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |Diego.SantaCruz@spinetix.co
| |m
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
--- Comment #1 from vuser1 <vu...@test123.ru> 2009-12-21 13:14:54 UTC ---
Same problem on Win32. There must be memory leak. Steps:
Step1. On Win2003 client, right-click WebDAV folder properties. Client
calculates folder size - enumerates subfolders.
Result: 117068 Files, 11537 sub-folders, server process httpd.exe has 602Mb
WorkingSet and 629Mb PrivateBytes.
Step2. On server, execute empty.exe ResourseKit utility which calls
SetProcessWorkingSetSize(-1, -1) for httpd.
Result: httpd.exe WorkingSet decreases to 2Mb
Step3. On client, close properties and open again.
Result: 610Mb WorkingSet and 900Mb PrivateBytes.
<Directory "H:/sklad">
Allow from all
Options FollowSymLinks Indexes
AllowOverride Limit
ReadmeName /README.html
HeaderName /HEADER.html
Dav On
AuthType Digest
AuthDigestDomain /files/sklad
AuthName "sklad"
AuthDigestProvider file
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 48130] DAV operations on large filesets consume
all the OS memory
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=48130
Ruediger Pluem <rp...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Blocks| |52123
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org