You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Jani Averbach <ja...@jaa.iki.fi> on 2004/06/16 17:46:12 UTC
svnserve uses over 120MB memory when adding many files
Hello,
When I add 20K files (which each are 1K), svnserve will use over 120MB
of memory during phase when client is displaying
'Adding filename'.
OS is w2k, svn + svnserve 1.0.5
Here is a recipe:
cd workdir
datatree.py datatree
svn mkdir -m "" svn://servername/reponame/trunk
svn co svn://servername/reponame/trunk datatree
cd datatree
svn add *
svn ci -m "here we go"
BR, Jani
datatree.py:
# -*- mode: python -*-
# $Rev: 1182 $
import os
directories = 100
files_per_dir = 200
file_size = 1 * 1024
data = 'x' * 1024
def dir_creator(pi_name,
pi_count):
os.makedirs(pi_name)
for i in range(0, pi_count):
f=open(pi_name + os.sep + '%s%07d.dat' %
(os.path.basename(pi_name), i), 'a+')
f.write('%s:%d' % (pi_name, i))
for i in range(0, file_size/len(data)):
f.write(data)
f.close()
if __name__ == '__main__':
if os.sys.argv == 1:
top_level_name = os.path.abspath(os.sys.argv[1])
else os.sys.argv:
print 'Usage', os.sys.argv[0], '<top level dir>'
os.sys.exit(1)
for i in range(0, directories):
dir_creator(top_level_name + os.sep + '%06d' % i, files_per_dir)
--
Jani Averbach
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnserve uses over 120MB memory when adding many files
Posted by Jani Averbach <ja...@jaa.iki.fi>.
On 2004-06-16 12:53-0500, Ben Collins-Sussman wrote:
>
> Jani, would you file this as a bug, assigned to the 'svnserve'
> component? The recipe looks great.
>
I will (at evening).
Sidenote: At the moment client (committing over ra_dav, in
'Transmitting file data' phase) is using over 120MB of memory and
increasing... (by 20-24K steps).
Br, Jani
--
Jani Averbach
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
svn add, ls and pl use whopping amount of memory when run against many files (20K)
Posted by Jani Averbach <ja...@jaa.iki.fi>.
On 2004-06-16 12:53-0500, Ben Collins-Sussman wrote:
> On Wed, 2004-06-16 at 12:46, Jani Averbach wrote:
> > Hello,
> >
> > When I add 20K files (which each are 1K), svnserve will use over 120MB
> > of memory during phase when client is displaying
> > 'Adding filename'.
> >
> > OS is w2k, svn + svnserve 1.0.5
> >
> > Here is a recipe:
>
> Jani, would you file this as a bug, assigned to the 'svnserve'
> component? The recipe looks great.
>
This is issue 1928:
http://subversion.tigris.org/issues/show_bug.cgi?id=1928
A special extra is that
svn pl -R file:///repopath/trunk
took over __1.8GB__ before I killed it.
This has been tested with r10009 and 1.0.5 on Linux. If someone could
verify, that would be nice.
BR, Jani
--
Jani Averbach
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnserve uses over 120MB memory when adding many files
Posted by Ben Collins-Sussman <su...@collab.net>.
On Wed, 2004-06-16 at 12:46, Jani Averbach wrote:
> Hello,
>
> When I add 20K files (which each are 1K), svnserve will use over 120MB
> of memory during phase when client is displaying
> 'Adding filename'.
>
> OS is w2k, svn + svnserve 1.0.5
>
> Here is a recipe:
Jani, would you file this as a bug, assigned to the 'svnserve'
component? The recipe looks great.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnserve uses over 120MB memory when adding many files
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2004-06-16 at 14:55, Ben Collins-Sussman wrote:
> Perhaps we should nominate it for 1.0.x backport, then? Memory leaks
> are generally considered serious.
It only happens in limited circumstances and it's not the simplest of
changes.
Also, this isn't a memory leak. The architecture of our commit system
is such that we use some memory for each committed file. r9185 just
reduces the amount used per file.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnserve uses over 120MB memory when adding many files
Posted by Ben Collins-Sussman <su...@collab.net>.
On Wed, 2004-06-16 at 13:25, Greg Hudson wrote:
> This problem was fixed on the trunk in r9185.
>
Perhaps we should nominate it for 1.0.x backport, then? Memory leaks
are generally considered serious.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnserve uses over 120MB memory when adding many files
Posted by Greg Hudson <gh...@MIT.EDU>.
This problem was fixed on the trunk in r9185.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org