You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Barry Scott <ba...@barrys-emacs.org> on 2004/09/03 19:54:29 UTC
Checkin needs giga bytes of memory
I've just tried to checking 12,000 files that only had svn:eol-style
native set on them and it failed after
1.2GB of VM was not enough. (1.0.6 windows client checkin to 1.1.0-rc2
FreeBSD HTTP server).
Is it worth a client refusing to attempt a checkin when there are a lot
of changes rather then wait
10 to 20 minutes and get a fatal error? The reason I ask is that I'm
wondering if I should implement
this as a test in pysvn WorkBench GUI.
Barry
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Checkin needs giga bytes of memory
Posted by Barry Scott <ba...@barrys-emacs.org>.
On Sep 3, 2004, at 21:29, Ben Collins-Sussman wrote:
> On Fri, 2004-09-03 at 14:54, Barry Scott wrote:
>> I've just tried to checking 12,000 files that only had svn:eol-style
>> native set on them and it failed after
>> 1.2GB of VM was not enough. (1.0.6 windows client checkin to 1.1.0-rc2
>> FreeBSD HTTP server).
>
> We should have no memory leaks like this. Everything should be
> streamy,
> and memery use should stay small and roughly constant-sized.
>
> Can you please be more specific so we can reproduce and debug?
> - Was the 1.2GB VM on the client or the server
Problem on the Windows Client. The server did not have any noticeable
resource use (very nice).
> - How do we reproduce? Just create 12000 files with that property
> on
> them, 'svn add' and 'svn commit'? How big should the files be? How
> are they distributed around the tree?
1. Created a new repo and add a trunk dir to it.
2. svn Checkout trunk
3. svn add 11093 files total 260MB and are a mix of mostly text and
some binary,
in 1300 directories.
4. svn ci - no problem that I remember
5. Noticed that auto-props have been ingored (have not confirmed this
problem yet)
6. issue 11000 svn ps svn:eol-style native commands
7. svn ci ... windows reports its low on VM, task manger shows 1.2GB of
VM on the SVN process
before it fails.
Given that the about took many hours to run you will forgive me not
repeating the sequence.
As for debugging this, personally I'd using dmalloc from dmalloc.com.
valgrind slows things
down to much.
Barry
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Checkin needs giga bytes of memory
Posted by Barry Scott <ba...@barrys-emacs.org>.
On Sep 3, 2004, at 21:29, Ben Collins-Sussman wrote:
> On Fri, 2004-09-03 at 14:54, Barry Scott wrote:
>> I've just tried to checking 12,000 files that only had svn:eol-style
>> native set on them and it failed after
>> 1.2GB of VM was not enough. (1.0.6 windows client checkin to 1.1.0-rc2
>> FreeBSD HTTP server).
>
> We should have no memory leaks like this. Everything should be
> streamy,
> and memery use should stay small and roughly constant-sized.
>
> Can you please be more specific so we can reproduce and debug?
I'd guess that the memory usage is proportional to the number of files.
In which case comparing the memory allocation patterns of a 10 file
checkin
vs. 100 file checkin should be sufficient to locate the blocks of
memory that
are taking the space.
Barry
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Checkin needs giga bytes of memory
Posted by Ben Collins-Sussman <su...@collab.net>.
On Fri, 2004-09-03 at 15:45, Michael wrote:
> Been like this for quite a while now.
Yeah, this is issue 1964, which you reported to us.
http://subversion.tigris.org/issues/show_bug.cgi?id=1964
I wonder if Barry is just reproducing this same issue.
It's also not clear if anyone has reproduced it yet. The mail thread
has a response from ghudson saying, "well, we *do* hold a tiny structure
in memory for every object being committed." It's not clear if that
design is really causing this problem, though.
Any developers interested in looking at this issue? It might be easy to
reproduce, but hard to diagnose.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Checkin needs giga bytes of memory
Posted by Michael <ec...@yahoo.com>.
--- Ben Collins-Sussman <su...@collab.net> wrote:
> We should have no memory leaks like this. Everything should be
> streamy, and memery use should stay small and roughly constant-sized.
>
> Can you please be more specific so we can reproduce and debug?
>
> - Was the 1.2GB VM on the client or the server?
>
> - How do we reproduce? Just create 12000 files with that property
> on
> them, 'svn add' and 'svn commit'? How big should the files be? How
> are they distributed around the tree?
I don't know about this leak, but there are plenty other leaks to
examine. I posted a simple formula for one a while back.
If you create tens of thousands of files distributed among 100
directories in a directory tree only two levels deep, svn add, then svn
commit, you will see the server and client grow in size with ra_svn. I
see the same growth in ra_local.
Been like this for quite a while now.
Michael Price
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Checkin needs giga bytes of memory
Posted by Ben Collins-Sussman <su...@collab.net>.
On Fri, 2004-09-03 at 14:54, Barry Scott wrote:
> I've just tried to checking 12,000 files that only had svn:eol-style
> native set on them and it failed after
> 1.2GB of VM was not enough. (1.0.6 windows client checkin to 1.1.0-rc2
> FreeBSD HTTP server).
We should have no memory leaks like this. Everything should be streamy,
and memery use should stay small and roughly constant-sized.
Can you please be more specific so we can reproduce and debug?
- Was the 1.2GB VM on the client or the server?
- How do we reproduce? Just create 12000 files with that property on
them, 'svn add' and 'svn commit'? How big should the files be? How
are they distributed around the tree?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org