You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Lukas Ruf <ru...@rawip.org> on 2004/04/29 20:28:36 UTC
svn consumes all memory
Dear all,
I have several large projects with huge files within svn that I would
like to checkout again.
However, svn does not proceed as I would expect but consumes nearly
all memory and ends without any notice.
top reports
10446 ruf 25 0 522m 370m 7460 R 99.5 41.9 241:51.21 svn
I am using svn on a Linux 2.6.5 box (client & repository):
svn, version 1.0.2 (r9423)
compiled Apr 27 2004, 17:31:38
apache 2.0.49-1
debian unstable (latest updates installed).
I access all the repository only via https.
svn stops after approx. 620MB -- the repostiory (du -h) reports 702MB.
I exported the cvs tree and imported the data into svn. The export
contains 2.2GB?
- Does svn write any logs anywhere that are human-readable?
- Is there any way to easily check the files in the repository against
the cvs-exported ones?
Thanks for any help!
wbr,
Lukas
--
Lukas Ruf | Wanna know anything about raw |
<http://www.lpr.ch> | IP? -> <http://www.rawip.org> |
eMail Style Guide: <http://www.rawip.org/style.html>|
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by McClain Looney <m...@loonsoft.com>.
the bdb developers consider such configuration "suicidal". try keeping your
repo local.
On Friday 30 April 2004 01:47 am, Lukas Ruf wrote:
> The repository is mounted via NFS (server Solaris 2.8) and accessed
> only by one client (Debian unstable, Linux 2.6.5) via https only.
--
McClain Looney
LoonSoft LLC
m@loonsoft.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by Lukas Ruf <ru...@rawip.org>.
Dear all,
coming back on the email I wroe a couple of days ago,
> Lukas Ruf <ru...@rawip.org> [2004-05-07 11:09]:
>
> > kfogel@collab.net <kf...@collab.net> [2004-05-06 21:16]:
> >
> > Lukas Ruf <ru...@rawip.org> writes:
> > > I put all the repositories to a local hard-disk, re-created the
> > > problematic repository from scratch, tried a checkout via https,
> > > killed the svn client by 'kill -9' (because I had to disconnect
> > > temporarely), restarted the checkout later, and experienced the
> > > same problem: Linux killed the client by abort (at least I saw
> > > this message on screen).
> > >
>
> > When you restarted the checkout, was the old (incomplete) working
> > copy still present, or did you restart absolutely fresh, with
> > nothing in the way? (Subversion is supposed to gracefully resume
> > even if the wc is partially present already, but there might be a
> > bug in the resumption code or something.)
> >
>
> I tried both. In both cases, same problems.
>
> > > - svn has the atomic operations. Is it possible that it crashes
> > > due to linux' limits because it saves also very large
> > > files/directories first in memory (one directory contained
> > > approx. 1.5GB of data but I have only 1GB RAM) before it writes
> > > the checkout to the disk?
> >
> > This seems unlikely -- Subversion tries pretty hard to be streamy.
> >
>
> Could it be a problem that I have files of 500MB and more?
>
I moved the large archives out of subversion (unfortunately) and,
hence, I worked to completion after a very long delay.
Now, I need to checkout the data onto another box -- and the svn
client shows the same behaviour for a directory of approx. 180MB:
top reports:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18690 ruf 17 0 120m 120m 2780 R 99.7 12.0 41:17.44 svn
> >
> > I think we need transcripts and concrete details at this point.
> >
>
> How can I create them? Are there any command line options?
>
Is there anything I can do to help -- except for giving access to the
repository/data?
wbr,
Lukas
--
Lukas Ruf | Wanna know anything about raw |
<http://www.lpr.ch> | IP? -> <http://www.rawip.org> |
eMail Style Guide: <http://www.rawip.org/style.html>|
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by Lukas Ruf <ru...@rawip.org>.
> kfogel@collab.net <kf...@collab.net> [2004-05-06 21:16]:
>
> Lukas Ruf <ru...@rawip.org> writes:
> > I put all the repositories to a local hard-disk, re-created the
> > problematic repository from scratch, tried a checkout via https,
> > killed the svn client by 'kill -9' (because I had to disconnect
> > temporarely), restarted the checkout later, and experienced the
> > same problem: Linux killed the client by abort (at least I saw
> > this message on screen).
> >
> > Hence my questions: - does 'killing a checkout process by -9'
> > destroy the repository? - if so what is the best way to stop a
> > large checkout?
>
> No, what you did shouldn't affect the repository at all.
>
thanks for the confirmation.
> When you restarted the checkout, was the old (incomplete) working
> copy still present, or did you restart absolutely fresh, with
> nothing in the way? (Subversion is supposed to gracefully resume
> even if the wc is partially present already, but there might be a
> bug in the resumption code or something.)
>
I tried both. In both cases, same problems.
> > - svn has the atomic operations. Is it possible that it crashes
> > due to linux' limits because it saves also very large
> > files/directories first in memory (one directory contained approx.
> > 1.5GB of data but I have only 1GB RAM) before it writes the
> > checkout to the disk?
>
> This seems unlikely -- Subversion tries pretty hard to be streamy.
>
Could it be a problem that I have files of 500MB and more?
> > I stumbled over this idea when I realized that after the abort
> > there were no data on disk.
>
> I think we need transcripts and concrete details at this point.
>
How can I create them? Are there any command line options?
wbr,
Lukas
--
Lukas Ruf | Wanna know anything about raw |
<http://www.lpr.ch> | IP? -> <http://www.rawip.org> |
eMail Style Guide: <http://www.rawip.org/style.html>|
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by kf...@collab.net.
Lukas Ruf <ru...@rawip.org> writes:
> I put all the repositories to a local hard-disk,
> re-created the problematic repository from scratch,
> tried a checkout via https,
> killed the svn client by 'kill -9' (because I had to disconnect temporarely),
> restarted the checkout later,
> and experienced the same problem: Linux killed the client by abort (at
> least I saw this message on screen).
>
> Hence my questions:
> - does 'killing a checkout process by -9' destroy the repository?
> - if so what is the best way to stop a large checkout?
No, what you did shouldn't affect the repository at all.
When you restarted the checkout, was the old (incomplete) working copy
still present, or did you restart absolutely fresh, with nothing in
the way? (Subversion is supposed to gracefully resume even if the wc
is partially present already, but there might be a bug in the
resumption code or something.)
> - svn has the atomic operations. Is it possible that it crashes due
> to linux' limits because it saves also very large files/directories
> first in memory (one directory contained approx. 1.5GB of data but I
> have only 1GB RAM) before it writes the checkout to the disk?
This seems unlikely -- Subversion tries pretty hard to be streamy.
> I stumbled over this idea when I realized that after the abort there
> were no data on disk.
I think we need transcripts and concrete details at this point.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by Lukas Ruf <ru...@rawip.org>.
> Lukas Ruf <ru...@rawip.org> [2004-05-03 20:05]:
>
> > Ben Collins-Sussman <su...@collab.net> [2004-05-03 16:49]:
> >
> > >
> > > I know I should not do this. However, for a bunch of other
> > > repositories, there is no problem with that configuration.
> >
> > "There is no problem?" Allow me to quote the book:
> >
> > "If you attempt to use Berkeley DB on a network share, the results
> > are unpredictable???you may see mysterious errors right away, or
> > it may be months before you discover that your repository database
> > is subtly corrupted."
> >
> > So you're working at your own risk here. Good luck.
> >
>
I put all the repositories to a local hard-disk,
re-created the problematic repository from scratch,
tried a checkout via https,
killed the svn client by 'kill -9' (because I had to disconnect temporarely),
restarted the checkout later,
and experienced the same problem: Linux killed the client by abort (at
least I saw this message on screen).
Hence my questions:
- does 'killing a checkout process by -9' destroy the repository?
- if so what is the best way to stop a large checkout?
- svn has the atomic operations. Is it possible that it crashes due
to linux' limits because it saves also very large files/directories
first in memory (one directory contained approx. 1.5GB of data but I
have only 1GB RAM) before it writes the checkout to the disk?
I stumbled over this idea when I realized that after the abort there
were no data on disk.
If this was the case, how could I handle large directories/files
then?
Thanks for any hints!
wbr,
Lukas
--
Lukas Ruf | Wanna know anything about raw |
<http://www.lpr.ch> | IP? -> <http://www.rawip.org> |
eMail Style Guide: <http://www.rawip.org/style.html>|
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by Lukas Ruf <ru...@rawip.org>.
> Ben Collins-Sussman <su...@collab.net> [2004-05-03 16:49]:
>
> >
> > I know I should not do this. However, for a bunch of other
> > repositories, there is no problem with that configuration.
>
> "There is no problem?" Allow me to quote the book:
>
> "If you attempt to use Berkeley DB on a network share, the results are
> unpredictable???you may see mysterious errors right away, or it may be
> months before you discover that your repository database is subtly
> corrupted."
>
> So you're working at your own risk here. Good luck.
>
Thanks for the wishes ;-)
>
> >
> > I copied the repository in question to the local disk and tried to
> > check out the stuff via https as usual -- without success. svn
> > started growing... Fortunately for me, I still have the original data
> > laying around.
>
> Sorry for the confusion here: you're talking about the svn *client*
> program growing huge, right?
>
yes.
> If so, is it possible for one of us to reproduce the problem? In other
> words, can you give us a URL to checkout/update, so we can try to
> witness/debug the problem firsthand?
>
>
the problem with that is that the repository in question contains data
which are not completely public...
I will re-create the repository on a local drive and check it again.
wbr,
Lukas
--
Lukas Ruf | Wanna know anything about raw |
<http://www.lpr.ch> | IP? -> <http://www.rawip.org> |
eMail Style Guide: <http://www.rawip.org/style.html>|
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by Ben Collins-Sussman <su...@collab.net>.
On Sun, 2004-05-02 at 09:57, Lukas Ruf wrote:
> > Ben Collins-Sussman <su...@collab.net> [2004-04-30 18:19]:
> >
> > On Fri, 2004-04-30 at 01:47, Lukas Ruf wrote:
> >
> > > The repository is mounted via NFS (server Solaris 2.8)
> >
> > Yes, there are warnings all over creation not to do this (README, FAQ,
> > Book...) Don't do this. If you move the repository to local disk, are
> > you able to reproduce the memory leak? I'd like to know.
> >
>
> I know I should not do this. However, for a bunch of other
> repositories, there is no problem with that configuration.
"There is no problem?" Allow me to quote the book:
"If you attempt to use Berkeley DB on a network share, the results are
unpredictable—you may see mysterious errors right away, or it may be
months before you discover that your repository database is subtly
corrupted."
So you're working at your own risk here. Good luck.
>
> I copied the repository in question to the local disk and tried to
> check out the stuff via https as usual -- without success. svn
> started growing... Fortunately for me, I still have the original data
> laying around.
Sorry for the confusion here: you're talking about the svn *client*
program growing huge, right?
If so, is it possible for one of us to reproduce the problem? In other
words, can you give us a URL to checkout/update, so we can try to
witness/debug the problem firsthand?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by Lukas Ruf <ru...@rawip.org>.
> Ben Collins-Sussman <su...@collab.net> [2004-04-30 18:19]:
>
> On Fri, 2004-04-30 at 01:47, Lukas Ruf wrote:
>
> > The repository is mounted via NFS (server Solaris 2.8)
>
> Yes, there are warnings all over creation not to do this (README, FAQ,
> Book...) Don't do this. If you move the repository to local disk, are
> you able to reproduce the memory leak? I'd like to know.
>
I know I should not do this. However, for a bunch of other
repositories, there is no problem with that configuration.
I copied the repository in question to the local disk and tried to
check out the stuff via https as usual -- without success. svn
started growing... Fortunately for me, I still have the original data
laying around.
Is there any tool that I can run over the repository to evaluate/check
the internal structure?
wbr,
Lukas
--
Lukas Ruf | Wanna know anything about raw |
<http://www.lpr.ch> | IP? -> <http://www.rawip.org> |
eMail Style Guide: <http://www.rawip.org/style.html>|
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by Ben Collins-Sussman <su...@collab.net>.
On Fri, 2004-04-30 at 01:47, Lukas Ruf wrote:
> The repository is mounted via NFS (server Solaris 2.8)
Yes, there are warnings all over creation not to do this (README, FAQ,
Book...) Don't do this. If you move the repository to local disk, are
you able to reproduce the memory leak? I'd like to know.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by Lukas Ruf <ru...@rawip.org>.
> Ben Collins-Sussman <su...@collab.net> [2004-04-29 22:41]:
>
> On Thu, 2004-04-29 at 15:28, Lukas Ruf wrote:
>
> > However, svn does not proceed as I would expect but consumes
> > nearly all memory and ends without any notice.
>
> This is an extremely important bug: we have no known memory leaks
> at the moment. If you've found a new one, we'd like to identify it
> ASAP.
>
If I can be of any help, I am happy to do!
> >
> > top reports
> >
> > 10446 ruf 25 0 522m 370m 7460 R 99.5 41.9 241:51.21 svn
> >
> > I am using svn on a Linux 2.6.5 box (client & repository):
> >
> > svn, version 1.0.2 (r9423) compiled Apr 27 2004, 17:31:38
> >
> > apache 2.0.49-1
> >
> > debian unstable (latest updates installed).
> >
> > I access all the repository only via https.
> >
> > svn stops after approx. 620MB -- the repostiory (du -h) reports
> > 702MB.
>
> What command are you running? Checkout? Export? I'd can't quite
> understand your email.
svn update
>
> Can you explain to us how to reproduce the memory growth? I'd like
> to see the same bug on my own computer.
>
The very first checkout was interrupted by accident. Hence, I run
svn cleanup
as recommended by svn and then I have been trying to run
svn update
to avoid additional data fetching.
Usually, I retried in the evening by putting
nohup svn update >& update.log
in the background.
Whenever I checked 'top' I always saw an entry as the one posted
initially. Usually, in the morning, svn disappeared without any
further notice. The update.log file, however, has not reached its
end.
The repository contains some large archives (~500MB). Could this
create problems?
The repository is mounted via NFS (server Solaris 2.8) and accessed
only by one client (Debian unstable, Linux 2.6.5) via https only.
wbr,
Lukas
--
Lukas Ruf | Wanna know anything about raw |
<http://www.lpr.ch> | IP? -> <http://www.rawip.org> |
eMail Style Guide: <http://www.rawip.org/style.html>|
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn consumes all memory
Posted by Ben Collins-Sussman <su...@collab.net>.
On Thu, 2004-04-29 at 15:28, Lukas Ruf wrote:
> However, svn does not proceed as I would expect but consumes nearly
> all memory and ends without any notice.
This is an extremely important bug: we have no known memory leaks at
the moment. If you've found a new one, we'd like to identify it ASAP.
>
> top reports
>
> 10446 ruf 25 0 522m 370m 7460 R 99.5 41.9 241:51.21 svn
>
> I am using svn on a Linux 2.6.5 box (client & repository):
>
> svn, version 1.0.2 (r9423)
> compiled Apr 27 2004, 17:31:38
>
> apache 2.0.49-1
>
> debian unstable (latest updates installed).
>
> I access all the repository only via https.
>
> svn stops after approx. 620MB -- the repostiory (du -h) reports 702MB.
What command are you running? Checkout? Export? I'd can't quite
understand your email.
Can you explain to us how to reproduce the memory growth? I'd like to
see the same bug on my own computer.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org