You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Chris Tashjian <ct...@thepond.com> on 2010/11/05 00:59:24 UTC
"svn blame -g" causing svnserve to hang & mem usage to hit 2GB
I posted this on the users list, but I'm confident that this is a bug.
Background:
For a while now (off and on for over a year, but more frequently in the
last 6+ months) we've been having problems with svn "crashing", yet
there's no error in the log file. In talking to someone the users list
it sounds like svn is actually just hanging. Clients get the following
response:
svn: Can't connect to host 'svn': No connection could be made
because the target machine actively refused it.
Our repository has 129K revisions. The format is "4 layout linear", it
was created with svnadmin 1.4.x and has since had "svnadmin upgrade" run
both in 1.5 and 1.6. We're currently running SlikSVN 1.6.13 (Win32),
but I have previously had this problem dating back to versions of 1.5,
both stock and from CollabNet. The issue now happens numerous times per
day and it looks like I've discovered why....
As a test I ran "svn blame -g" on a file with a bunch of revisions and
watched memory usage on the server spin up to 2GB.
I restarted the server and then tried "svn blame" (no -g) on the same
file and it ran quickly, with no major up-tick in memory usage.
I tried it again with -g and it blew up to 2GB.
Next, (after restarting the server) I tried another file with more
revisions. "svn blame" spins memory usage to 45MB and then eventually
comes back with output.
"svn blame -g" on this same file, causes svnserve to quickly (in about
45 seconds) climb to 2GB of memory usage and doesn't come back (at least
after 5 minutes). At that point, I just killed svnserve.exe on the server.
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Paul Burba <pt...@gmail.com>.
On Mon, Nov 8, 2010 at 8:17 PM, Paul Burba <pt...@gmail.com> wrote:
> On Mon, Nov 8, 2010 at 1:23 PM, Paul Burba <pt...@gmail.com> wrote:
>> On Mon, Nov 8, 2010 at 10:35 AM, Paul Burba <pt...@gmail.com> wrote:
>>> On Mon, Nov 8, 2010 at 9:13 AM, Paul Burba <pt...@gmail.com> wrote:
>>>> On Thu, Nov 4, 2010 at 8:59 PM, Chris Tashjian <ct...@thepond.com> wrote:
>>>>> I posted this on the users list, but I'm confident that this is a bug.
>>>>>
>>>>> Background:
>>>>> For a while now (off and on for over a year, but more frequently in the last
>>>>> 6+ months) we've been having problems with svn "crashing", yet there's no
>>>>> error in the log file. In talking to someone the users list it sounds like
>>>>> svn is actually just hanging. Clients get the following response:
>>>>>
>>>>> svn: Can't connect to host 'svn': No connection could be made
>>>>> because the target machine actively refused it.
>>>>>
>>>>> Our repository has 129K revisions. The format is "4 layout linear", it was
>>>>> created with svnadmin 1.4.x and has since had "svnadmin upgrade" run both in
>>>>> 1.5 and 1.6. We're currently running SlikSVN 1.6.13 (Win32), but I have
>>>>> previously had this problem dating back to versions of 1.5, both stock and
>>>>> from CollabNet. The issue now happens numerous times per day and it looks
>>>>> like I've discovered why....
>>>>>
>>>>>
>>>>> As a test I ran "svn blame -g" on a file with a bunch of revisions and
>>>>> watched memory usage on the server spin up to 2GB.
>>>>
>>>> Chris,
>>>>
>>>> By a "bunch of revisions" what exactly do you mean? Many revisions
>>>> which were the result of a merge? Or simply many changes made
>>>> directly to the file (not the result of a merge)?
>>>>
>>>>> Paul - I'll see if I can get a test repo up with the error. In the
>>>>> meantime, would a copy of the svn:mergeinfo help?
>>>>
>>>> Any luck?
>>>
>>> Chris,
>>>
>>> Don't sweat the replication effort too much, I think I'm on the trail
>>> of this problem. Using a copy of the old (pre-ASF) Subversion
>>> repository I'm seeing unexpectedly high memory use when using log -g
>>> on a file with a "lot" of merge history:
>>>
>>> 1.6.13.client>svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py
>>> 25 MB Peak Working Set Memory svnserve.exe
>>>
>>> 1.6.13.client>svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g
>>> 291 MB Peak Working Set Memory svnserve.exe
>> ^^^
>> That should have been 691 MB! Sorry!
>>
>>>
>>> More soon...
>
> Hi Chris,
>
> Ok, I think I got it. Switching to a Subversion trunk@1032651 (debug)
> build and using this repository as a test*:
>
> http://svn.collab.net/tmp/subversion-data-backup/subversion-history-20091115.tar.bz2
>
> The peak working set memory of
>
> 'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py' was 21 MB
> 'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g' was 574 MB
>
> That's comparable to what you saw with 1.6.13.
>
> Applying some standard pool-fu to
> libsvn_repos/rev_hunt.c:(find_merged_revisions|find_interesting_revisions),
> see attached patch, and things look a *lot* better:
>
> The peak working set memory of
>
> 'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py' stays at 21 MB
> 'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g'
> drops to 71 MB
>
> A tidy 88% reduction in peak memory usage!
>
> Running the test suite on this patch. Assuming no problems I will be
> committing and nominating for backport to 1.6.x.
Tests pass, committed
http://svn.apache.org/viewvc?view=revision&revision=1032808
Paul
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Paul Burba <pt...@gmail.com>.
On Mon, Nov 8, 2010 at 8:17 PM, Paul Burba <pt...@gmail.com> wrote:
> On Mon, Nov 8, 2010 at 1:23 PM, Paul Burba <pt...@gmail.com> wrote:
>> On Mon, Nov 8, 2010 at 10:35 AM, Paul Burba <pt...@gmail.com> wrote:
>>> On Mon, Nov 8, 2010 at 9:13 AM, Paul Burba <pt...@gmail.com> wrote:
>>>> On Thu, Nov 4, 2010 at 8:59 PM, Chris Tashjian <ct...@thepond.com> wrote:
>>>>> I posted this on the users list, but I'm confident that this is a bug.
>>>>>
>>>>> Background:
>>>>> For a while now (off and on for over a year, but more frequently in the last
>>>>> 6+ months) we've been having problems with svn "crashing", yet there's no
>>>>> error in the log file. In talking to someone the users list it sounds like
>>>>> svn is actually just hanging. Clients get the following response:
>>>>>
>>>>> svn: Can't connect to host 'svn': No connection could be made
>>>>> because the target machine actively refused it.
>>>>>
>>>>> Our repository has 129K revisions. The format is "4 layout linear", it was
>>>>> created with svnadmin 1.4.x and has since had "svnadmin upgrade" run both in
>>>>> 1.5 and 1.6. We're currently running SlikSVN 1.6.13 (Win32), but I have
>>>>> previously had this problem dating back to versions of 1.5, both stock and
>>>>> from CollabNet. The issue now happens numerous times per day and it looks
>>>>> like I've discovered why....
>>>>>
>>>>>
>>>>> As a test I ran "svn blame -g" on a file with a bunch of revisions and
>>>>> watched memory usage on the server spin up to 2GB.
>>>>
>>>> Chris,
>>>>
>>>> By a "bunch of revisions" what exactly do you mean? Many revisions
>>>> which were the result of a merge? Or simply many changes made
>>>> directly to the file (not the result of a merge)?
>>>>
>>>>> Paul - I'll see if I can get a test repo up with the error. In the
>>>>> meantime, would a copy of the svn:mergeinfo help?
>>>>
>>>> Any luck?
>>>
>>> Chris,
>>>
>>> Don't sweat the replication effort too much, I think I'm on the trail
>>> of this problem. Using a copy of the old (pre-ASF) Subversion
>>> repository I'm seeing unexpectedly high memory use when using log -g
>>> on a file with a "lot" of merge history:
>>>
>>> 1.6.13.client>svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py
>>> 25 MB Peak Working Set Memory svnserve.exe
>>>
>>> 1.6.13.client>svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g
>>> 291 MB Peak Working Set Memory svnserve.exe
>> ^^^
>> That should have been 691 MB! Sorry!
>>
>>>
>>> More soon...
>
> Hi Chris,
>
> Ok, I think I got it. Switching to a Subversion trunk@1032651 (debug)
> build and using this repository as a test*:
>
> http://svn.collab.net/tmp/subversion-data-backup/subversion-history-20091115.tar.bz2
>
> The peak working set memory of
>
> 'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py' was 21 MB
> 'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g' was 574 MB
>
> That's comparable to what you saw with 1.6.13.
>
> Applying some standard pool-fu to
> libsvn_repos/rev_hunt.c:(find_merged_revisions|find_interesting_revisions),
> see attached patch, and things look a *lot* better:
>
> The peak working set memory of
>
> 'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py' stays at 21 MB
> 'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g'
> drops to 71 MB
>
> A tidy 88% reduction in peak memory usage!
>
> Running the test suite on this patch. Assuming no problems I will be
> committing and nominating for backport to 1.6.x.
Nominated r1032808 for backport to 1.6.14.
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Paul Burba <pt...@gmail.com>.
On Mon, Nov 8, 2010 at 10:01 PM, Paul Burba <pt...@gmail.com> wrote:
> On Mon, Nov 8, 2010 at 9:09 PM, Chris Tashjian <ct...@thepond.com> wrote:
>>
>> On 11/8/2010 8:54 PM, Paul Burba wrote:
>>>
>>> On Mon, Nov 8, 2010 at 8:28 PM, Chris Tashjian<ct...@thepond.com> wrote:
>>>>
>>>> Paul - 1 more question. What happens when you run additional blame -g
>>>> commands? Does the memory usage keep growing until it runs out of memory
>>>> or
>>>> will it cap itself at some point? In my tests it seems like memory
>>>> wasn't
>>>> getting released very often. Granted, memory usage spun up so fast, I'm
>>>> not
>>>> positive that it really had much of a chance for that to happen.
>>>
>>> Chris,
>>>
>>> I was describing *peak* memory use. The actual working set memory
>>> drops from 71 MB to 7 MB once the blame -g is processed in my example.
>>> Not sure how familiar you are with APR pools, but all the pools that
>>> get allocated to process the request should be destroyed (freed) when
>>> the server is done. If memory use is growing slowly after each blame
>>> -g, then that isn't happening and we have a separate bug. I just did
>>> a little ad hoc testing and there doesn't appear to be a problem.
>>> After several blame -g hits, the peak working set for my svnserve.exe
>>> process always returns to around 7 MB:
>>>
>>> 7464 K
>>> 7652 K
>>> 7500 K
>>> 7596 K
>>>
>>> If you discover anything contrary to this please let us know on the dev
>>> list.
>>
>> I just restarted our svn server and ran blame -g against a file. No one
>> else is hitting the server right now and after 5 minutes mem usage is at
>> 681,208K and the peak was 681,316K. (taskmgr screenshot:
>> http://img215.imageshack.us/img215/2060/svnserve.jpg)
>
> I just checked with a trunk build prior to r1032808 it while the
> memory leak manifests itself while processing the blame -g request
> (peak memory at 571 MB), the peak working set returns to 9 MB once the
> blame is processed. So it's not a problem on trunk.
>
> I do see what you are describing on 1.6.x though. Working memory
> peaks at 591 MB and stays there! With r1032808 memory only peaks at
> 91 MB, but stay there...so yup there is another bug. But it will have
> to wait till tomorrow for me :-)
Hi Chris,
I think we've reached the end of the road with this (in a good way :-)
The problem with svnserve not freeing memory was fixed on
^/subversion/branches/performance by Stefan Fuhrmann a few months
back:
http://svn.apache.org/viewvc?view=revision&revision=982355
And cherry-picked over to trunk a few weeks ago:
http://svn.apache.org/viewvc?view=revision&revision=1022675
With an unmodified 1.6.x@1033049 my blame -g example results in:
svnserve.exe starting working set memory 6,456 K
svnserve.exe peak working set memory 617,696 K
svnserve.exe ending working set memory 617,628 K
With r1022675 merged to 1.6.x my blame -g example results in:
svnserve.exe starting working set memory 6,572 K
svnserve.exe peak working set memory 616,996 K
svnserve.exe ending working set memory 11,364K
With r1022675 and r1032808 both merged to 1.6.x:
svnserve.exe starting working set memory 6,500 K
svnserve.exe peak working set memory 92,572 K
svnserve.exe ending working set memory 8,780 K
I nominated r1022675 for backport to 1.6.14. Now we just have to
await review/votes/vetoes.
Paul
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Paul Burba <pt...@gmail.com>.
On Mon, Nov 8, 2010 at 9:09 PM, Chris Tashjian <ct...@thepond.com> wrote:
>
> On 11/8/2010 8:54 PM, Paul Burba wrote:
>>
>> On Mon, Nov 8, 2010 at 8:28 PM, Chris Tashjian<ct...@thepond.com> wrote:
>>>
>>> Paul - 1 more question. What happens when you run additional blame -g
>>> commands? Does the memory usage keep growing until it runs out of memory
>>> or
>>> will it cap itself at some point? In my tests it seems like memory
>>> wasn't
>>> getting released very often. Granted, memory usage spun up so fast, I'm
>>> not
>>> positive that it really had much of a chance for that to happen.
>>
>> Chris,
>>
>> I was describing *peak* memory use. The actual working set memory
>> drops from 71 MB to 7 MB once the blame -g is processed in my example.
>> Not sure how familiar you are with APR pools, but all the pools that
>> get allocated to process the request should be destroyed (freed) when
>> the server is done. If memory use is growing slowly after each blame
>> -g, then that isn't happening and we have a separate bug. I just did
>> a little ad hoc testing and there doesn't appear to be a problem.
>> After several blame -g hits, the peak working set for my svnserve.exe
>> process always returns to around 7 MB:
>>
>> 7464 K
>> 7652 K
>> 7500 K
>> 7596 K
>>
>> If you discover anything contrary to this please let us know on the dev
>> list.
>
> I just restarted our svn server and ran blame -g against a file. No one
> else is hitting the server right now and after 5 minutes mem usage is at
> 681,208K and the peak was 681,316K. (taskmgr screenshot:
> http://img215.imageshack.us/img215/2060/svnserve.jpg)
I just checked with a trunk build prior to r1032808 it while the
memory leak manifests itself while processing the blame -g request
(peak memory at 571 MB), the peak working set returns to 9 MB once the
blame is processed. So it's not a problem on trunk.
I do see what you are describing on 1.6.x though. Working memory
peaks at 591 MB and stays there! With r1032808 memory only peaks at
91 MB, but stay there...so yup there is another bug. But it will have
to wait till tomorrow for me :-)
Paul
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Chris Tashjian <ct...@thepond.com>.
On 11/8/2010 8:54 PM, Paul Burba wrote:
> On Mon, Nov 8, 2010 at 8:28 PM, Chris Tashjian<ct...@thepond.com> wrote:
>> Paul - 1 more question. What happens when you run additional blame -g
>> commands? Does the memory usage keep growing until it runs out of memory or
>> will it cap itself at some point? In my tests it seems like memory wasn't
>> getting released very often. Granted, memory usage spun up so fast, I'm not
>> positive that it really had much of a chance for that to happen.
> Chris,
>
> I was describing *peak* memory use. The actual working set memory
> drops from 71 MB to 7 MB once the blame -g is processed in my example.
> Not sure how familiar you are with APR pools, but all the pools that
> get allocated to process the request should be destroyed (freed) when
> the server is done. If memory use is growing slowly after each blame
> -g, then that isn't happening and we have a separate bug. I just did
> a little ad hoc testing and there doesn't appear to be a problem.
> After several blame -g hits, the peak working set for my svnserve.exe
> process always returns to around 7 MB:
>
> 7464 K
> 7652 K
> 7500 K
> 7596 K
>
> If you discover anything contrary to this please let us know on the dev list.
I just restarted our svn server and ran blame -g against a file. No one
else is hitting the server right now and after 5 minutes mem usage is at
681,208K and the peak was 681,316K. (taskmgr screenshot:
http://img215.imageshack.us/img215/2060/svnserve.jpg)
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Paul Burba <pt...@gmail.com>.
On Mon, Nov 8, 2010 at 1:23 PM, Paul Burba <pt...@gmail.com> wrote:
> On Mon, Nov 8, 2010 at 10:35 AM, Paul Burba <pt...@gmail.com> wrote:
>> On Mon, Nov 8, 2010 at 9:13 AM, Paul Burba <pt...@gmail.com> wrote:
>>> On Thu, Nov 4, 2010 at 8:59 PM, Chris Tashjian <ct...@thepond.com> wrote:
>>>> I posted this on the users list, but I'm confident that this is a bug.
>>>>
>>>> Background:
>>>> For a while now (off and on for over a year, but more frequently in the last
>>>> 6+ months) we've been having problems with svn "crashing", yet there's no
>>>> error in the log file. In talking to someone the users list it sounds like
>>>> svn is actually just hanging. Clients get the following response:
>>>>
>>>> svn: Can't connect to host 'svn': No connection could be made
>>>> because the target machine actively refused it.
>>>>
>>>> Our repository has 129K revisions. The format is "4 layout linear", it was
>>>> created with svnadmin 1.4.x and has since had "svnadmin upgrade" run both in
>>>> 1.5 and 1.6. We're currently running SlikSVN 1.6.13 (Win32), but I have
>>>> previously had this problem dating back to versions of 1.5, both stock and
>>>> from CollabNet. The issue now happens numerous times per day and it looks
>>>> like I've discovered why....
>>>>
>>>>
>>>> As a test I ran "svn blame -g" on a file with a bunch of revisions and
>>>> watched memory usage on the server spin up to 2GB.
>>>
>>> Chris,
>>>
>>> By a "bunch of revisions" what exactly do you mean? Many revisions
>>> which were the result of a merge? Or simply many changes made
>>> directly to the file (not the result of a merge)?
>>>
>>>> Paul - I'll see if I can get a test repo up with the error. In the
>>>> meantime, would a copy of the svn:mergeinfo help?
>>>
>>> Any luck?
>>
>> Chris,
>>
>> Don't sweat the replication effort too much, I think I'm on the trail
>> of this problem. Using a copy of the old (pre-ASF) Subversion
>> repository I'm seeing unexpectedly high memory use when using log -g
>> on a file with a "lot" of merge history:
>>
>> 1.6.13.client>svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py
>> 25 MB Peak Working Set Memory svnserve.exe
>>
>> 1.6.13.client>svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g
>> 291 MB Peak Working Set Memory svnserve.exe
> ^^^
> That should have been 691 MB! Sorry!
>
>>
>> More soon...
Hi Chris,
Ok, I think I got it. Switching to a Subversion trunk@1032651 (debug)
build and using this repository as a test*:
http://svn.collab.net/tmp/subversion-data-backup/subversion-history-20091115.tar.bz2
The peak working set memory of
'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py' was 21 MB
'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g' was 574 MB
That's comparable to what you saw with 1.6.13.
Applying some standard pool-fu to
libsvn_repos/rev_hunt.c:(find_merged_revisions|find_interesting_revisions),
see attached patch, and things look a *lot* better:
The peak working set memory of
'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py' stays at 21 MB
'svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g'
drops to 71 MB
A tidy 88% reduction in peak memory usage!
Running the test suite on this patch. Assuming no problems I will be
committing and nominating for backport to 1.6.x.
log:
[[[
Fix blame -g server-side memory leaks.
See http://svn.haxx.se/dev/archive-2010-11/0102.shtml
* subversion/libsvn_repos/rev_hunt.c
(find_interesting_revisions): Implement result/scratch two-pool paradigm.
Don't needlessly allocate path_revision structures until we are sure
we are going to keep it. Rename local variable "iter_pool" to the
de facto standard "iterpool".
(find_merged_revisions): Use a separate iterpool for each nested loop.
Update call to find_interesting_revisions, passing, you guessed it, an
iterpool. Rename local variable "iter_pool" to the de facto standard
"iterpool".
]]]
Paul
* This is the old Tigris Subversion repository, see
http://svn.apache.org/repos/asf/subversion/README.
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Paul Burba <pt...@gmail.com>.
On Mon, Nov 8, 2010 at 10:35 AM, Paul Burba <pt...@gmail.com> wrote:
> On Mon, Nov 8, 2010 at 9:13 AM, Paul Burba <pt...@gmail.com> wrote:
>> On Thu, Nov 4, 2010 at 8:59 PM, Chris Tashjian <ct...@thepond.com> wrote:
>>> I posted this on the users list, but I'm confident that this is a bug.
>>>
>>> Background:
>>> For a while now (off and on for over a year, but more frequently in the last
>>> 6+ months) we've been having problems with svn "crashing", yet there's no
>>> error in the log file. In talking to someone the users list it sounds like
>>> svn is actually just hanging. Clients get the following response:
>>>
>>> svn: Can't connect to host 'svn': No connection could be made
>>> because the target machine actively refused it.
>>>
>>> Our repository has 129K revisions. The format is "4 layout linear", it was
>>> created with svnadmin 1.4.x and has since had "svnadmin upgrade" run both in
>>> 1.5 and 1.6. We're currently running SlikSVN 1.6.13 (Win32), but I have
>>> previously had this problem dating back to versions of 1.5, both stock and
>>> from CollabNet. The issue now happens numerous times per day and it looks
>>> like I've discovered why....
>>>
>>>
>>> As a test I ran "svn blame -g" on a file with a bunch of revisions and
>>> watched memory usage on the server spin up to 2GB.
>>
>> Chris,
>>
>> By a "bunch of revisions" what exactly do you mean? Many revisions
>> which were the result of a merge? Or simply many changes made
>> directly to the file (not the result of a merge)?
>>
>>> Paul - I'll see if I can get a test repo up with the error. In the
>>> meantime, would a copy of the svn:mergeinfo help?
>>
>> Any luck?
>
> Chris,
>
> Don't sweat the replication effort too much, I think I'm on the trail
> of this problem. Using a copy of the old (pre-ASF) Subversion
> repository I'm seeing unexpectedly high memory use when using log -g
> on a file with a "lot" of merge history:
>
> 1.6.13.client>svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py
> 25 MB Peak Working Set Memory svnserve.exe
>
> 1.6.13.client>svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g
> 291 MB Peak Working Set Memory svnserve.exe
^^^
That should have been 691 MB! Sorry!
>
> More soon...
>
> Paul
>
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Paul Burba <pt...@gmail.com>.
On Mon, Nov 8, 2010 at 9:13 AM, Paul Burba <pt...@gmail.com> wrote:
> On Thu, Nov 4, 2010 at 8:59 PM, Chris Tashjian <ct...@thepond.com> wrote:
>> I posted this on the users list, but I'm confident that this is a bug.
>>
>> Background:
>> For a while now (off and on for over a year, but more frequently in the last
>> 6+ months) we've been having problems with svn "crashing", yet there's no
>> error in the log file. In talking to someone the users list it sounds like
>> svn is actually just hanging. Clients get the following response:
>>
>> svn: Can't connect to host 'svn': No connection could be made
>> because the target machine actively refused it.
>>
>> Our repository has 129K revisions. The format is "4 layout linear", it was
>> created with svnadmin 1.4.x and has since had "svnadmin upgrade" run both in
>> 1.5 and 1.6. We're currently running SlikSVN 1.6.13 (Win32), but I have
>> previously had this problem dating back to versions of 1.5, both stock and
>> from CollabNet. The issue now happens numerous times per day and it looks
>> like I've discovered why....
>>
>>
>> As a test I ran "svn blame -g" on a file with a bunch of revisions and
>> watched memory usage on the server spin up to 2GB.
>
> Chris,
>
> By a "bunch of revisions" what exactly do you mean? Many revisions
> which were the result of a merge? Or simply many changes made
> directly to the file (not the result of a merge)?
>
>> Paul - I'll see if I can get a test repo up with the error. In the
>> meantime, would a copy of the svn:mergeinfo help?
>
> Any luck?
Chris,
Don't sweat the replication effort too much, I think I'm on the trail
of this problem. Using a copy of the old (pre-ASF) Subversion
repository I'm seeing unexpectedly high memory use when using log -g
on a file with a "lot" of merge history:
1.6.13.client>svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py
25 MB Peak Working Set Memory svnserve.exe
1.6.13.client>svn blame ^/trunk/subversion/tests/cmdline/merge_tests.py -g
291 MB Peak Working Set Memory svnserve.exe
More soon...
Paul
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Paul Burba <pt...@gmail.com>.
On Thu, Nov 4, 2010 at 8:59 PM, Chris Tashjian <ct...@thepond.com> wrote:
> I posted this on the users list, but I'm confident that this is a bug.
>
> Background:
> For a while now (off and on for over a year, but more frequently in the last
> 6+ months) we've been having problems with svn "crashing", yet there's no
> error in the log file. In talking to someone the users list it sounds like
> svn is actually just hanging. Clients get the following response:
>
> svn: Can't connect to host 'svn': No connection could be made
> because the target machine actively refused it.
>
> Our repository has 129K revisions. The format is "4 layout linear", it was
> created with svnadmin 1.4.x and has since had "svnadmin upgrade" run both in
> 1.5 and 1.6. We're currently running SlikSVN 1.6.13 (Win32), but I have
> previously had this problem dating back to versions of 1.5, both stock and
> from CollabNet. The issue now happens numerous times per day and it looks
> like I've discovered why....
>
>
> As a test I ran "svn blame -g" on a file with a bunch of revisions and
> watched memory usage on the server spin up to 2GB.
Chris,
By a "bunch of revisions" what exactly do you mean? Many revisions
which were the result of a merge? Or simply many changes made
directly to the file (not the result of a merge)?
> Paul - I'll see if I can get a test repo up with the error. In the
> meantime, would a copy of the svn:mergeinfo help?
Any luck? I should have asked this out of the gate (though I'm sure I
know the answer): Is your repos public? Obviously the most direct
route to replicating this would be with a copy of your actual data :-)
Paul
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Philip Martin <ph...@wandisco.com>.
Paul Burba <pt...@gmail.com> writes:
> I recall a similar issue
> http://subversion.tigris.org/issues/show_bug.cgi?id=3397.
> Unfortunately nothing conclusive came of that.
There is also
http://subversion.tigris.org/issues/show_bug.cgi?id=3593
which I suspect is the in-memory-caching added to FSFS in 1.6. I guess
blame could be triggering the same usage as dump.
--
Philip
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Paul Burba <pt...@gmail.com>.
On Fri, Nov 5, 2010 at 2:54 PM, Chris Tashjian <ct...@thepond.com> wrote:
> Paul - I'll see if I can get a test repo up with the error. In the
> meantime, would a copy of the svn:mergeinfo help?
No harm in sending it along, though barring something very odd I don't
hold out much hope it's going to tell us much.
> On 11/5/2010 9:59 AM, Paul Burba wrote:
>>
>> Chris,
>>
>> I recall a similar issue
>> http://subversion.tigris.org/issues/show_bug.cgi?id=3397.
>> Unfortunately nothing conclusive came of that.
>>
>> When I wrap up what I am working on now I will take a look at this.
>> In the meantime, any chance you could try to reproduce the problem
>> using a simple test repository? No worries if you can't, but
>> obviously being able to reproduce the problem on our side would be
>> very helpful.
>>
>> Paul
>>
>> On Thu, Nov 4, 2010 at 8:59 PM, Chris Tashjian<ct...@thepond.com> wrote:
>>>
>>> I posted this on the users list, but I'm confident that this is a bug.
>>>
>>> Background:
>>> For a while now (off and on for over a year, but more frequently in the
>>> last
>>> 6+ months) we've been having problems with svn "crashing", yet there's no
>>> error in the log file. In talking to someone the users list it sounds
>>> like
>>> svn is actually just hanging. Clients get the following response:
>>>
>>> svn: Can't connect to host 'svn': No connection could be made
>>> because the target machine actively refused it.
>>>
>>> Our repository has 129K revisions. The format is "4 layout linear", it
>>> was
>>> created with svnadmin 1.4.x and has since had "svnadmin upgrade" run both
>>> in
>>> 1.5 and 1.6. We're currently running SlikSVN 1.6.13 (Win32), but I have
>>> previously had this problem dating back to versions of 1.5, both stock
>>> and
>>> from CollabNet. The issue now happens numerous times per day and it
>>> looks
>>> like I've discovered why....
>>>
>>>
>>> As a test I ran "svn blame -g" on a file with a bunch of revisions and
>>> watched memory usage on the server spin up to 2GB.
>>>
>>> I restarted the server and then tried "svn blame" (no -g) on the same
>>> file
>>> and it ran quickly, with no major up-tick in memory usage.
>>>
>>> I tried it again with -g and it blew up to 2GB.
>>>
>>> Next, (after restarting the server) I tried another file with more
>>> revisions. "svn blame" spins memory usage to 45MB and then eventually
>>> comes
>>> back with output.
>>>
>>> "svn blame -g" on this same file, causes svnserve to quickly (in about 45
>>> seconds) climb to 2GB of memory usage and doesn't come back (at least
>>> after
>>> 5 minutes). At that point, I just killed svnserve.exe on the server.
>>>
>
Re: "svn blame -g" causing svnserve to hang & mem usage to hit 2GB
Posted by Paul Burba <pt...@gmail.com>.
Chris,
I recall a similar issue
http://subversion.tigris.org/issues/show_bug.cgi?id=3397.
Unfortunately nothing conclusive came of that.
When I wrap up what I am working on now I will take a look at this.
In the meantime, any chance you could try to reproduce the problem
using a simple test repository? No worries if you can't, but
obviously being able to reproduce the problem on our side would be
very helpful.
Paul
On Thu, Nov 4, 2010 at 8:59 PM, Chris Tashjian <ct...@thepond.com> wrote:
> I posted this on the users list, but I'm confident that this is a bug.
>
> Background:
> For a while now (off and on for over a year, but more frequently in the last
> 6+ months) we've been having problems with svn "crashing", yet there's no
> error in the log file. In talking to someone the users list it sounds like
> svn is actually just hanging. Clients get the following response:
>
> svn: Can't connect to host 'svn': No connection could be made
> because the target machine actively refused it.
>
> Our repository has 129K revisions. The format is "4 layout linear", it was
> created with svnadmin 1.4.x and has since had "svnadmin upgrade" run both in
> 1.5 and 1.6. We're currently running SlikSVN 1.6.13 (Win32), but I have
> previously had this problem dating back to versions of 1.5, both stock and
> from CollabNet. The issue now happens numerous times per day and it looks
> like I've discovered why....
>
>
> As a test I ran "svn blame -g" on a file with a bunch of revisions and
> watched memory usage on the server spin up to 2GB.
>
> I restarted the server and then tried "svn blame" (no -g) on the same file
> and it ran quickly, with no major up-tick in memory usage.
>
> I tried it again with -g and it blew up to 2GB.
>
> Next, (after restarting the server) I tried another file with more
> revisions. "svn blame" spins memory usage to 45MB and then eventually comes
> back with output.
>
> "svn blame -g" on this same file, causes svnserve to quickly (in about 45
> seconds) climb to 2GB of memory usage and doesn't come back (at least after
> 5 minutes). At that point, I just killed svnserve.exe on the server.
>