You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Stefan Küng <to...@gmail.com> on 2008/06/14 15:53:44 UTC
merge very slow
Hi,
I've noticed that a merge done with a client built with VS2008 is twice
as slow as a client built with VC6. Has anyone here noticed that too?
I've described my efforts to solve this problem here:
http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
If anyone has any idea what could cause this, I'd really appreciate any
help.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Stefan Küng <to...@gmail.com>.
Mark Phippard wrote:
> On Sat, Jun 14, 2008 at 11:53 AM, Stefan Küng <to...@gmail.com> wrote:
>
>> I've noticed that a merge done with a client built with VS2008 is twice as
>> slow as a client built with VC6. Has anyone here noticed that too?
>> I've described my efforts to solve this problem here:
>> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>>
>> If anyone has any idea what could cause this, I'd really appreciate any
>> help.
>
> Is there any way to profile it? I see the times in the packets was
> different, do you think the problem is in the networking layer? I had
> been debating switching our CollabNet builds to using VS 2008. I
> guess I'll hold off on that.
VS2008 Prof doesn't come with a profiler. And I don't have the Team
System edition.
Also, I don't think it would help much: I can't build and profile with
VC6 (I don't have VC6) to compare.
> You are using the same APR? I believe we are using 1.2.12 and Neon 0.28.2.
I'm using apr 1.2.12 and neon 0.28.2, so that's the same.
> Have you tried using any different servers or protocols?
I can't see any difference when using file:///, and I don't have an
svn:// repo available to test there.
The strange thing is: other commands are not slower (checkout, update,
commit), at least not much slower to really notice. So what is different
in a merge (network wise)?
Just ran the tests on XP (instead of Vista, where I did my tests
before): it's the same there too.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Stefan Küng <to...@gmail.com>.
On Sat, Jun 14, 2008 at 19:27, Stefan Küng <to...@gmail.com> wrote:
> Mark Phippard wrote:
>>
>> On Sat, Jun 14, 2008 at 12:51 PM, Mark Phippard <ma...@gmail.com>
>> wrote:
>>
>> So I ran the same merge commands from boxes in the CollabNet network,
>> using the same working copy (which is stored in my HOME folder on
>> Samba server). Here are the times:
>>
>> Linux: 10 seconds
>> Windows CSVN: 72 - 97 seconds
>> Windows TSVN: 97 - 101 seconds
>>
>> I do not think the information really helps, but thought I would share
>> it nonetheless.
>>
>> Since the time between Linux and Windows is so astonishing, I thought
>> I would throw in the times for svn status --ignore-externals on this
>> working copy:
>>
>> Linux: 0.3 seconds
>> Windows CSVN: 80 seconds
>> Windows TSVN: 85 seconds
>>
>
> Yikes!
> And I was wondering why nobody mentioned the slow merges before :)
>
> I stepped through the svn code for mergin a little bit. Most commands only
> connect once and then use that connection. However, the merge command does
> multiple 'connect()' calls, which I think is one reason why merging is so
> slow.
> I'm now doing some tests with svn:// - it seems that having IPv6 enabled
> *has* some impact on speed. Currently rebuilding without IPv6 support for
> more tests with svn://
Disabling IPv6 improved the merge time to the same as with the RC9
build from open.collab.net.
But it had no effect on a merge via http:// :(
Oh, and even using serf instead of neon doesn't help: my build takes
double the time for a merge.
What I've noticed while stepping through the code:
the ra_svn library exchanges the capabilities with the server only
once, but ra_neon does this for *every* connection made - and for a
merge, there are many connections done. So that is also an important
speed factor (svn vs http), but of course has nothing to do with my
problem here.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Stefan Küng <to...@gmail.com>.
Mark Phippard wrote:
> On Sat, Jun 14, 2008 at 12:51 PM, Mark Phippard <ma...@gmail.com> wrote:
>
> So I ran the same merge commands from boxes in the CollabNet network,
> using the same working copy (which is stored in my HOME folder on
> Samba server). Here are the times:
>
> Linux: 10 seconds
> Windows CSVN: 72 - 97 seconds
> Windows TSVN: 97 - 101 seconds
>
> I do not think the information really helps, but thought I would share
> it nonetheless.
>
> Since the time between Linux and Windows is so astonishing, I thought
> I would throw in the times for svn status --ignore-externals on this
> working copy:
>
> Linux: 0.3 seconds
> Windows CSVN: 80 seconds
> Windows TSVN: 85 seconds
>
Yikes!
And I was wondering why nobody mentioned the slow merges before :)
I stepped through the svn code for mergin a little bit. Most commands
only connect once and then use that connection. However, the merge
command does multiple 'connect()' calls, which I think is one reason why
merging is so slow.
I'm now doing some tests with svn:// - it seems that having IPv6 enabled
*has* some impact on speed. Currently rebuilding without IPv6 support
for more tests with svn://
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Mark Phippard <ma...@gmail.com>.
On Sat, Jun 14, 2008 at 12:51 PM, Mark Phippard <ma...@gmail.com> wrote:
So I ran the same merge commands from boxes in the CollabNet network,
using the same working copy (which is stored in my HOME folder on
Samba server). Here are the times:
Linux: 10 seconds
Windows CSVN: 72 - 97 seconds
Windows TSVN: 97 - 101 seconds
I do not think the information really helps, but thought I would share
it nonetheless.
Since the time between Linux and Windows is so astonishing, I thought
I would throw in the times for svn status --ignore-externals on this
working copy:
Linux: 0.3 seconds
Windows CSVN: 80 seconds
Windows TSVN: 85 seconds
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: merge very slow
Posted by Mark Phippard <ma...@gmail.com>.
On Sat, Jun 14, 2008 at 12:48 PM, Stefan Küng <to...@gmail.com> wrote:
> Mark Phippard wrote:
>>
>> On Sat, Jun 14, 2008 at 11:53 AM, Stefan Küng <to...@gmail.com>
>> wrote:
>>
>>> I've noticed that a merge done with a client built with VS2008 is twice
>>> as
>>> slow as a client built with VC6. Has anyone here noticed that too?
>>> I've described my efforts to solve this problem here:
>>> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>>>
>>> If anyone has any idea what could cause this, I'd really appreciate any
>>> help.
>>
>> Is there any way to profile it? I see the times in the packets was
>> different, do you think the problem is in the networking layer? I had
>> been debating switching our CollabNet builds to using VS 2008. I
>> guess I'll hold off on that.
>>
>> You are using the same APR? I believe we are using 1.2.12 and Neon
>> 0.28.2.
>>
>> Have you tried using any different servers or protocols?
>
> Update: I've tried a big merge with svn:// on a test repository running
> svnserve on my local machine. The RC9 from open.collab.net timed this at 4.5
> seconds, my build timed it at 13 seconds.
> So it's the very same with svn:// than with http://
>
> I can't see a difference when using file:///, but that may simply be because
> it's much much faster anyway.
That still sounds like the networking is involved though. That said,
I do not see why other commands would not show the same.
Are you interested in seeing my timings for the same commands? I am
in the process of installing the latest versions so I can test. On
Linux, this is what I got though:
$ time svn merge --dry-run --accept postpone -r11761:11762
http://tortoisesvn.tigris.org/svn/tortoisesvn/branches/1.4.x
tortoisesvn
real 0m9.895s
user 0m1.539s
sys 0m0.416s
I did that from a box inside the CollabNet network though, so it was
essentially contacting Tigris via the LAN.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: merge very slow
Posted by Stefan Küng <to...@gmail.com>.
Mark Phippard wrote:
> On Sat, Jun 14, 2008 at 11:53 AM, Stefan Küng <to...@gmail.com> wrote:
>
>> I've noticed that a merge done with a client built with VS2008 is twice as
>> slow as a client built with VC6. Has anyone here noticed that too?
>> I've described my efforts to solve this problem here:
>> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>>
>> If anyone has any idea what could cause this, I'd really appreciate any
>> help.
>
> Is there any way to profile it? I see the times in the packets was
> different, do you think the problem is in the networking layer? I had
> been debating switching our CollabNet builds to using VS 2008. I
> guess I'll hold off on that.
>
> You are using the same APR? I believe we are using 1.2.12 and Neon 0.28.2.
>
> Have you tried using any different servers or protocols?
Update: I've tried a big merge with svn:// on a test repository running
svnserve on my local machine. The RC9 from open.collab.net timed this at
4.5 seconds, my build timed it at 13 seconds.
So it's the very same with svn:// than with http://
I can't see a difference when using file:///, but that may simply be
because it's much much faster anyway.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Mark Phippard <ma...@gmail.com>.
On Sat, Jun 14, 2008 at 11:53 AM, Stefan Küng <to...@gmail.com> wrote:
> I've noticed that a merge done with a client built with VS2008 is twice as
> slow as a client built with VC6. Has anyone here noticed that too?
> I've described my efforts to solve this problem here:
> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>
> If anyone has any idea what could cause this, I'd really appreciate any
> help.
Is there any way to profile it? I see the times in the packets was
different, do you think the problem is in the networking layer? I had
been debating switching our CollabNet builds to using VS 2008. I
guess I'll hold off on that.
You are using the same APR? I believe we are using 1.2.12 and Neon 0.28.2.
Have you tried using any different servers or protocols?
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: merge very slow
Posted by Jens Peters <jp...@gmx.de>.
Stefan Küng wrote:
> Seems I have to patch neon in my build to get the speed up to par with
> a VC6 build.
>
> Stefan
Oh men, the bottleneck on my system was not svn but my firewall on my
somewhat older server machine. I do use Kerio Winroute firewall with
some very safe, but apparently slow settings ;).
Now I have similar timings:
VC 2008, x64 release:
unpatched neon: 4:16
patched neon (As you wrote on svn dev list): 2:04
VC6 (rc9 from collab.net)
2:06
Thus, case more or less solved (congrats!), at least about the
differences between VC6 and VC2008.
With my very limited c knowledge I would say the pre-proc. directive is
not correct.
#if defined(TCP_NODELAY) && defined(HAVE_SETSOCKOPT) && defined(IPPROTO_TCP)
if stated like above even the VS syntax highlighting says that the
condition is false. If stated like:
#if defined(TCP_NODELAY) && defined(HAVE_SETSOCKOPT) //&&
defined(IPPROTO_TCP)
then its true (according to VS). So it seems that defined does not like
enum values directly. (But i guess you already know that ;))
Regards,
Jens
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: merge very slow
Posted by Stefan Küng <to...@gmail.com>.
On Sun, Jun 15, 2008 at 17:19, Stefan Küng <to...@gmail.com> wrote:
> While doing some tests with the VS2005 build I rememberd that the
> devpartner community edition profiler works with VS2005 (it does not
> with VS2008). So I installed it and ran the test merge with profiling
> enabled:
>
> 84% of the time, the svn.exe spends in raw_poll() in file ne_socket
> (neon). And in that function, the most time is spent with
> ret = select(fdno + 1, &rdfds, &wrfds, NULL, tvp);
>
> The time spent there is around 4 minutes 17 seconds. Considering the
> whole merge takes about a minute more, I'd say that this is the
> problem.
>
> Now does anyone know why a select call would take so much time? Does
> that mean the data does not arrive as fast as expected? Or that it
> waits for more data (e.g., to fill a buffer) and then times out and
> returns with the data it already has? Or could it mean that the *send*
> of the data does not immediately send if some buffer is not full yet?
>
> I'm currently rebuilding everything with VS2008 and I'll do some more
> tests once that's done.
As it turned out, it was a problem with sending.
In neon, ne_socket.c, line 1229 is an
#if defined
statement which would set the TCP_NODELAY option for send sockets. But
for some reason I still haven't figured out completely, that code is
not compiled in. Even though all three conditions are set.
I guess when building with VC6, the old windows headers are used which
have those defined differently. I don't have those headers available
to compare though. One thing I noticed is that IPPROTO_TCP is not a
define but an enum in my windows headers.
So I removed the
#if defined
line from ne_socket.c and compiled again. The fact that simply
removing those lines and not getting any compiler errors clearly shows
that all conditions for the ifdef are met.
The time for my test merge went from more than five minutes to around
2 and a half minute, so I'm not at the same (still very slow) speed as
the RC9 from open.collab.net.
Seems I have to patch neon in my build to get the speed up to par with
a VC6 build.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Stefan Küng <to...@gmail.com>.
On Sun, Jun 15, 2008 at 15:39, Stefan Küng <to...@gmail.com> wrote:
> Update: the merge is the same when built with VS2005 than with VS2008.
While doing some tests with the VS2005 build I rememberd that the
devpartner community edition profiler works with VS2005 (it does not
with VS2008). So I installed it and ran the test merge with profiling
enabled:
84% of the time, the svn.exe spends in raw_poll() in file ne_socket
(neon). And in that function, the most time is spent with
ret = select(fdno + 1, &rdfds, &wrfds, NULL, tvp);
The time spent there is around 4 minutes 17 seconds. Considering the
whole merge takes about a minute more, I'd say that this is the
problem.
Now does anyone know why a select call would take so much time? Does
that mean the data does not arrive as fast as expected? Or that it
waits for more data (e.g., to fill a buffer) and then times out and
returns with the data it already has? Or could it mean that the *send*
of the data does not immediately send if some buffer is not full yet?
I'm currently rebuilding everything with VS2008 and I'll do some more
tests once that's done.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Stefan Küng <to...@gmail.com>.
Mark Phippard wrote:
> Merge in 1.5 has to crawl the working copy, and I am pretty sure that
> 1.4 did not. So while I am sure it could be made faster by
> implementing your suggestions, I do not think it can get down to the
> same times as 1.4. At least not on Windows where the working copy
> crawl is so slow.
I don't think it's the working copy crawl in 1.5 that causes the
slowdown. When I step through the code, it often checks whether a
working copy path exists in the repository - and those repo connections
cause the slowdown between 1.4 and 1.5.
> Out of curiosity, what are your timings for svn status on your working
> copy? I'd expect them to be a lot faster than the ones I posted. In
> the tests I did, merge was only 10-12 seconds slower than status. But
> as I pointed out, I was running them in a scenario with high bandwidth
> and low latency.
An 'svn st --ignore-externals' on my TSVN working copy is done in 2-3
seconds. But I'm currently using an svn client built with VS2005 instead
of VS2008. I'm trying to find out whether the merge time is the same as
with a VS2008 build.
> I think it would be interesting to see if VS 2008 has added any time
> to svn status though. My tests with the VC6 build of Subversion
> showed that merge time could be attributed primarily to the status
> crawl.
I think there were some optimizations when fetching the status of a
working copy in 1.5 which would more than compensate any delays caused
by VS008.
Update: the merge is the same when built with VS2005 than with VS2008.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Mark Phippard <ma...@gmail.com>.
On Sun, Jun 15, 2008 at 6:22 AM, Stefan Küng <to...@gmail.com> wrote:
> Lieven Govaerts wrote:
>>
>> Stefan Küng wrote:
>>>
>>> Hi,
>>>
>>> I've noticed that a merge done with a client built with VS2008 is twice
>>> as slow as a client built with VC6. Has anyone here noticed that too?
>>> I've described my efforts to solve this problem here:
>>> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>>>
>>> If anyone has any idea what could cause this, I'd really appreciate any
>>> help.
>>
>> The obvious question first: you are using a Release build of course?
>>
>> Concerning profiling, IBM has a 15-days trial edition of Rational
>> PurifyPlus at their website:
>> http://www-306.ibm.com/software/awdtools/purifyplus/
>> I've never used it (well, not in the last 10 years), but at first sight it
>> should allow you to profile different builds of svn.
>
> Sure, but it requires to have the app built with purifyplus installed. And
> my problem is that I can't do that with VC6 since I don't have it.
>
>> With every release of VS comes a new version of the CRT. I wouldn't be
>> surprised of some of the improvements it brings are in fact downgrading
>> performance in the algorithms used in 'svn merge'. While it could be
>> networking related, it could be slower memory block allocation etc.
>> How does svn 1.4.6 compare in merge time in VC6 vs VS2008?
>
> I guess many people will get really annoyed/angry with the 1.5 release of
> Subversion. I've ran some tests with different builds, all tests done with
> the command line:
> svn merge -r11761:11762
> http://tortoiosesvn.tigris.org/svn/tortoisesvn/branches/1.4.x --dry-run
>
> (for the 1.5 clients, I also added '--accept postpone' to the command line).
>
> Here are my results:
>
> my build from 1.5.x branch, built with VS2008:
> 5:17 .. 5:34
>
> 1.5.0-RC9 from open.collab.net:
> 2:22 .. 2:50
>
> 1.4.6 client:
> 0:12 .. 0:15
>
> Which means a simple merge that took a few seconds with an 1.4 client now
> takes minutes!
> (the numbers above are minutes:seconds)
>
>
> Since I've been trying to optimize/tweak my build for more than two days
> now, I can tell you what I found so far:
>
> The delays happen because (at least on Windows) a 'connect' is expensive. It
> seems that a build with VS2008 adds some more sanity check code for those
> calls which makes those take a few miliseconds longer than a build with VC6.
> Most svn commands only connect once or twice to the repository, that's why
> most people never had performance problems with an 1.5.x build (those who
> don't do merging but only use checkout/commit/update/log).
>
> But a merge command loops over all merge ranges, connecting several times
> for each range. If it would re-use the connection it already has, it would
> be *much* faster, maybe almost as fast as an 1.4.x client.
Merge in 1.5 has to crawl the working copy, and I am pretty sure that
1.4 did not. So while I am sure it could be made faster by
implementing your suggestions, I do not think it can get down to the
same times as 1.4. At least not on Windows where the working copy
crawl is so slow.
Out of curiosity, what are your timings for svn status on your working
copy? I'd expect them to be a lot faster than the ones I posted. In
the tests I did, merge was only 10-12 seconds slower than status. But
as I pointed out, I was running them in a scenario with high bandwidth
and low latency.
I think it would be interesting to see if VS 2008 has added any time
to svn status though. My tests with the VC6 build of Subversion
showed that merge time could be attributed primarily to the status
crawl.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: merge very slow
Posted by Branko Čibej <br...@xbc.nu>.
Stefan Küng wrote:
> Branko Čibej wrote:
>> Stefan Küng wrote:
>>> Lieven Govaerts wrote:
>>>> Stefan Küng wrote:
>>>>> Hi,
>>>>>
>>>>> I've noticed that a merge done with a client built with VS2008 is
>>>>> twice as slow as a client built with VC6. Has anyone here noticed
>>>>> that too?
>>>>> I've described my efforts to solve this problem here:
>>>>> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>>>>>
>>>>> If anyone has any idea what could cause this, I'd really
>>>>> appreciate any help.
>>>> The obvious question first: you are using a Release build of course?
>>>>
>>>> Concerning profiling, IBM has a 15-days trial edition of Rational
>>>> PurifyPlus at their website:
>>>> http://www-306.ibm.com/software/awdtools/purifyplus/
>>>> I've never used it (well, not in the last 10 years), but at first
>>>> sight it should allow you to profile different builds of svn.
>>>
>>> Sure, but it requires to have the app built with purifyplus
>>> installed. And my problem is that I can't do that with VC6 since I
>>> don't have it.
>>
>> For this kind of analysis I'd very strongly recommend Intel's VTune
>> over Quantify. VTune is a sampling profiler that doesn't affect the
>> programs execution speed much -- and thus leaves the interaction
>> between program, runtime lib and network stack pretty much unchanged.
>> Quantify is horrible on I/O bound tests because it totally changes
>> all timing-related effects.
>
> Thanks for the hint.
> I've installed and tried both. I couldn't get much usable data from
> both: it's not a CPU issue. The CPU usage stays at almost 0% during
> the whole merge.
That looks like something hanging for a timeout when the connection is
being made. My best guess: APR with IPv6 enabled tries a v6 connection
first, regardless of whether the network stack itself supports IPv6
(i.e. if the protocol driver is installed), and/or even if the resolved
address is IPv4. Now it's time to dig into the APR sources ...
> And I had to use a restore point to get rid of those tools again. I've
> rarely seen such intrusive applications!
Well, the most intrusive application I've ever seen is Windows, it took
me ages to get a box without it and it still lives on in a virtual
machine on this Mac. :)
>>> The delays happen because (at least on Windows) a 'connect' is
>>> expensive. It seems that a build with VS2008 adds some more sanity
>>> check code for those calls which makes those take a few miliseconds
>>> longer than a build with VC6.
>>
>> Oh duh. A few extra milliseconds on modern hardware? What, are they
>> doing a virus scan on the data? As if the networks stack on Windows
>> wasn't already rotten enough.
>
> My hardware is quite good:
> T7500@2.20GHz, 4GB RAM
>
> So that's definitely not an issue.
No no, you misunderstood -- I mean that a few milliseconds on modern
hardware is a *huge* amount of time. Really totally unacceptable.
> Also, it doesn't help to rant about the Windows network stack. If you
> like it or not, many people have to live with it and are quite happy.
> Other applications can deal with it well enough.
Hey, I'm trying to help with a Windows issue, surely I'm allowed the
occasional rant? :)
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: merge very slow
Posted by Stefan Küng <to...@gmail.com>.
Branko Čibej wrote:
> Stefan Küng wrote:
>> Lieven Govaerts wrote:
>>> Stefan Küng wrote:
>>>> Hi,
>>>>
>>>> I've noticed that a merge done with a client built with VS2008 is
>>>> twice as slow as a client built with VC6. Has anyone here noticed
>>>> that too?
>>>> I've described my efforts to solve this problem here:
>>>> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>>>>
>>>> If anyone has any idea what could cause this, I'd really appreciate
>>>> any help.
>>> The obvious question first: you are using a Release build of course?
>>>
>>> Concerning profiling, IBM has a 15-days trial edition of Rational
>>> PurifyPlus at their website:
>>> http://www-306.ibm.com/software/awdtools/purifyplus/
>>> I've never used it (well, not in the last 10 years), but at first
>>> sight it should allow you to profile different builds of svn.
>>
>> Sure, but it requires to have the app built with purifyplus installed.
>> And my problem is that I can't do that with VC6 since I don't have it.
>
> For this kind of analysis I'd very strongly recommend Intel's VTune over
> Quantify. VTune is a sampling profiler that doesn't affect the programs
> execution speed much -- and thus leaves the interaction between program,
> runtime lib and network stack pretty much unchanged. Quantify is
> horrible on I/O bound tests because it totally changes all
> timing-related effects.
Thanks for the hint.
I've installed and tried both. I couldn't get much usable data from
both: it's not a CPU issue. The CPU usage stays at almost 0% during the
whole merge.
And I had to use a restore point to get rid of those tools again. I've
rarely seen such intrusive applications!
>> The delays happen because (at least on Windows) a 'connect' is
>> expensive. It seems that a build with VS2008 adds some more sanity
>> check code for those calls which makes those take a few miliseconds
>> longer than a build with VC6.
>
> Oh duh. A few extra milliseconds on modern hardware? What, are they
> doing a virus scan on the data? As if the networks stack on Windows
> wasn't already rotten enough.
My hardware is quite good:
T7500@2.20GHz, 4GB RAM
So that's definitely not an issue. Also, it doesn't help to rant about
the Windows network stack. If you like it or not, many people have to
live with it and are quite happy. Other applications can deal with it
well enough.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Branko Čibej <br...@xbc.nu>.
Stefan Küng wrote:
> Lieven Govaerts wrote:
>> Stefan Küng wrote:
>>> Hi,
>>>
>>> I've noticed that a merge done with a client built with VS2008 is
>>> twice as slow as a client built with VC6. Has anyone here noticed
>>> that too?
>>> I've described my efforts to solve this problem here:
>>> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>>>
>>> If anyone has any idea what could cause this, I'd really appreciate
>>> any help.
>> The obvious question first: you are using a Release build of course?
>>
>> Concerning profiling, IBM has a 15-days trial edition of Rational
>> PurifyPlus at their website:
>> http://www-306.ibm.com/software/awdtools/purifyplus/
>> I've never used it (well, not in the last 10 years), but at first
>> sight it should allow you to profile different builds of svn.
>
> Sure, but it requires to have the app built with purifyplus installed.
> And my problem is that I can't do that with VC6 since I don't have it.
For this kind of analysis I'd very strongly recommend Intel's VTune over
Quantify. VTune is a sampling profiler that doesn't affect the programs
execution speed much -- and thus leaves the interaction between program,
runtime lib and network stack pretty much unchanged. Quantify is
horrible on I/O bound tests because it totally changes all
timing-related effects.
>> With every release of VS comes a new version of the CRT. I wouldn't
>> be surprised of some of the improvements it brings are in fact
>> downgrading performance in the algorithms used in 'svn merge'. While
>> it could be networking related, it could be slower memory block
>> allocation etc.
>> How does svn 1.4.6 compare in merge time in VC6 vs VS2008?
>
> I guess many people will get really annoyed/angry with the 1.5 release
> of Subversion. I've ran some tests with different builds, all tests
> done with the command line:
> svn merge -r11761:11762
> http://tortoiosesvn.tigris.org/svn/tortoisesvn/branches/1.4.x --dry-run
>
> (for the 1.5 clients, I also added '--accept postpone' to the command
> line).
>
> Here are my results:
>
> my build from 1.5.x branch, built with VS2008:
> 5:17 .. 5:34
>
> 1.5.0-RC9 from open.collab.net:
> 2:22 .. 2:50
>
> 1.4.6 client:
> 0:12 .. 0:15
>
> Which means a simple merge that took a few seconds with an 1.4 client
> now takes minutes!
> (the numbers above are minutes:seconds)
>
>
> Since I've been trying to optimize/tweak my build for more than two
> days now, I can tell you what I found so far:
>
> The delays happen because (at least on Windows) a 'connect' is
> expensive. It seems that a build with VS2008 adds some more sanity
> check code for those calls which makes those take a few miliseconds
> longer than a build with VC6.
Oh duh. A few extra milliseconds on modern hardware? What, are they
doing a virus scan on the data? As if the networks stack on Windows
wasn't already rotten enough.
> But a merge command loops over all merge ranges, connecting several
> times for each range. If it would re-use the connection it already
> has, it would be *much* faster, maybe almost as fast as an 1.4.x client.
I wonder if this is even an option ... opinions, anyone?
> * Serf is twice as fast as neon for merges, but only if IPv6 support
> is disabled.
> * ra_neon exchanges the capabilities for *every* connect, which means
> that's done several times for one single merge (i.e., a lot of data
> sent over the wire with no benefit at all)
> * ra_svn is much faster than ra_neon/ra_serf, but the same here: with
> IPv6 enabled it takes twice as long
The svnserve protocol is stateful, and doesn't have all the HTTP/DAV
cruft, so it's not surprising that it's faster.
> * IPv6 has no (or almost no) impact on ra_neon when doing a merge. Of
> those 5 and a half minutes, the gain by disabling IPv6 is around 10
> seconds
This pretty much points at an IPv6-related problem in APR -- since Neon
doesn't use APR for its network layer, while Serf and ra_svn do.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: merge very slow
Posted by Stefan Küng <to...@gmail.com>.
Lieven Govaerts wrote:
> If the merge code can be optimized to make less ra calls, sure, that's
> better. But in the case that those 149 merge/update reports are really
> needed, I don't see a reason why we cannot reuse earlier idle
> connections in the ra_serf library. Isn't one of the principles of serf
> to allow maximum number of requests over a minimum number of sockets?
This might seem a little bit off topic now, but may I suggest to enhance
the svn_client_* API so that sessions can be re-used between those calls
too?
>> I've looked to merge.c code and I've shocked: we create many ra
>> sessions when merging. This is source of all slowdowns. And also there
>> is no comments why we doing so. Could someone explain why we creating
>> new sessions instead of reusing?
>>
>
> Did you look at the numbers I got from my test case? Only 7 sessions,
> and all of those were created at the start of the merge. So either you
> misinterpreted the code or my code tracing was incorrect. I don't know
> the merge code well enough to know who's right or wrong.
Yes, and I think my assumption was wrong. I'm now testing in another
direction. I'll report back as soon as I find something.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Lieven Govaerts <sv...@mobsol.be>.
Ivan Zhakov wrote:
> On Sun, Jun 15, 2008 at 5:42 AM, Lieven Govaerts <sv...@mobsol.be> wrote:
>> Stefan Küng wrote:
>>> Lieven Govaerts wrote:
>>>> Stefan Küng wrote:
[..]
>>> I guess many people will get really annoyed/angry with the 1.5 release of
>>> Subversion. I've ran some tests with different builds, all tests done with
>>> the command line:
>>> svn merge -r11761:11762
>>> http://tortoiosesvn.tigris.org/svn/tortoisesvn/branches/1.4.x --dry-run
>>>
>>> (for the 1.5 clients, I also added '--accept postpone' to the command
>>> line).
>>>
>>> Here are my results:
>>>
>>> my build from 1.5.x branch, built with VS2008:
>>> 5:17 .. 5:34
>>>
>>> 1.5.0-RC9 from open.collab.net:
>>> 2:22 .. 2:50
>>>
>>> 1.4.6 client:
>>> 0:12 .. 0:15
> Bah! That's awfull. IMHO it could be showstopper of 1.5.x :(
>
>>> * Serf is twice as fast as neon for merges, but only if IPv6 support is
>>> disabled.
>> I did a merge from a range of revisions of svn /trunk to a svn 1.5 working
>> copy with a trunk svn and ra_serf. This is the result:
>>
>> Nr. of opened sessions: 7
>> Nr. of reports sent to the server: 149
>> Nr. of opened sockets: 241
>>
>> This was a debug build, so the total time is not really important.
>>
>> The difference in nr. of reports and nr. of sockets is the fact that ra_serf
>> opens more than one socket per report, in order to send multiple requests
>> in parallel.
>>
>> Anyway, no less than 241 sockets is way to high. It should be possible to
>> reuse those sockets more, but this should be solved in the ra layers. I'll
>> have a look at ra_serf to see what we can do.
>>
> Reusing sockets in ra layers is bad idea. It's will be very implicit
> behavior, so problem will just hide one more level deeper.
>
If the merge code can be optimized to make less ra calls, sure, that's
better. But in the case that those 149 merge/update reports are really
needed, I don't see a reason why we cannot reuse earlier idle
connections in the ra_serf library. Isn't one of the principles of serf
to allow maximum number of requests over a minimum number of sockets?
> I've looked to merge.c code and I've shocked: we create many ra
> sessions when merging. This is source of all slowdowns. And also there
> is no comments why we doing so. Could someone explain why we creating
> new sessions instead of reusing?
>
Did you look at the numbers I got from my test case? Only 7 sessions,
and all of those were created at the start of the merge. So either you
misinterpreted the code or my code tracing was incorrect. I don't know
the merge code well enough to know who's right or wrong.
Lieven
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: merge very slow
Posted by Ivan Zhakov <ch...@gmail.com>.
On Sun, Jun 15, 2008 at 5:42 AM, Lieven Govaerts <sv...@mobsol.be> wrote:
> Stefan Küng wrote:
>>
>> Lieven Govaerts wrote:
>>>
>>> Stefan Küng wrote:
>>>>
>>>> Hi,
>>>>
>>>> I've noticed that a merge done with a client built with VS2008 is twice
>>>> as slow as a client built with VC6. Has anyone here noticed that too?
>>>> I've described my efforts to solve this problem here:
>>>> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>>>>
>>>> If anyone has any idea what could cause this, I'd really appreciate any
>>>> help.
>>>
>>> The obvious question first: you are using a Release build of course?
>>>
>>> Concerning profiling, IBM has a 15-days trial edition of Rational
>>> PurifyPlus at their website:
>>> http://www-306.ibm.com/software/awdtools/purifyplus/
>>> I've never used it (well, not in the last 10 years), but at first sight
>>> it should allow you to profile different builds of svn.
>>
>> Sure, but it requires to have the app built with purifyplus installed. And
>> my problem is that I can't do that with VC6 since I don't have it.
>>
>>> With every release of VS comes a new version of the CRT. I wouldn't be
>>> surprised of some of the improvements it brings are in fact downgrading
>>> performance in the algorithms used in 'svn merge'. While it could be
>>> networking related, it could be slower memory block allocation etc.
>>> How does svn 1.4.6 compare in merge time in VC6 vs VS2008?
>>
>> I guess many people will get really annoyed/angry with the 1.5 release of
>> Subversion. I've ran some tests with different builds, all tests done with
>> the command line:
>> svn merge -r11761:11762
>> http://tortoiosesvn.tigris.org/svn/tortoisesvn/branches/1.4.x --dry-run
>>
>> (for the 1.5 clients, I also added '--accept postpone' to the command
>> line).
>>
>> Here are my results:
>>
>> my build from 1.5.x branch, built with VS2008:
>> 5:17 .. 5:34
>>
>> 1.5.0-RC9 from open.collab.net:
>> 2:22 .. 2:50
>>
>> 1.4.6 client:
>> 0:12 .. 0:15
Bah! That's awfull. IMHO it could be showstopper of 1.5.x :(
>> * Serf is twice as fast as neon for merges, but only if IPv6 support is
>> disabled.
>
> I did a merge from a range of revisions of svn /trunk to a svn 1.5 working
> copy with a trunk svn and ra_serf. This is the result:
>
> Nr. of opened sessions: 7
> Nr. of reports sent to the server: 149
> Nr. of opened sockets: 241
>
> This was a debug build, so the total time is not really important.
>
> The difference in nr. of reports and nr. of sockets is the fact that ra_serf
> opens more than one socket per report, in order to send multiple requests
> in parallel.
>
> Anyway, no less than 241 sockets is way to high. It should be possible to
> reuse those sockets more, but this should be solved in the ra layers. I'll
> have a look at ra_serf to see what we can do.
>
Reusing sockets in ra layers is bad idea. It's will be very implicit
behavior, so problem will just hide one more level deeper.
I've looked to merge.c code and I've shocked: we create many ra
sessions when merging. This is source of all slowdowns. And also there
is no comments why we doing so. Could someone explain why we creating
new sessions instead of reusing?
--
Ivan Zhakov
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: merge very slow
Posted by Lieven Govaerts <sv...@mobsol.be>.
Stefan Küng wrote:
> Lieven Govaerts wrote:
>> Stefan Küng wrote:
>>> Hi,
>>>
>>> I've noticed that a merge done with a client built with VS2008 is
>>> twice as slow as a client built with VC6. Has anyone here noticed
>>> that too?
>>> I've described my efforts to solve this problem here:
>>> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>>>
>>> If anyone has any idea what could cause this, I'd really appreciate
>>> any help.
>> The obvious question first: you are using a Release build of course?
>>
>> Concerning profiling, IBM has a 15-days trial edition of Rational
>> PurifyPlus at their website:
>> http://www-306.ibm.com/software/awdtools/purifyplus/
>> I've never used it (well, not in the last 10 years), but at first
>> sight it should allow you to profile different builds of svn.
>
> Sure, but it requires to have the app built with purifyplus installed.
> And my problem is that I can't do that with VC6 since I don't have it.
>
>> With every release of VS comes a new version of the CRT. I wouldn't be
>> surprised of some of the improvements it brings are in fact
>> downgrading performance in the algorithms used in 'svn merge'. While
>> it could be networking related, it could be slower memory block
>> allocation etc.
>> How does svn 1.4.6 compare in merge time in VC6 vs VS2008?
>
> I guess many people will get really annoyed/angry with the 1.5 release
> of Subversion. I've ran some tests with different builds, all tests done
> with the command line:
> svn merge -r11761:11762
> http://tortoiosesvn.tigris.org/svn/tortoisesvn/branches/1.4.x --dry-run
>
> (for the 1.5 clients, I also added '--accept postpone' to the command
> line).
>
> Here are my results:
>
> my build from 1.5.x branch, built with VS2008:
> 5:17 .. 5:34
>
> 1.5.0-RC9 from open.collab.net:
> 2:22 .. 2:50
>
> 1.4.6 client:
> 0:12 .. 0:15
>
> Which means a simple merge that took a few seconds with an 1.4 client
> now takes minutes!
> (the numbers above are minutes:seconds)
>
>
> Since I've been trying to optimize/tweak my build for more than two days
> now, I can tell you what I found so far:
>
> The delays happen because (at least on Windows) a 'connect' is
> expensive. It seems that a build with VS2008 adds some more sanity check
> code for those calls which makes those take a few miliseconds longer
> than a build with VC6.
Hm, it'd have to be a bit more than a few milliseconds per connect.
So what you're saying is that svn merge 1.4 is ten times faster than svn
merge 1.5/VC6, which on its turn is two times faster as svn merge
1.5/VS2k8 right?
Do you have the option to return to VS2k5 for TSVN? That doesn't solve
the general slowdown (see below), but at least TSVN is than equally slow
as svn - assuming that VS2k5 has the same performance characteristics as
VC60.
> Most svn commands only connect once or twice to the repository, that's
> why most people never had performance problems with an 1.5.x build
> (those who don't do merging but only use checkout/commit/update/log).
>
> But a merge command loops over all merge ranges, connecting several
> times for each range. If it would re-use the connection it already has,
> it would be *much* faster, maybe almost as fast as an 1.4.x client.
>
> * Serf is twice as fast as neon for merges, but only if IPv6 support is
> disabled.
I did a merge from a range of revisions of svn /trunk to a svn 1.5
working copy with a trunk svn and ra_serf. This is the result:
Nr. of opened sessions: 7
Nr. of reports sent to the server: 149
Nr. of opened sockets: 241
This was a debug build, so the total time is not really important.
The difference in nr. of reports and nr. of sockets is the fact that
ra_serf opens more than one socket per report, in order to send multiple
requests in parallel.
Anyway, no less than 241 sockets is way to high. It should be possible
to reuse those sockets more, but this should be solved in the ra layers.
I'll have a look at ra_serf to see what we can do.
> * ra_neon exchanges the capabilities for *every* connect, which means
> that's done several times for one single merge (i.e., a lot of data sent
> over the wire with no benefit at all)
> * ra_svn is much faster than ra_neon/ra_serf, but the same here: with
> IPv6 enabled it takes twice as long
Isn't apr following a 'connect over ipv6' -> timeout -> 'retry over
ipv4' scheme?
Lieven
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: merge very slow
Posted by Stefan Küng <to...@gmail.com>.
Lieven Govaerts wrote:
> Stefan Küng wrote:
>> Hi,
>>
>> I've noticed that a merge done with a client built with VS2008 is
>> twice as slow as a client built with VC6. Has anyone here noticed that
>> too?
>> I've described my efforts to solve this problem here:
>> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>>
>> If anyone has any idea what could cause this, I'd really appreciate
>> any help.
> The obvious question first: you are using a Release build of course?
>
> Concerning profiling, IBM has a 15-days trial edition of Rational
> PurifyPlus at their website:
> http://www-306.ibm.com/software/awdtools/purifyplus/
> I've never used it (well, not in the last 10 years), but at first sight
> it should allow you to profile different builds of svn.
Sure, but it requires to have the app built with purifyplus installed.
And my problem is that I can't do that with VC6 since I don't have it.
> With every release of VS comes a new version of the CRT. I wouldn't be
> surprised of some of the improvements it brings are in fact downgrading
> performance in the algorithms used in 'svn merge'. While it could be
> networking related, it could be slower memory block allocation etc.
> How does svn 1.4.6 compare in merge time in VC6 vs VS2008?
I guess many people will get really annoyed/angry with the 1.5 release
of Subversion. I've ran some tests with different builds, all tests done
with the command line:
svn merge -r11761:11762
http://tortoiosesvn.tigris.org/svn/tortoisesvn/branches/1.4.x --dry-run
(for the 1.5 clients, I also added '--accept postpone' to the command line).
Here are my results:
my build from 1.5.x branch, built with VS2008:
5:17 .. 5:34
1.5.0-RC9 from open.collab.net:
2:22 .. 2:50
1.4.6 client:
0:12 .. 0:15
Which means a simple merge that took a few seconds with an 1.4 client
now takes minutes!
(the numbers above are minutes:seconds)
Since I've been trying to optimize/tweak my build for more than two days
now, I can tell you what I found so far:
The delays happen because (at least on Windows) a 'connect' is
expensive. It seems that a build with VS2008 adds some more sanity check
code for those calls which makes those take a few miliseconds longer
than a build with VC6.
Most svn commands only connect once or twice to the repository, that's
why most people never had performance problems with an 1.5.x build
(those who don't do merging but only use checkout/commit/update/log).
But a merge command loops over all merge ranges, connecting several
times for each range. If it would re-use the connection it already has,
it would be *much* faster, maybe almost as fast as an 1.4.x client.
* Serf is twice as fast as neon for merges, but only if IPv6 support is
disabled.
* ra_neon exchanges the capabilities for *every* connect, which means
that's done several times for one single merge (i.e., a lot of data sent
over the wire with no benefit at all)
* ra_svn is much faster than ra_neon/ra_serf, but the same here: with
IPv6 enabled it takes twice as long
* IPv6 has no (or almost no) impact on ra_neon when doing a merge. Of
those 5 and a half minutes, the gain by disabling IPv6 is around 10 seconds
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
Re: merge very slow
Posted by Lieven Govaerts <sv...@mobsol.be>.
Stefan Küng wrote:
> Hi,
>
> I've noticed that a merge done with a client built with VS2008 is
> twice as slow as a client built with VC6. Has anyone here noticed that
> too?
> I've described my efforts to solve this problem here:
> http://tortoisesvn.tigris.org/servlets/ReadMsg?list=dev&msgNo=34103
>
> If anyone has any idea what could cause this, I'd really appreciate
> any help.
The obvious question first: you are using a Release build of course?
Concerning profiling, IBM has a 15-days trial edition of Rational
PurifyPlus at their website:
http://www-306.ibm.com/software/awdtools/purifyplus/
I've never used it (well, not in the last 10 years), but at first sight
it should allow you to profile different builds of svn.
With every release of VS comes a new version of the CRT. I wouldn't be
surprised of some of the improvements it brings are in fact downgrading
performance in the algorithms used in 'svn merge'. While it could be
networking related, it could be slower memory block allocation etc.
How does svn 1.4.6 compare in merge time in VC6 vs VS2008?
I have neither VS2008 nor VC6 here so I can't help you testing.
Lieven
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org