You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Bert Huijben <be...@vmoo.com> on 2008/10/05 00:23:16 UTC

FindFirstFile delay on merging resolved via profiling (r33447)

> -----Original Message-----
> From: rhuijben@tigris.org [mailto:rhuijben@tigris.org]
> Sent: zondag 5 oktober 2008 1:08
> To: svn@subversion.tigris.org
> Subject: svn commit: r33447 - trunk/subversion/libsvn_subr
> 
> Author: rhuijben
> Date: Sat Oct  4 16:08:22 2008
> New Revision: 33447
> 
> Log:
> Remove Windows specific side effects from svn_dirent_get_absolute.
> Passing
> the flag APR_FILEPATH_TRUENAME to apr_filepath_merge made it check and
> alter
> the casing of the passed path to the actual on-disk path. This makes
> the call
> several orders slower on windows than on other operating systems that
> don't
> check the path at all. This without actual profit as a case-changed
> path would
> fail on further processing.
> 
> Subversion fixes the casing of all user provided paths in
> svn_opt__arg_canonicalize_path, so we shouldn't check it again when we
> make
> a path absolute.
> 
> This patch saves several seconds on typical branch merges on Windows.
> (Removes
> a few thousand FindFirstFile calls on my test set involving only 21
> file
> changes on our 1.5.x branch.). The result is that it increases the
> merge
> performance on Windows by over 30%.

This two line patch on svn_dirent_get_absolute is the result of several hours of profiling and it makes the merge test we profiled 30% faster. (It probably increases the speed of many other subversion commands as well).

This very Windows specific slowdown was added in r13 when the call to _fullpath() from the Microsoft C++ library was replaced by the more generic apr_filepath_merge.

On all platforms except windows apr_filepath_merge can only make the path absolute, but on Windows it also allows the flag APR_FILEPATH_TRUENAME that actually verifies if the path exists by checking the exact casing of the path and all its ancestors.

!! It actually performs disk IO !!

On a typical test merge:
svn merge -c 31121,31211,31733,31846,32497,32800,32836,32837,33156,33178 http://svn.collab.net/repos/svn/trunk F:\wc\1.5.x
(F:\wc\1.5.x = 1.5.x branch @33444)

This svn_dirent_get_absolute involves several thousands of calls to FindFirstFile, while only 22 files receive updates.

Removing this flag shaves 7 seconds from the merge time on my quad core pc.
When I run it on a local mirror of the subversion repository the difference is well over 30%. (Over webdav most time is spent waiting on bits to arrive from the USA)


Lgo found some other performance issues around canonicalize (taking 20% of merge on Linux) and we are still looking for other issues, so I hope we will see some other performance improvements in the next few days.


	Bert


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org


Re: FindFirstFile delay on merging resolved via profiling (r33447)

Posted by Erik Huelsmann <eh...@gmail.com>.
On Mon, Oct 6, 2008 at 7:55 PM, Sander Rijken <sr...@d-90.nl> wrote:
>>
>> This two line patch on svn_dirent_get_absolute is the result of several
>> hours of profiling and it makes the merge test we profiled 30% faster. (It
>> probably increases the speed of many other subversion commands as well).
>>
>> This very Windows specific slowdown was added in r13 when the call to
>> _fullpath() from the Microsoft C++ library was replaced by the more generic
>> apr_filepath_merge.
>>
>> On all platforms except windows apr_filepath_merge can only make the path
>> absolute, but on Windows it also allows the flag APR_FILEPATH_TRUENAME that
>> actually verifies if the path exists by checking the exact casing of the
>> path and all its ancestors.
>>
>> !! It actually performs disk IO !!
>>
>> On a typical test merge:
>> svn merge -c 31121,31211,31733,31846,32497,32800,32836,32837,33156,33178
>> http://svn.collab.net/repos/svn/trunk F:\wc\1.5.x
>> (F:\wc\1.5.x = 1.5.x branch @33444)
>>
>> This svn_dirent_get_absolute involves several thousands of calls to
>> FindFirstFile, while only 22 files receive updates.
>>
>> Removing this flag shaves 7 seconds from the merge time on my quad core
>> pc.
>> When I run it on a local mirror of the subversion repository the
>> difference is well over 30%. (Over webdav most time is spent waiting on bits
>> to arrive from the USA)
>>
>
> Actually in the above merge example, 9 merges are performed (one of them has
> already been merged), and for each merge svn_sleep_for_timestamps is called
> (sleeping for 1.1 seconds)
> This means that going from 19 to 12 seconds, should actually be explained as
> going from 9 to 2 seconds, which is a far greater improvement.
>
> Do we need the sleep function to be called after merging every revision
> (most commands that call it only call it when they're finished)?

No, we need it only at the end.

Bye,

Erik.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: FindFirstFile delay on merging resolved via profiling (r33447)

Posted by Sander Rijken <sr...@d-90.nl>.
> 
> This two line patch on svn_dirent_get_absolute is the result of  
> several hours of profiling and it makes the merge test we profiled  
> 30% faster. (It probably increases the speed of many other  
> subversion commands as well).
>
> This very Windows specific slowdown was added in r13 when the call  
> to _fullpath() from the Microsoft C++ library was replaced by the  
> more generic apr_filepath_merge.
>
> On all platforms except windows apr_filepath_merge can only make the  
> path absolute, but on Windows it also allows the flag  
> APR_FILEPATH_TRUENAME that actually verifies if the path exists by  
> checking the exact casing of the path and all its ancestors.
>
> !! It actually performs disk IO !!
>
> On a typical test merge:
> svn merge -c  
> 31121,31211,31733,31846,32497,32800,32836,32837,33156,33178 http://svn.collab.net/repos/svn/trunk 
>  F:\wc\1.5.x
> (F:\wc\1.5.x = 1.5.x branch @33444)
>
> This svn_dirent_get_absolute involves several thousands of calls to  
> FindFirstFile, while only 22 files receive updates.
>
> Removing this flag shaves 7 seconds from the merge time on my quad  
> core pc.
> When I run it on a local mirror of the subversion repository the  
> difference is well over 30%. (Over webdav most time is spent waiting  
> on bits to arrive from the USA)
>

Actually in the above merge example, 9 merges are performed (one of  
them has already been merged), and for each merge  
svn_sleep_for_timestamps is called (sleeping for 1.1 seconds)
This means that going from 19 to 12 seconds, should actually be  
explained as going from 9 to 2 seconds, which is a far greater  
improvement.

Do we need the sleep function to be called after merging every  
revision (most commands that call it only call it when they're  
finished)?

-- 
Sander Rijken

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org