You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Jaap Vermeulen <ja...@skyscape.com> on 2004/09/24 21:30:29 UTC
Subversion slows down with large number of files?
Hi,
I'm doing a test migration of our current source control system (~ 2.5 Gb)
to Subversion.
I'm running 1.1.0 RC 3 using the svnserve server on one machine, and use the
svn: protocol on the client machine.
Everything has been working fine until I hit a directory with approx. 10332
files in it (small HTML files, combined total size of 18.2 Mb). This seems
to bog down SVN (or Tortoise SVN 1.1.0 RC2) to an enormous extent. If I use
the command line svn command, it will eat up close to 100% CPU time for 10
minutes or so on a commit. If I use the RepoBrowser and try to expand the
directory with those files, the TortoiseProc will eat up 100% for a long
time. If I try to open or refresh an explorer window, it goes to 100% for a
couple of minutes.
Is this to be expected? Doesn't seem like it should cause such a dramatic
slowdown.
Thanks,
-Jaap-
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion slows down with large number of files?
Posted by Julian Foad <ju...@btopenworld.com>.
Jaap Vermeulen wrote:
>
> I'm running 1.1.0 RC 3 using the svnserve server on one machine, and use the
> svn: protocol on the client machine.
>
> Everything has been working fine until I hit a directory with approx. 10332
> files in it (small HTML files, combined total size of 18.2 Mb). This seems
> to bog down SVN (or Tortoise SVN 1.1.0 RC2) to an enormous extent. If I use
> the command line svn command, it will eat up close to 100% CPU time for 10
> minutes or so on a commit. If I use the RepoBrowser and try to expand the
> directory with those files, the TortoiseProc will eat up 100% for a long
> time. If I try to open or refresh an explorer window, it goes to 100% for a
> couple of minutes.
>
> Is this to be expected? Doesn't seem like it should cause such a dramatic
> slowdown.
It is a known problem - that Subversion is slow when using the BDB back-end and there are very many files in the same directory. There doesn't seem to be an entry for it in the issue tracker. Please could you file one?
In the thread "Perf issues with BDB and directories with a large number of items",
Branko Čibej wrote:
> Brian W. Fitzpatrick wrote:
>
>> Summary: If you have a directory in your repository with a large number
>> of items in it, the BDB backend gets slower as you add more items (due
>> parsing/unparsing skels I'll bet).
>>
>> FSFS does not exhibit this behavior.
>>
>> Details and Pretty Pictures here: http://www.red-bean.com/fitz/svn/
>>
> This is a textbook example of quadratic vs. logarithmic behaviour, and
> it's a safe bet that it's caused by our saving directory data in
> unsorted lists and rewriting the whole list at every directory change.
- Julian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion slows down with large number of files?
Posted by Greg Hudson <gh...@MIT.EDU>.
Please don't cross-post to the users and dev list--just pick one. In
this case, I think the users list would have been more appropriate, so
I'm trimming the cc line to that list.
On Fri, 2004-09-24 at 17:30, Jaap Vermeulen wrote:
> Everything has been working fine until I hit a directory with approx. 10332
> files in it (small HTML files, combined total size of 18.2 Mb). This seems
> to bog down SVN (or Tortoise SVN 1.1.0 RC2) to an enormous extent. If I use
> the command line svn command, it will eat up close to 100% CPU time for 10
> minutes or so on a commit.
Subversion has traditionally had some difficulties with single
directories containing many files. You might get better mileage using
the FSFS back end, which is optimized a little better for such
directories, if you're not doing so already.
> If I use the RepoBrowser and try to expand the
> directory with those files, the TortoiseProc will eat up 100% for a long
> time. If I try to open or refresh an explorer window, it goes to 100% for a
> couple of minutes.
Again, I think you might have better luck with the FSFS back end, though
I can't be sure.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org