You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Victor Sudakov <su...@sibptus.tomsk.ru> on 2011/02/08 18:32:47 UTC
Re: Betr.: Re: "svnadmin load" a huge file
Johan Corveleyn wrote:
[dd]
> But that doesn't explain why the resulting repository is so large
> (compared to the original CVS repository). Sure, there might be memory
> usage problems in dump/load (it uses more memory than the resulting
> repository uses diskspace), but I think there is more going on.
>
> That's why I'm guessing on rev files being large (and the
> corresponding memory structures) because of the amount of dir entries
> in each revision. I'm not that intimately familiar with how this is
> all represented, and how the rev files are structured and all that, so
> I'm just guessing ... I seem to remember something like this from
> another discussion in the past.
I have created a small testcase script:
#!/bin/sh
for i in `jot 15000`
do
cat > Testfile_${i}.txt << __END__
This is a small test file.
This is a small test file.
This is a small test file.
This is a small test file.
This is a small test file.
This is a small test file.
__END__
svn add Testfile_${i}.txt
svn commit -m "Iteration $i"
done
After the 15000th commit, the size of the repository on disk is 5.5G
with the working directory size being 120M. Besides, after several
thousand commits to this directory SVN slows down considerably. This
must be some design flaw (or peculiarity if you like) of SVN.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
sip:sudakov@sibptus.tomsk.ru
Re: Betr.: Re: "svnadmin load" a huge file
Posted by Victor Sudakov <su...@sibptus.tomsk.ru>.
Les Mikesell wrote:
> >On Tue, Feb 08, 2011 at 11:32:47PM +0600, Victor Sudakov wrote:
> >>After the 15000th commit, the size of the repository on disk is 5.5G
> >>with the working directory size being 120M. Besides, after several
> >>thousand commits to this directory SVN slows down considerably. This
> >>must be some design flaw (or peculiarity if you like) of SVN.
> >
> >Probably related to the way directories are represented in the repository.
> >See http://svn.haxx.se/dev/archive-2011-02/0007.shtml
> >and also http://svn.haxx.se/dev/archive-2011-02/0014.shtml for some hints
> >to how this currently works.
>
> I'd expect even local operations like the compare against the pristine
> versions to decide what to commit to become slow when you put many
> thousands of files in one directory because most filesystems aren't good
> at that either (although they make fake it with caching). It's one of
> those "if it hurts, don't do it" things.
I did not know it would hurt until I tried to migrate this particular
repository from CVS to SVN.
FreeBSD by itself handles large directories very well due to its
dirhash feature.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
sip:sudakov@sibptus.tomsk.ru
Re: Betr.: Re: "svnadmin load" a huge file
Posted by Les Mikesell <le...@gmail.com>.
On 2/8/2011 1:34 PM, Stefan Sperling wrote:
> On Tue, Feb 08, 2011 at 11:32:47PM +0600, Victor Sudakov wrote:
>> After the 15000th commit, the size of the repository on disk is 5.5G
>> with the working directory size being 120M. Besides, after several
>> thousand commits to this directory SVN slows down considerably. This
>> must be some design flaw (or peculiarity if you like) of SVN.
>
> Probably related to the way directories are represented in the repository.
> See http://svn.haxx.se/dev/archive-2011-02/0007.shtml
> and also http://svn.haxx.se/dev/archive-2011-02/0014.shtml for some hints
> to how this currently works.
I'd expect even local operations like the compare against the pristine
versions to decide what to commit to become slow when you put many
thousands of files in one directory because most filesystems aren't good
at that either (although they make fake it with caching). It's one of
those "if it hurts, don't do it" things.
--
Les Mikesell
lesmikesell@gmail.com
Re: Betr.: Re: "svnadmin load" a huge file
Posted by Victor Sudakov <su...@sibptus.tomsk.ru>.
Stefan Sperling wrote:
> > After the 15000th commit, the size of the repository on disk is 5.5G
> > with the working directory size being 120M. Besides, after several
> > thousand commits to this directory SVN slows down considerably. This
> > must be some design flaw (or peculiarity if you like) of SVN.
>
> Probably related to the way directories are represented in the repository.
> See http://svn.haxx.se/dev/archive-2011-02/0007.shtml
> and also http://svn.haxx.se/dev/archive-2011-02/0014.shtml for some hints
> to how this currently works.
BTW I use the FSFS backend if it makes any difference.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
sip:sudakov@sibptus.tomsk.ru
Re: Betr.: Re: "svnadmin load" a huge file
Posted by Stefan Sperling <st...@elego.de>.
On Tue, Feb 08, 2011 at 11:32:47PM +0600, Victor Sudakov wrote:
> After the 15000th commit, the size of the repository on disk is 5.5G
> with the working directory size being 120M. Besides, after several
> thousand commits to this directory SVN slows down considerably. This
> must be some design flaw (or peculiarity if you like) of SVN.
Probably related to the way directories are represented in the repository.
See http://svn.haxx.se/dev/archive-2011-02/0007.shtml
and also http://svn.haxx.se/dev/archive-2011-02/0014.shtml for some hints
to how this currently works.