Repository bloat.

[Oscar Fuentes <os...@telefonica.net>]

Hi.

I've commented on the past (on the user's mailing list) about small
commits (a few lines diff) causing svn repositories grow fast. For
example, in one of my repositories, a small commit made on certain
directory results on a 50+KB file on the svn repository (FSFS). The
directory contains almost 1000 files, and the problem is that the
revision file lists them several times.

Is this really needed? Is there any plan to fix it in the future?

The diff of the last 10 commits are about a few hundred lines, but the
revision files uses 500 KB, which seems a real waste.

Sample revision file attached.

-- 
Oscar

Re: Repository bloat.

[Daniel Berlin <db...@dberlin.org>]

Oscar Fuentes wrote:
> "Peter N. Lundblad" <pe...@famlundblad.se> writes:
> 
>> What you are seeing here is the change "bubling up" to the root.
>> Since you are replacing one node revision of a file with another, we
>> need to update the parent directory with a reference to that new
>> revision.  This, in turn, means that the parent has changed, and its
>> parent needs to be updated.  This process continues up to the root.
> 
> [snip]
> 
> Peter,
> 
> Thanks for your clear and informative answer. I'm afraid this is one
> of those things that needs a deep redesign to be fixed, so no hope for
> change soon.
It doesn't really need a deep redesign, it's just a chosen tradeoff.
You could store the dirs a different way (though it would be an
incompatible format change), but i'm not sure what way you think will do
better.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Repository bloat.

[Greg Hudson <gh...@MIT.EDU>]

On Mon, 2006-06-05 at 22:57 -0500, Ben Collins-Sussman wrote:
> It's not proportional to the number of files in the project, it's
> proportional to the depth of the file you're changing.  If you put
> 10,000 files in a single top-level directory, then changing a single
> file would result in only 2 new nodes being created in the new
> revision:  the file itself, and the top-level directory.

Yes... but one of those nodes is a new revision of a directory, with
10,000 entries in it.

I once came up with a vague idea for how to efficiently store directory
revisions: a binary tree or B-tree with multiple roots.  A single-entry
change to such a structure would have an O(lg(n)) space cost.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Repository bloat.

[Ben Collins-Sussman <su...@red-bean.com>]

On 6/5/06, Oscar Fuentes <os...@telefonica.net> wrote:
> "Peter N. Lundblad" <pe...@famlundblad.se> writes:
>
> > What you are seeing here is the change "bubling up" to the root.
> > Since you are replacing one node revision of a file with another, we
> > need to update the parent directory with a reference to that new
> > revision.  This, in turn, means that the parent has changed, and its
> > parent needs to be updated.  This process continues up to the root.
>
> [snip]
>
> Peter,
>
> Thanks for your clear and informative answer. I'm afraid this is one
> of those things that needs a deep redesign to be fixed, so no hope for
> change soon.
>
> It's a bad bad thing that a commit causes a repository grow that is
> proportional to the number of files of (part of) the working copy.

It's not proportional to the number of files in the project, it's
proportional to the depth of the file you're changing.  If you put
10,000 files in a single top-level directory, then changing a single
file would result in only 2 new nodes being created in the new
revision:  the file itself, and the top-level directory.  If you
change a file that's 20 levels deep within the repository, every
parent needs to be tweaked, so you'll see 21 new nodes created.

As others have already pointed out:  it's a deliberate decision, not a
"deep design" bug.  If we were to store directories compressed, it
would be *very* slow to access that file 20 levels deep.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Repository bloat.

[Oscar Fuentes <os...@telefonica.net>]

"Peter N. Lundblad" <pe...@famlundblad.se> writes:

> What you are seeing here is the change "bubling up" to the root.
> Since you are replacing one node revision of a file with another, we
> need to update the parent directory with a reference to that new
> revision.  This, in turn, means that the parent has changed, and its
> parent needs to be updated.  This process continues up to the root.

[snip]

Peter,

Thanks for your clear and informative answer. I'm afraid this is one
of those things that needs a deep redesign to be fixed, so no hope for
change soon.

It's a bad bad thing that a commit causes a repository grow that is
proportional to the number of files of (part of) the working copy. I
see some of my repositories jumping in the multi-gigabyte zone on a
matter of months.

Regards,

-- 
Oscar

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Repository bloat.

["Peter N. Lundblad" <pe...@famlundblad.se>]

Oscar Fuentes writes:
 > I've commented on the past (on the user's mailing list) about small
 > commits (a few lines diff) causing svn repositories grow fast. For
 > example, in one of my repositories, a small commit made on certain
 > directory results on a 50+KB file on the svn repository (FSFS). The
 > directory contains almost 1000 files, and the problem is that the
 > revision file lists them several times.
 > 

What you are seeing here is the change "bubling up" to the root.  Since you are
replacing one node revision of a file with another, we need to update the
parent directory with a reference to that new revision.  This, in
turn, means that the parent has changed, and its parent needs to be
updated.  This process continues up to the root.

I think Subfersion sometimes in the past (long before I got active in
the project) used to store directory changes as deltas, but that was a
performance problem.  So, instead, directories are always stored in
fulltext, meaning that large directories on the way up to the root
will be listed in the new revision.  I guess this is a classical speed
versus space tradeoff.  I don't know of any plans to change this
situation.  (One idea that sprints to mind would be to gzip compress
the directories, but I don't know how much that would cost
performance-wise.)

Regards,
//Peter

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org