350GB SVN repo creates around 1MB revision for simplest task

["FAISAL YAQOOB, BLOOMBERG/ 731 LEXIN" <fy...@bloomberg.net>]

This all started when I noticed that my repository size is increasing at a daily rate of 1GB. I did a simple test. Created a branch/tag of an existing folder that had a size of 35KB. I took note of revision number and went to $REPO/db/revs/<K-rev>/rev-number/ and checked the size of the revision. It was 1 mega byte. That sounds fishy. Any ideas on what might be wrong here. My repo is about 350GB in size with about 600,000 revisions.

P.S. I have already started a rebuild of the whole repository to see if that makes any difference but it will probably take days to complete.


Question: Is there anything in svn that when it reaches a certain size every commit will be that large?

Re: 350GB SVN repo creates around 1MB revision for simplest task

[B Smith-Mannschott <bs...@gmail.com>]

On Wed, Oct 13, 2010 at 00:45, FAISAL YAQOOB, BLOOMBERG/ 731 LEXIN
<fy...@bloomberg.net> wrote:
>
>
> This all started when I noticed that my repository size is increasing at a daily rate of 1GB. I did a simple test. Created a branch/tag of an existing folder that had a size of 35KB. I took note of revision number and went to $REPO/db/revs/<K-rev>/rev-number/ and checked the size of the revision. It was 1 mega byte. That sounds fishy. Any ideas on what might be wrong here. My repo is about 350GB in size with about 600,000 revisions.
>
> P.S. I have already started a rebuild of the whole repository to see if that makes any difference but it will probably take days to complete.
>
>
> Question: Is there anything in svn that when it reaches a certain size every commit will be that large?

Does your repository contain a directory with very many entries? Are
the changes that produce the large commits being made in or below such
a directory?

Let's assume to commit a single change to a single file to your
repository. Let's further assume the file is located here, in your
repository:

/project/trunk/some-really-large-directory/notes/blah.txt

When you commit the change to blah.txt, the new revision will rewrite
the directory nodes between 'blah.txt' and the root of the repository:
/project/trunk/some-really-large-directory/notes,
/project/trunk/some-really-large-directory, /project/trunk, /project,
/. When rewriting a directory node, FSFS always stores the new version
in its entirety. (This is different from the way changes to files are
stored, which are generally as differences to some previous version of
the same file.)

If /project/trunk/some-really-large-directory/ contains, say 10000
files, then each commit to blah.txt will store a full copy of this
directory (with its 10'000 names) in your repository.

I noticed this when I started keeping a personal wiki under version
control a few years ago. It was a flat directory of over 10'000 text
files. I quickly noticed that commits were pretty big. (I've since
switched to git for that task, for this and other reasons.)

see also
http://svn.apache.org/repos/asf/subversion/trunk/notes/subversion-design.html#server.fs.struct.bubble-up

// Ben

Re: 350GB SVN repo creates around 1MB revision for simplest task

[Ryan Schmidt <su...@ryandesign.com>]

On Oct 12, 2010, at 17:45, FAISAL YAQOOB, BLOOMBERG/ 731 LEXIN wrote:

> This all started when I noticed that my repository size is increasing at a daily rate of 1GB. I did a simple test. Created a branch/tag of an existing folder that had a size of 35KB. I took note of revision number and went to $REPO/db/revs/<K-rev>/rev-number/ and checked the size of the revision. It was 1 mega byte. That sounds fishy. Any ideas on what might be wrong here. My repo is about 350GB in size with about 600,000 revisions.
> 
> P.S. I have already started a rebuild of the whole repository to see if that makes any difference but it will probably take days to complete.
> 
> 
> Question: Is there anything in svn that when it reaches a certain size every commit will be that large?

Are there a lot of files in a single directory? The file/directory structure is stored in each revision. (Not sure if that's the file/directory structure of everything or just of the directories to which the commit applies.)

Re: 350GB SVN repo creates around 1MB revision for simplest task

[Ryan Schmidt <su...@ryandesign.com>]

On Oct 12, 2010, at 17:45, FAISAL YAQOOB, BLOOMBERG/ 731 LEXIN wrote:

> This all started when I noticed that my repository size is increasing at a daily rate of 1GB. I did a simple test. Created a branch/tag of an existing folder that had a size of 35KB. I took note of revision number and went to $REPO/db/revs/<K-rev>/rev-number/ and checked the size of the revision. It was 1 mega byte. That sounds fishy. Any ideas on what might be wrong here. My repo is about 350GB in size with about 600,000 revisions.
> 
> P.S. I have already started a rebuild of the whole repository to see if that makes any difference but it will probably take days to complete.
> 
> 
> Question: Is there anything in svn that when it reaches a certain size every commit will be that large?

Are there a lot of files in a single directory? The file/directory structure is stored in each revision. (Not sure if that's the file/directory structure of everything or just of the directories to which the commit applies.)