Deltas?

[Stefan Boberg <sb...@team17.com>]

  Hi,

  Pardon the intrusion again! If you remember, I posted a question
during last week regarding problems with repository disk usage etc. That
was using 0.9, I downloaded the latest tar-ball the other day, and I
just ran some tests on it again, with much better results, but some
things still puzzle me...

  SVN no longer generates massive amounts of log data, which was the
main problem in my previous tests. However, it doesn't appear as though
deltas are currently used? Whenever I commit even minute changes (like a
single modified line for example), it appears as though the database
grows by the size of the entire modified file on every commit! This
appears to happen for both text and binary data. Is this how it is
supposed to be right now? Is there something I need to configure in
order to use delta storage?

  The tests were performed on a RH6.2 system if it makes any difference.

Thanks,
Stef! :)
--
Stefan Boberg - R&D Manager, Team17 Software Ltd.
boberg@team17.com


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Deltas?

[Daniel Berlin <da...@dberlin.org>]

On Sun, 10 Mar 2002, Daniel Berlin wrote:

> On Sun, 10 Mar 2002, Greg Hudson wrote:
> 
> > On Sun, 10 Mar 2002, Daniel Berlin wrote:
> > > Right now, anything over the SVN stream chunk size is not deltified.
> > >
> > > The default chunk size is 100 * 1024 right now.
> > > This will change when we have either a delta-combiner or skip deltas.
> > 
> > I don't think we need a delta combiner or skip deltas to change this. 
> 
> The reason its done is because getting old revisions is *way* too slow 
> otherwise.
> I tried to retrieve  revision 10 out of 1500 of a 1 meg changelog, and it 
> took >10 minutes

For comparison sake, the patch I use to make every 10th revision 
fulltext, lets me do the same retrieval basically instantly.

This is obviously with the check that causes things > chunk size to be 
fulltext'd, disabled, in both cases.

--Dan


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Deltas?

[Daniel Berlin <da...@dberlin.org>]

On Sun, 10 Mar 2002, Greg Hudson wrote:

> On Sun, 10 Mar 2002, Daniel Berlin wrote:
> > Right now, anything over the SVN stream chunk size is not deltified.
> >
> > The default chunk size is 100 * 1024 right now.
> > This will change when we have either a delta-combiner or skip deltas.
> 
> I don't think we need a delta combiner or skip deltas to change this. 

The reason its done is because getting old revisions is *way* too slow 
otherwise.
I tried to retrieve  revision 10 out of 1500 of a 1 meg changelog, and it 
took >10 minutes

It's not memory, its the time required to apply so many large deltas, one 
at a time.

--Dan


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Deltas?

[Greg Hudson <gh...@MIT.EDU>]

On Sun, 10 Mar 2002, Daniel Berlin wrote:
> Right now, anything over the SVN stream chunk size is not deltified.
>
> The default chunk size is 100 * 1024 right now.
> This will change when we have either a delta-combiner or skip deltas.

I don't think we need a delta combiner or skip deltas to change this.  It
shouldn't take any more memory to apply a delta to a large stream (that's
the whole point of the windows); it sounded like we had a bug which caused
memory to balloon in this case.  So we need to fix the bug (or verify that
it has already been fixed), which is orthogonal to features which limit
the number of delta applications to get to a revision.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Deltas?

[Stefan Boberg <sb...@team17.com>]

> Right now, anything over the SVN stream chunk size is not deltified.
> 
> The default chunk size is 100 * 1024 right now.
> This will change when we have either a delta-combiner or skip deltas.

  Ouch, I see... thanks for the information. Is there an ETA for this?
Also, can I change the stream chunk size to something very large without
breaking things? (I don't really care about memory usage... memory is
cheap).

  If not, I guess I will have to go ahead with my original plan and
implement my own temporary solution for now :(

Thanks,
Stef! :)
--
Stefan Boberg - R&D Manager, Team17 Software Ltd.
boberg@team17.com



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Deltas?

[Daniel Berlin <da...@dberlin.org>]

On Sun, 10 Mar 2002, Stefan Boberg wrote:

> > Now, all of this writing is happening in the *.db files in the
> > repository.  If you look at their sizes, they should barely grow at
> > all -- just by the size of the one-line delta.
> 
>   Yes, I understand that this is how it is supposed to work. But as far
> as I can see, this is not what is happening in my very basic tests. I
> understand that the log files are used by BDB for transaction
> journaling, and I can live with those. The real problem I have is that
> when I look at the database directory, the "strings" file - which I
> assume is the BDB table which stores the file deltas/data - grows much,
> much more than the delta data size.
> 
>   My test data is a large Maya 3D scene, "test.ma", saved out in ASCII
> format (file size = 49MB).


Right now, anything over the SVN stream chunk size is not deltified.

The default chunk size is 100 * 1024 right now.
This will change when we have either a delta-combiner or skip deltas.

--Dan


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Deltas?

[Stefan Boberg <sb...@team17.com>]

> Now, all of this writing is happening in the *.db files in the
> repository.  If you look at their sizes, they should barely grow at
> all -- just by the size of the one-line delta.

  Yes, I understand that this is how it is supposed to work. But as far
as I can see, this is not what is happening in my very basic tests. I
understand that the log files are used by BDB for transaction
journaling, and I can live with those. The real problem I have is that
when I look at the database directory, the "strings" file - which I
assume is the BDB table which stores the file deltas/data - grows much,
much more than the delta data size.

  My test data is a large Maya 3D scene, "test.ma", saved out in ASCII
format (file size = 49MB). I add the file to the repository, and as
expected the "strings" table data file grows to around 49-50MB. Now, I
move a single object in the scene along one axis and save the scene out
to a file again. I verify that the change is in TWO lines only using the
standard "diff" program. Now I proceed to commit my changes. As
expected, the log data generated is around 50MB, but the "strings" file
also grows by the same amount! (i.e. to around 100MB). Subsequent
modifications, no matter how small, generate exactly the same amount of
data - i.e. a full 49MB's worth of data.

  So as far as I can see, delta storage is not actually used in this
simple test case. Or, in case it is, it does not calculate the deltas
very well.

  As for the log.* files, I think it would be a very good idea to have
SVN delete them automatically... in my case especially, they could quite
easily dwarf the main repository data very quickly :)

Cheers,
Stef! :)
--
Stefan Boberg - R&D Manager, Team17 Software Ltd.
boberg@team17.com



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Deltas?

[Ben Collins-Sussman <su...@collab.net>]

"Stefan Boberg" <sb...@team17.com> writes:

>   SVN no longer generates massive amounts of log data, which was the
> main problem in my previous tests. However, it doesn't appear as though
> deltas are currently used? Whenever I commit even minute changes (like a
> single modified line for example), it appears as though the database
> grows by the size of the entire modified file on every commit! 

SVN does the same thing RCS does:  the latest version of a file is
stored as fulltext, and all previous versions are stored as deltas.

But to understand what's happening, you need to understand how
Berkeley DB works.  When you commit a one-line change to a file,
here's what happens:

   * the one-line change moves over the network
   * the change is applied to the latest version of the file.
   * the new version of the file is written as fulltext
   * the previous version of the file is converted from fulltext to a
     delta against the new fulltext.

Now, all of this writing is happening in the *.db files in the
repository.  If you look at their sizes, they should barely grow at
all -- just by the size of the one-line delta.

But because Berkeley is transacted, all writes are journaled in the
*.log files.  This allows Berkeley to replay transactions and recover
itself to particular checkpoints.  So what you're seeing is that the
logfile grows by the size of the newly-written fulltext.

Log-file management is the real issue here.  Once a log-file grows to
10 megs, a new log file is created, and the Berkeley tools usually
will tell you that the old 10 meg file can be thrown away.  (Another
thread has discussed the possibility of having the svn-repository
automatically delete old logfiles.)


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org