You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by "Wadsworth, Eric (Contractor)" <wa...@fhu.disa.mil> on 2003/11/14 18:35:19 UTC
RE: Deltification (WAS: Re: Subversion 0.33.0 released).
> I also think that installing a post-commit hook is easy and should not
> be a major "barrier to entry" for most users. Certainly it's
> a heckuva
> lot easier than building subversion from source.
Don't mistake me for a complainer, but for those of us with tight
development schedules, who also have to maintain Subversion in addition to
their development work, having to dig around in the documentation to learn
about post-commit hooks is annoying.
I know that the document "INSTALLING SUBVERSION - A Quick Guide" is already
seventeen pages long, but since the installation process doesn't
automatically set up the deltification functionality, a little hand-holding
would be appreciated in the installation guide.
--- Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deltification (WAS: Re: Subversion 0.33.0 released).
Posted by kf...@collab.net.
Ryan Hunt <rh...@hp.com> writes:
> So should I expect that if I made a further change of just a few bytes
> and performed a check in that I would see the repository only grow by
> that much??
Er, "in the long run", yes. It might take two commits to get the full
benefit of the already-allocated space, I don't know.
(But any testing you have time for would be appreciated, of course!)
In issue #1573, we've discussed one possible solution to the
inefficiency you're noticing. From the issue:
> * a better, post-1.0 long-term solution: it takes N minutes for
> the client to figure out the correct vdelta data to send to the
> repository, then takes another N minutes for the repository to
> calculate the *same* delta in reverse! This is a huge waste of
> time. It would be nice, someday, to remember the vdelta
> windows and invert them when deltifying after the commit.
> (vdelta *is* invertible if you have both fulltexts, which the
> repository does.) And it would be very quick, too.
However, there are no plans to implement that right now. If the
problem you've pointed out -- temporary intermediate fulltexts --
starts to be a source of scalability trouble, we should open an issue,
with the above as one proposed solution.
Hope this helps,
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Deltification (WAS: Re: Subversion 0.33.0 released).
Posted by kf...@collab.net.
Ryan Hunt <rh...@hp.com> writes:
> So should I expect that if I made a further change of just a few bytes
> and performed a check in that I would see the repository only grow by
> that much??
Er, "in the long run", yes. It might take two commits to get the full
benefit of the already-allocated space, I don't know.
(But any testing you have time for would be appreciated, of course!)
In issue #1573, we've discussed one possible solution to the
inefficiency you're noticing. From the issue:
> * a better, post-1.0 long-term solution: it takes N minutes for
> the client to figure out the correct vdelta data to send to the
> repository, then takes another N minutes for the repository to
> calculate the *same* delta in reverse! This is a huge waste of
> time. It would be nice, someday, to remember the vdelta
> windows and invert them when deltifying after the commit.
> (vdelta *is* invertible if you have both fulltexts, which the
> repository does.) And it would be very quick, too.
However, there are no plans to implement that right now. If the
problem you've pointed out -- temporary intermediate fulltexts --
starts to be a source of scalability trouble, we should open an issue,
with the above as one proposed solution.
Hope this helps,
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deltification (WAS: Re: Subversion 0.33.0 released).
Posted by Ryan Hunt <rh...@hp.com>.
On Monday, November 17, 2003, at 02:01 PM, kfogel@collab.net wrote:
> Ryan Hunt <rh...@hp.com> writes:
>> The subsequent checkin took 160 min, and after removing the unused log
>> files the repository had grown by 1GB to 6GB indicating to me that the
>> entire 1GB file had been added to the repository rather than the
>> Delta.
>>
>> This is significant as I was using 0.29 not 0.33.
>>
>> So the question is, on large binaries is deltification being
>> implemented correctly at all?
>
> That depends on what you mean by "correctly". The entire 1GB will be
> (temporarily) in the repository, so BDB's on-disk storage will grow by
> that much. But that space will get reused by future commits, so over
> time this works out to a decent behavior. 0.33.* will behave the same
> as 0.29 in this respect (if you turn on deltification at all).
>
So should I expect that if I made a further change of just a few bytes
and performed a check in that I would see the repository only grow by
that much??
-Ryan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deltification (WAS: Re: Subversion 0.33.0 released).
Posted by Ryan Hunt <rh...@hp.com>.
On Monday, November 17, 2003, at 02:01 PM, kfogel@collab.net wrote:
> Ryan Hunt <rh...@hp.com> writes:
>> The subsequent checkin took 160 min, and after removing the unused log
>> files the repository had grown by 1GB to 6GB indicating to me that the
>> entire 1GB file had been added to the repository rather than the
>> Delta.
>>
>> This is significant as I was using 0.29 not 0.33.
>>
>> So the question is, on large binaries is deltification being
>> implemented correctly at all?
>
> That depends on what you mean by "correctly". The entire 1GB will be
> (temporarily) in the repository, so BDB's on-disk storage will grow by
> that much. But that space will get reused by future commits, so over
> time this works out to a decent behavior. 0.33.* will behave the same
> as 0.29 in this respect (if you turn on deltification at all).
>
So should I expect that if I made a further change of just a few bytes
and performed a check in that I would see the repository only grow by
that much??
-Ryan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Deltification (WAS: Re: Subversion 0.33.0 released).
Posted by kf...@collab.net.
Ryan Hunt <rh...@hp.com> writes:
> The subsequent checkin took 160 min, and after removing the unused log
> files the repository had grown by 1GB to 6GB indicating to me that the
> entire 1GB file had been added to the repository rather than the Delta.
>
> This is significant as I was using 0.29 not 0.33.
>
> So the question is, on large binaries is deltification being
> implemented correctly at all?
That depends on what you mean by "correctly". The entire 1GB will be
(temporarily) in the repository, so BDB's on-disk storage will grow by
that much. But that space will get reused by future commits, so over
time this works out to a decent behavior. 0.33.* will behave the same
as 0.29 in this respect (if you turn on deltification at all).
> Though one serious issue I have is that no matter what I can't seem to
> get more than 30MB per minute through put...
Your tests are almost certainly running into
http://subversion.tigris.org/issues/show_bug.cgi?id=1582
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deltification (WAS: Re: Subversion 0.33.0 released).
Posted by kf...@collab.net.
Ryan Hunt <rh...@hp.com> writes:
> The subsequent checkin took 160 min, and after removing the unused log
> files the repository had grown by 1GB to 6GB indicating to me that the
> entire 1GB file had been added to the repository rather than the Delta.
>
> This is significant as I was using 0.29 not 0.33.
>
> So the question is, on large binaries is deltification being
> implemented correctly at all?
That depends on what you mean by "correctly". The entire 1GB will be
(temporarily) in the repository, so BDB's on-disk storage will grow by
that much. But that space will get reused by future commits, so over
time this works out to a decent behavior. 0.33.* will behave the same
as 0.29 in this respect (if you turn on deltification at all).
> Though one serious issue I have is that no matter what I can't seem to
> get more than 30MB per minute through put...
Your tests are almost certainly running into
http://subversion.tigris.org/issues/show_bug.cgi?id=1582
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Deltification (WAS: Re: Subversion 0.33.0 released).
Posted by Ryan Hunt <rh...@hp.com>.
I am jumping on this thread a bit late, but have reason to believe the
deltification issue may be related to the problems I was reporting in
the thread: multiple access bog down on HP-UX 11i
I am trying to understand when the change was made to manual
deltification. Over the weekend I set up a repository on Mac OS X and
ran some tests. I set up 4 binary files for import. 3 of them
approximately 1GB and the 4th 2GB. Totaling 5GB These took 111 min. to
import.
I then checked out the entire set. about 59 min.
Then to one of the 1GB binary files I catted the single string "change"
increasing the size by just a couple of bytes.
The subsequent checkin took 160 min, and after removing the unused log
files the repository had grown by 1GB to 6GB indicating to me that the
entire 1GB file had been added to the repository rather than the Delta.
This is significant as I was using 0.29 not 0.33.
So the question is, on large binaries is deltification being
implemented correctly at all?
I plan to upgrade to 0.33 on my HP-UX box and see how performance
changes.
Though one serious issue I have is that no matter what I can't seem to
get more than 30MB per minute through put...
Any ideas on that?
-Ryan
On Friday, November 14, 2003, at 12:27 PM, kfogel@collab.net wrote:
> "Wadsworth, Eric (Contractor)" <wa...@fhu.disa.mil> writes:
>> Don't mistake me for a complainer, but for those of us with tight
>> development schedules, who also have to maintain Subversion in
>> addition to
>> their development work, having to dig around in the documentation to
>> learn
>> about post-commit hooks is annoying.
>>
>> I know that the document "INSTALLING SUBVERSION - A Quick Guide" is
>> already
>> seventeen pages long, but since the installation process doesn't
>> automatically set up the deltification functionality, a little
>> hand-holding
>> would be appreciated in the installation guide.
>
> Understandable, yes. I've added a new FAQ item about this, hope it
> helps:
>
> http://subversion.tigris.org/project_faq.html#deltification
>
> -Karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deltification (WAS: Re: Subversion 0.33.0 released).
Posted by Ryan Hunt <rh...@hp.com>.
I am jumping on this thread a bit late, but have reason to believe the
deltification issue may be related to the problems I was reporting in
the thread: multiple access bog down on HP-UX 11i
I am trying to understand when the change was made to manual
deltification. Over the weekend I set up a repository on Mac OS X and
ran some tests. I set up 4 binary files for import. 3 of them
approximately 1GB and the 4th 2GB. Totaling 5GB These took 111 min. to
import.
I then checked out the entire set. about 59 min.
Then to one of the 1GB binary files I catted the single string "change"
increasing the size by just a couple of bytes.
The subsequent checkin took 160 min, and after removing the unused log
files the repository had grown by 1GB to 6GB indicating to me that the
entire 1GB file had been added to the repository rather than the Delta.
This is significant as I was using 0.29 not 0.33.
So the question is, on large binaries is deltification being
implemented correctly at all?
I plan to upgrade to 0.33 on my HP-UX box and see how performance
changes.
Though one serious issue I have is that no matter what I can't seem to
get more than 30MB per minute through put...
Any ideas on that?
-Ryan
On Friday, November 14, 2003, at 12:27 PM, kfogel@collab.net wrote:
> "Wadsworth, Eric (Contractor)" <wa...@fhu.disa.mil> writes:
>> Don't mistake me for a complainer, but for those of us with tight
>> development schedules, who also have to maintain Subversion in
>> addition to
>> their development work, having to dig around in the documentation to
>> learn
>> about post-commit hooks is annoying.
>>
>> I know that the document "INSTALLING SUBVERSION - A Quick Guide" is
>> already
>> seventeen pages long, but since the installation process doesn't
>> automatically set up the deltification functionality, a little
>> hand-holding
>> would be appreciated in the installation guide.
>
> Understandable, yes. I've added a new FAQ item about this, hope it
> helps:
>
> http://subversion.tigris.org/project_faq.html#deltification
>
> -Karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Deltification (WAS: Re: Subversion 0.33.0 released).
Posted by kf...@collab.net.
"Wadsworth, Eric (Contractor)" <wa...@fhu.disa.mil> writes:
> Don't mistake me for a complainer, but for those of us with tight
> development schedules, who also have to maintain Subversion in addition to
> their development work, having to dig around in the documentation to learn
> about post-commit hooks is annoying.
>
> I know that the document "INSTALLING SUBVERSION - A Quick Guide" is already
> seventeen pages long, but since the installation process doesn't
> automatically set up the deltification functionality, a little hand-holding
> would be appreciated in the installation guide.
Understandable, yes. I've added a new FAQ item about this, hope it
helps:
http://subversion.tigris.org/project_faq.html#deltification
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Deltification (WAS: Re: Subversion 0.33.0 released).
Posted by kf...@collab.net.
"Wadsworth, Eric (Contractor)" <wa...@fhu.disa.mil> writes:
> Don't mistake me for a complainer, but for those of us with tight
> development schedules, who also have to maintain Subversion in addition to
> their development work, having to dig around in the documentation to learn
> about post-commit hooks is annoying.
>
> I know that the document "INSTALLING SUBVERSION - A Quick Guide" is already
> seventeen pages long, but since the installation process doesn't
> automatically set up the deltification functionality, a little hand-holding
> would be appreciated in the installation guide.
Understandable, yes. I've added a new FAQ item about this, hope it
helps:
http://subversion.tigris.org/project_faq.html#deltification
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org