You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Max Bowsher <ma...@ukf.net> on 2004/05/21 10:19:34 UTC
Subversion branch deltification policy is more space-hungry than CVS
Subversion stores more fulltexts than CVS, for the same history, if branching is used at all.
This becomes very visible when people convert from CVS to subversion, and their repositories increase significantly in size.
CVS stores 1 fulltext per file.
SVN stores 1 fulltext per file *plus* an additional fulltext for every branch on which a file is modified.
This is a tradeoff of space vs. checkout speed which is different to that made by CVS, and needs to be carefully considered, as it
may well dissuade people from switching from CVS to SVN.
Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry
than CVS
Posted by Greg Hudson <gh...@MIT.EDU>.
On Fri, 2004-05-21 at 07:02, Edmund Horner wrote:
> The trick is to do the deltas the other way round, isn't it? I read it
> in some file somewhere, but I couldn't easily find it in notes/ or
> libsvn_fs_base/notes/. But I think it's also what libsvn_fs_fs does:
I floated the idea in
http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgId=269155
FSFS does work this way, which is why the server has to do roughly twice
as much work for a head checkout (judging by the Subversion project
repository). Of course, the observed time for a checkout also involves
network transport and WC population, both of which generally take much
longer than the server work in my tests, so the wall-clock penalty is
pretty small.
A plaintext cache could eliminate the penalty, but it's not clear that
one is really needed, and it would add a great deal of complexity. If
someone actually tries switching from BDB to FSFS in the real world and
says, "well, I like the advantages, but it brings my server to its
knees," that would be a reason to think about designing a cache.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than
CVS
Posted by Edmund Horner <ch...@chrysophylax.cjb.net>.
Max Bowsher wrote:
> Subversion stores more fulltexts than CVS, for the same history, if branching is used at all.
>
> This becomes very visible when people convert from CVS to subversion, and their repositories increase significantly in size.
>
> CVS stores 1 fulltext per file.
>
> SVN stores 1 fulltext per file *plus* an additional fulltext for every branch on which a file is modified.
>
>
> This is a tradeoff of space vs. checkout speed which is different to that made by CVS, and needs to be carefully considered, as it
> may well dissuade people from switching from CVS to SVN.
The trick is to do the deltas the other way round, isn't it? I read it
in some file somewhere, but I couldn't easily find it in notes/ or
libsvn_fs_base/notes/. But I think it's also what libsvn_fs_fs does:
First revision is stored as plaintext. Succeeding revisions are stored
as deltas against earlier ones.
The tradeoff is that it's faster to commit (I think), but slower to read
new revisions (particularly HEAD). But that can be improved by a cache,
can't it?
Anyway without thinking about it too hard, I think it would also improve
the space usage of branches.
Hopefully someone can figure out where it's described properly by
whichever you you core devs thought of it.
Edmund.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry
than CVS
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2004-06-02 at 06:36, Vincent Lefevre wrote:
> On 2004-05-29 12:49:57 -0400, Greg Hudson wrote:
> > This argument makes perfect sense, but it turns out to be wrong. It is
> > only when lines of development diverge that you get multiple plaintext
> > representations of a file, not simply when a new copy-ID is created.
> [...]
>
> OK. So, if I understand, files from the "trunk" are not necessarily
> represented as plaintext in the repository
Correct.
> , and a checkout of the trunk may take a long time.
Probably not, even if there is a great deal of this going on, because
skip-deltas kick in at 32 revs of a file, and prevent the need for
assembling more than O(lg(n)) deltas.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by Vincent Lefevre <vi...@vinc17.org>.
On 2004-05-29 12:49:57 -0400, Greg Hudson wrote:
> This argument makes perfect sense, but it turns out to be wrong. It is
> only when lines of development diverge that you get multiple plaintext
> representations of a file, not simply when a new copy-ID is created.
[...]
OK. So, if I understand, files from the "trunk" are not necessarily
represented as plaintext in the repository, and a checkout of the
trunk may take a long time. However, I don't think such a problem
would be very common.
--
Vincent Lefèvre <vi...@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry
than CVS
Posted by Greg Hudson <gh...@MIT.EDU>.
On Fri, 2004-05-28 at 11:11, Vincent Lefevre wrote:
> If I've understood correctly, there seems to be a problem, though,
> when directories or files are moved/renamed. It has been said that
> this was implemented as a copy+delete. But Mark Benedetto King said
> that fulltexts live forever, even if one deletes the branch (here
> the branch would correspond to the original filename). And as the
> copy would normally be modified since it is the main file (under a
> new name), this would give a second fulltext.
This argument makes perfect sense, but it turns out to be wrong. It is
only when lines of development diverge that you get multiple plaintext
representations of a file, not simply when a new copy-ID is created. In
your explample, the node-rev history of the file might look something
like:
1.1.1 -- 1.1.2 -- 1.1.3
\
1.2.4 -- 1.2.5
(The first number is the node-ID, always 1 in this example, the second
number is the copy-ID, which in this example is 1 before the move and 2
after, and the third number is the txn-ID.) When 1.2.4 is committed,
the deltification process walks along the predecessor list without
respect to copy-ID changes, so 1.1.3 is deltified against 1.2.4.
If the file were not moved but were instead copied and then changed in
both locations, then the node-rev history might look like;
1.1.1 -- 1.1.2 -- 1.1.3 -- 1.1.6
\
1.2.4 -- 1.2.5
When 1.1.6 is committed, 1.1.3 is deltified against it, smashing the
previous delta representation against 1.2.4. This is not because 1.1.6
has the same copy-ID as 1.1.3, but merely because (in this example)
1.1.6 came along later. (Ideally, we might be able to notice that 1.1.3
already has a delta representation with the same distance, and pick
whichever delta representation is smaller. But we're not that smart,
and with the current schema we don't have any way of being that smart
because we don't have a distance field in representations.)
In case it hasn't filtered through, incidentally, FSFS does not have
multiple plaintexts even when lines of development diverge. So, in 1.1,
if users are seeing unacceptable expansions of their CVS repositories,
they will have an alternative which doesn't use as much space. (And
while it does use more time for a head checkout than BDB does, it scales
way better than CVS does for branches.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by Vincent Lefevre <vi...@vinc17.org>.
[Note: sorry if this has already been dealt with, but I have very
limited e-mail access until Monday.]
On 2004-05-26 20:57:20 -0400, John Peacock wrote:
> But the number of fulltexts generated per branch is going to vary
> wildly among various projects; in some cases the branch is going to
> be mostly node copies, in others it might have lots of modified
> files.
If I've understood correctly, there seems to be a problem, though,
when directories or files are moved/renamed. It has been said that
this was implemented as a copy+delete. But Mark Benedetto King said
that fulltexts live forever, even if one deletes the branch (here
the branch would correspond to the original filename). And as the
copy would normally be modified since it is the main file (under a
new name), this would give a second fulltext.
--
Vincent Lefèvre <vi...@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more
space-hungrythan CVS
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2004-05-26 at 18:05, Max Bowsher wrote:
> As for assessing the impact on a practical workload, I'm not really sure how
> best to do that, nor even how to define "practical workload".
"I performed this experiment with the Subversion project repository and
found that we could save X% space with an alternate branching policy,
while only suffering a Y% cost in branch checkout time..."
"At work, we use branching for every commit. By performing the same
sequences of branching and comitting in a CVS repository and an svn
repository, I found that svn consumes X% more space."
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungrythan CVS
Posted by Max Bowsher <ma...@ukf.net>.
Greg Hudson wrote:
> On Wed, 2004-05-26 at 17:13, Max Bowsher wrote:
>> 1. Create svn repository
>> 2. Add a few test files on /trunk
>> 3. Copy a branch
>> 4. Commit some branch modifications
>> 5. Examine the repository. Observe that there are multiple fulltexts
stored
>> per file. One for HEAD of /trunk, one for HEAD of /branches/foo.
>
> We knew that much already. The question is: how big of a deal is this
> for a practical workload? So far, the only data we have seems to be
> from cvs2svn conversions, and that's not very good data because we know
> that cvs2svn is inefficient in other ways.
Please leave cvs2svn out of this discussion. It's inefficiencies are small
and getting smaller, anyway, but this is a problem even if cvs2svn isn't
involved at all.
As for assessing the impact on a practical workload, I'm not really sure how
best to do that, nor even how to define "practical workload".
Essentially, I'm worried about svn repository administrators wanting to
discourage branching, to avoid repository bloat.
Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry
than CVS
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2004-05-26 at 17:13, Max Bowsher wrote:
> 1. Create svn repository
> 2. Add a few test files on /trunk
> 3. Copy a branch
> 4. Commit some branch modifications
> 5. Examine the repository. Observe that there are multiple fulltexts stored
> per file. One for HEAD of /trunk, one for HEAD of /branches/foo.
We knew that much already. The question is: how big of a deal is this
for a practical workload? So far, the only data we have seems to be
from cvs2svn conversions, and that's not very good data because we know
that cvs2svn is inefficient in other ways.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than
CVS
Posted by John Peacock <jp...@rowman.com>.
kfogel@collab.net wrote:
> John Peacock <jp...@rowman.com> writes:
>
>>As I understand the cvs2svn issue, when a branch is created, the files
>>are _not_ copied from the trunk, so you wind up having fulltexts for
>>all branch files, not just for modified branch files. That is a big
>>problem.
>
>
> No, no, absolutely not. cvs2svn creates branches by copying.
If I am misremembering things, I'm sorry. I thought there there was a
problem where cvs2svn wouldn't get that right (perhaps when revivifying
dead revisions).
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by kf...@collab.net.
John Peacock <jp...@rowman.com> writes:
> As I understand the cvs2svn issue, when a branch is created, the files
> are _not_ copied from the trunk, so you wind up having fulltexts for
> all branch files, not just for modified branch files. That is a big
> problem.
No, no, absolutely not. cvs2svn creates branches by copying.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than
CVS
Posted by John Peacock <jp...@rowman.com>.
Max Bowsher wrote:
> Ignore cvs2svn. It's irrelevant to this discussion. It was merely the
> discovery site for this problem. This is a problem for freshly started svn
> repositories with no previous history, just as much as for converted ones.
Except that conversion from cvs to svn (whatever the program used) is where the
effect is *most* noticable. Historically I believe that has been a byproduct of
the conversion process itself; if cvs2svn is better about making copies instead
of adding new files for branches (which is the goal, I know), that's wonderful.
But the number of fulltexts generated per branch is going to vary wildly among
various projects; in some cases the branch is going to be mostly node copies, in
others it might have lots of modified files. I don't think that you have
demonstrated that the use of fulltexts on copies is significant to outweigh the
performance gains by not having to regenerate the files on checkout from the deltas.
I fully and freely admit that fulltexts on modified copies will yield [slightly]
larger repositories. I just don't think you have demonstrated yet that it
matters in the grand scheme of things. And personally *I* am willing to trade
[some] diskspace for performance (which is where the current scheme comes from,
does it not?)...
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4720 Boston Way
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5747
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by Max Bowsher <ma...@ukf.net>.
John Peacock wrote:
> Max Bowsher wrote:
>
>> 5. Examine the repository. Observe that there are multiple fulltexts
stored
>> per file. One for HEAD of /trunk, one for HEAD of /branches/foo.
>
> Since "branch" is a concept that exists only in the mind of the developer,
> that makes sense. A copy of any file in the repository is a node copy
only
> until that file is changed, when it becomes a fully populated node, hence
a
> "second" fulltext. Unless you are saying that all copies with history
should
> be stored as a delta from the original, there is nothing odd about this
> revelation.
>
> For large repositories with lots of tags/branches, where the branches and
tags
> are largely unmodified copies, the space-savings (for node copies) should
> vastly outweigh any additional fulltexts for modified branch copies (with
> fulltexts).
Sorry, this is incorrect. CVS branching is cheap in diskspace too - just a
short piece of metadata per RCS file, marking the branchpoint. So, the
"space-savings for node copies", compared to CVS, are negligable. However,
since on first modification, svn creates a fulltext, whereas cvs creates a
delta, svn loses.
I guess I am saying that the use of deltification with copies needs to be
increased, if svn repository sizes are to compete with cvs.
> As I understand the cvs2svn issue, when a branch is created, the files are
> _not_ copied from the trunk, so you wind up having fulltexts for all
branch
> files, not just for modified branch files. That is a big problem.
Ignore cvs2svn. It's irrelevant to this discussion. It was merely the
discovery site for this problem. This is a problem for freshly started svn
repositories with no previous history, just as much as for converted ones.
(And the above paragraph about cvs2svn isn't actually correct, anyway.)
See my reply to ghudson for more.
Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than
CVS
Posted by John Peacock <jp...@rowman.com>.
Max Bowsher wrote:
> 5. Examine the repository. Observe that there are multiple fulltexts stored
> per file. One for HEAD of /trunk, one for HEAD of /branches/foo.
Since "branch" is a concept that exists only in the mind of the developer, that
makes sense. A copy of any file in the repository is a node copy only until
that file is changed, when it becomes a fully populated node, hence a "second"
fulltext. Unless you are saying that all copies with history should be stored
as a delta from the original, there is nothing odd about this revelation.
For large repositories with lots of tags/branches, where the branches and tags
are largely unmodified copies, the space-savings (for node copies) should vastly
outweigh any additional fulltexts for modified branch copies (with fulltexts).
As I understand the cvs2svn issue, when a branch is created, the files are _not_
copied from the trunk, so you wind up having fulltexts for all branch files, not
just for modified branch files. That is a big problem.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by Max Bowsher <ma...@ukf.net>.
kfogel@collab.net wrote:
> John Peacock <jp...@rowman.com> writes:
>> Have you factored out the relative inefficiency of cvs2svn (if that is
>> indeed how most people are converting their repositories)? I know
>> that there are some attempts to streamline the process, but that
>> people with lots of branches and tags were seeing [very] inefficient
>> usage due to the way that cvs2svn was structured (well, in fairness,
>> also due to the way that CVS was designed too).
Rather than factoring things out of cvs2svn, I simply used pure svn tools:
1. Create svn repository
2. Add a few test files on /trunk
3. Copy a branch
4. Commit some branch modifications
5. Examine the repository. Observe that there are multiple fulltexts stored
per file. One for HEAD of /trunk, one for HEAD of /branches/foo.
> The only reliable way to test the overhead of Subversion's branch
> storage is to start two repositories out empty, and make the same
> series of commits, branches, and branch commits to both.
>
> (Max may well have done this, however.)
I haven't done the equivalent CVS operations to compare numbers, but the
simple fact that there are excess fulltexts shows that the size will be
greater.
Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by kf...@collab.net.
John Peacock <jp...@rowman.com> writes:
> Have you factored out the relative inefficiency of cvs2svn (if that is
> indeed how most people are converting their repositories)? I know
> that there are some attempts to streamline the process, but that
> people with lots of branches and tags were seeing [very] inefficient
> usage due to the way that cvs2svn was structured (well, in fairness,
> also due to the way that CVS was designed too).
>
> When I convert my primary CVS repository to SVN in a couple of weeks,
> I'll try both vcp and cvs2svn as a comparison (don't really have time
> right now).
Good question, hadn't even thought of that.
The only reliable way to test the overhead of Subversion's branch
storage is to start two repositories out empty, and make the same
series of commits, branches, and branch commits to both.
(Max may well have done this, however.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than
CVS
Posted by John Peacock <jp...@rowman.com>.
Max Bowsher wrote:
>>Are you sure that this is the reason people's repositories are
>>increasing in overall size?
>
>
> I am sure that it is a major contributing factor. There may be other issues
> with the same symptom, but this is a real problem.
Have you factored out the relative inefficiency of cvs2svn (if that is indeed
how most people are converting their repositories)? I know that there are some
attempts to streamline the process, but that people with lots of branches and
tags were seeing [very] inefficient usage due to the way that cvs2svn was
structured (well, in fairness, also due to the way that CVS was designed too).
When I convert my primary CVS repository to SVN in a couple of weeks, I'll try
both vcp and cvs2svn as a comparison (don't really have time right now).
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4720 Boston Way
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5747
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry
than CVS
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2004-06-02 at 13:50, John Pybus wrote:
>> Only if people *are* getting bitten by this :-). We should first
>> make sure that this effect isn't lost in the noise of other
>> things.
> Reading the svn lists, people converting from CVS seem to think
> so.
If you've been following this thread, you should realize that cvs2svn
inefficiency is precisely the "noise of other things" which Karl is
referring to.
> If in case iii) the cost of accessing old nodes shared with trunk is
> acceptable, what is the benefit of storing fulltexts for the changed
> nodes even on long lived branches?
It means it's more efficient to work on a branch while the branch is
still being actively worked on.
But, really, the most important thing is that it flows naturally out of
the simplest implementation of deltification: walk some portion of the
node-rev's predecessor list and redeltify some number of predecessors
against the newly-committed node-rev.
Any other strategy is going to be significantly more complicated.
(Except for the FSFS strategy of making all deltas point backwards in
time. We could implement that in BDB with somewhat less code than we
have now. Existing repositories would become kind of weird, but they
would still work and be reasonable efficient even with a naive
implementation of the new strategy.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by kf...@collab.net.
John Pybus <jo...@zoology.oxford.ac.uk> writes:
> > Only if people *are* getting bitten by this :-). We should first make
> > sure that this effect isn't lost in the noise of other things.
>
> Reading the svn lists, people converting from CVS seem to think
> so. This would be because they will come in with long-lived
> repositories containing real examples of branch usage over time (of
> which there must be relatively few with SVN so far?), and because they
> have a definite size to compare to. People starting new projects, or
> not trying to import old history, wouldn't be expected to notice for
> some time, and having no hard number to compare wouldn't notice so
> strongly. (I'm ignoring all the cvs2svn inefficiency discussions)
Let's be careful:
People converting from CVS are noticing larger repositories. Whether
those large repositories are due to this, or to something else, or to
some combination of factors, is an open question.
> Surely having some exploration of the space of possible branch
> strategies, and the disk usage associated with it, will be more
> informative than none. Though clearly the usual warnings about
> synthetic benchmarking would apply.
Yes. (Who would disagree with that? :-) )
> If in case iii) the cost of accessing old nodes shared with trunk is
> acceptable, what is the benefit of storing fulltexts for the changed
> nodes even on long lived branches? On short lived branches, the cost
> of fulltexts born even when the branch is no longer required seem no
> more sensible.
That's an excellent question.
The answer, I think, is simply that this was the way things fell out
in the most obvious implementation. In other words, we didn't do
anything special to get this behavior, we simply use the same
deltification policy for all paths. If we were to change it, we would
probably be making the libsvn_fs code more complex (which is why I
don't want to do it without clear stats indicating that we need to).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than
CVS
Posted by John Pybus <jo...@zoology.oxford.ac.uk>.
kfogel@collab.net wrote:
> "Max Bowsher" <ma...@ukf.net> writes:
>
>>kfogel@collab.net wrote:
> Only if people *are* getting bitten by this :-). We should first make
> sure that this effect isn't lost in the noise of other things.
Reading the svn lists, people converting from CVS seem to think so.
This would be because they will come in with long-lived repositories
containing real examples of branch usage over time (of which there must
be relatively few with SVN so far?), and because they have a definite
size to compare to. People starting new projects, or not trying to
import old history, wouldn't be expected to notice for some time, and
having no hard number to compare wouldn't notice so strongly. (I'm
ignoring all the cvs2svn inefficiency discussions)
>>I'm willing to perform some experiments to try to quantify the size of the
>>problem, but I will need some help deciding what to test.
>
> I don't mean to sound snide by this at all, but: having a test for the
> proposed problem should be equivalent to fully understanding the
> problem in the first place. IOW, if we don't know exactly how to test
> for the problem, then what exactly *is* the problem? A precise
> description should be a blueprint for a test. If it's not, then maybe
> the problem hasn't really been precisely described.
Surely having some exploration of the space of possible branch
strategies, and the disk usage associated with it, will be more
informative than none. Though clearly the usual warnings about
synthetic benchmarking would apply.
I do have a slight worry about Subversion's deltification strategy,
though perhaps this was gone through back in the mists of time, and I
just don't understand.
Say a project branches at a stable release (1.0) and then continues to
make many changes to the head, while making fixes to the 1.0 branch
which only touch a few files. The the relative cost of operations would be:
i) Checkout of trunk: Cheap, full texts available[0], and common.
ii) Checkout of 1.0 tag: Relatively expensive, has to apply many
deltas[1], and not that likely, though maybe people would want fulltexts
as a target for diff.
iii) Checkout of 1.0 branch: Almost as expensive as ii) since most of
the nodes are shared with only a few are stored fulltext. More common
than ii) since developers patching and packaging need access to it,
maybe even deployers would be exporting.
[0]: Assuming BDB not FSFS here of course
[1]: Yes I know not as many as it might be; skip-lists etc
If in case iii) the cost of accessing old nodes shared with trunk is
acceptable, what is the benefit of storing fulltexts for the changed
nodes even on long lived branches? On short lived branches, the cost of
fulltexts born even when the branch is no longer required seem no more
sensible.
John
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by kf...@collab.net.
"Max Bowsher" <ma...@ukf.net> writes:
> kfogel@collab.net wrote:
> > My question is: is there reason to believe that, in practice, this
> > property of active Subversion branches is a significant factor in
> > increased repository size?
>
> It's not just _active_ branches, though. No space is saved when the branch
> ceases to be active, and becomes only historical record.
What I meant was, there will be no extra space used in the first
place, if the branch is *not* active. That is, just making the branch
doesn't cost you anything. It's when you start to use it that you get
the tip fulltexts, and even then you only get them for the files you
actually change, of course.
The extra fulltexts don't go away when you 'svn rm' the branch, nor
when you stop working on it, of course -- never meant to imply
otherwise.
> OK. So far, I've been viewing this as "we use more space than CVS, isn't
> that wrong?". I will stop considering the question from that viewpoint.
>
> If we are to deliberately make a different space/time tradeoff than CVS,
> then at very least there should be some documentation that we can point
> users who get bitten by this at, so they understand.
Only if people *are* getting bitten by this :-). We should first make
sure that this effect isn't lost in the noise of other things.
> I'm willing to perform some experiments to try to quantify the size of the
> problem, but I will need some help deciding what to test.
I don't mean to sound snide by this at all, but: having a test for the
proposed problem should be equivalent to fully understanding the
problem in the first place. IOW, if we don't know exactly how to test
for the problem, then what exactly *is* the problem? A precise
description should be a blueprint for a test. If it's not, then maybe
the problem hasn't really been precisely described.
> Here is a scenario particularly worrying to me:
> * Suppose you refactor a piece of software on a branch. All you need is one
> change to some function, which then needs to have all its callsites changed,
> and you have many tiny changes scattered over many files. Each file gets
> stored as a fulltext, rather than a tiny diff.
Yup. This is probably the worst-case scenario.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by Max Bowsher <ma...@ukf.net>.
This is a combined reply to two emails which raised similar points:
kfogel@collab.net wrote:
> My question is: is there reason to believe that, in practice, this
> property of active Subversion branches is a significant factor in
> increased repository size?
It's not just _active_ branches, though. No space is saved when the branch
ceases to be active, and becomes only historical record.
> This isn't a question of how the Subversion filesystem works, it's a
> question about statistical patterns in real-lifed repositories. For
> example, if it turns out that DB overhead is using 10x the space that
> these branch tip fulltexts are using, then optimizing active tips of
> branches would be pointless.
John Peacock wrote:
> Except that conversion from cvs to svn (whatever the program used) is
where
> the effect is *most* noticable. Historically I believe that has been a
> byproduct of the conversion process itself; if cvs2svn is better about
making
> copies instead of adding new files for branches (which is the goal, I
know),
> that's wonderful.
Indeed, cvs2svn is now much better about branching.
The effect is very likely to be most noticable when converting from cvs to
svn, because that causes the instant creation of a repository full of
history. The size of a subversion repository begun from scratch will
gradually diverge from the size of it's hypothetical cvs counterpart, at a
rate dependent on the usage of branching.
> But the number of fulltexts generated per branch is going to vary wildly
among
> various projects; in some cases the branch is going to be mostly node
copies,
> in others it might have lots of modified files. I don't think that you
have
> demonstrated that the use of fulltexts on copies is significant to
outweigh
> the performance gains by not having to regenerate the files on checkout
from
> the deltas.
>
> I fully and freely admit that fulltexts on modified copies will yield
> [slightly] larger repositories. I just don't think you have demonstrated
yet
> that it
> matters in the grand scheme of things. And personally *I* am willing to
trade
> [some] diskspace for performance (which is where the current scheme comes
> from, does it not?)...
OK. So far, I've been viewing this as "we use more space than CVS, isn't
that wrong?". I will stop considering the question from that viewpoint.
If we are to deliberately make a different space/time tradeoff than CVS,
then at very least there should be some documentation that we can point
users who get bitten by this at, so they understand.
I'm willing to perform some experiments to try to quantify the size of the
problem, but I will need some help deciding what to test.
Here is a scenario particularly worrying to me:
* Suppose you refactor a piece of software on a branch. All you need is one
change to some function, which then needs to have all its callsites changed,
and you have many tiny changes scattered over many files. Each file gets
stored as a fulltext, rather than a tiny diff.
Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by kf...@collab.net.
"Max Bowsher" <ma...@ukf.net> writes:
> Karl: Yes, I've seen one report of unexpectedly large size even with
> cvs2svn --trunk-only.
> However, all other factors aside, svn stores multiple fulltexts for a file
> when it is copied and then modified, whereas cvs stores only one fulltext.
Sure, I wasn't questioning that, it's obviously factually correct.
My question is: is there reason to believe that, in practice, this
property of active Subversion branches is a significant factor in
increased repository size?
This isn't a question of how the Subversion filesystem works, it's a
question about statistical patterns in real-lifed repositories. For
example, if it turns out that DB overhead is using 10x the space that
these branch tip fulltexts are using, then optimizing active tips of
branches would be pointless.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by Max Bowsher <ma...@ukf.net>.
Mark Benedetto King wrote:
> On Mon, May 24, 2004 at 10:34:26AM -0500, kfogel@collab.net wrote:
>> "Max Bowsher" <ma...@ukf.net> writes:
>>>>> This becomes very visible when people convert from CVS to subversion,
>>>>> and their repositories increase significantly in size.
>>>>
>>>> Are you sure that this is the reason people's repositories are
>>>> increasing in overall size?
>>>
>>> I am sure that it is a major contributing factor. There may be other
issues
>>> with the same symptom, but this is a real problem.
>>
>> Hmmm. I guess I'm surprised. We see repository size increases
>> (compared to CVS) even in repositories that don't have a lot of active
>> development on branches, or so I thought. I guess I'd want to do some
>> careful measuring -- or maybe you have already? What were the stats?
>
> IIUC, even if the branches aren't actively developed, once you've made
> changes on a branch those fulltexts live forever, even if you
> delete the branch (except on libsvn_fs_fs).
Karl: Yes, I've seen one report of unexpectedly large size even with
cvs2svn --trunk-only.
However, all other factors aside, svn stores multiple fulltexts for a file
when it is copied and then modified, whereas cvs stores only one fulltext.
Mark: Correct. Deleting the branch does not reclaim the fulltexts. (at least
for BDB)
Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by Mark Benedetto King <mb...@lowlatency.com>.
On Mon, May 24, 2004 at 10:34:26AM -0500, kfogel@collab.net wrote:
> "Max Bowsher" <ma...@ukf.net> writes:
> > >> This becomes very visible when people convert from CVS to subversion,
> > >> and their repositories increase significantly in size.
> > >
> > > Are you sure that this is the reason people's repositories are
> > > increasing in overall size?
> >
> > I am sure that it is a major contributing factor. There may be other issues
> > with the same symptom, but this is a real problem.
>
> Hmmm. I guess I'm surprised. We see repository size increases
> (compared to CVS) even in repositories that don't have a lot of active
> development on branches, or so I thought. I guess I'd want to do some
> careful measuring -- or maybe you have already? What were the stats?
>
IIUC, even if the branches aren't actively developed, once you've made
changes on a branch those fulltexts live forever, even if you
delete the branch (except on libsvn_fs_fs).
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than
CVS
Posted by Branko Čibej <br...@xbc.nu>.
Michael W Thelen wrote:
>* Michael W Thelen <th...@cs.utah.edu> [2004-05-25 15:43]:
>
>
>>I'm currently in the process of converting my company's CVS repository
>>to Subversion (hooray!). Using cvs2svn, the 4.4GB CVS repository
>>becomes an approximately 18GB Subversion repository
>>
>>
>
>Hmm, never mind. The strings file ended up being just 1.6GB (smaller
>than the original CVS repository!), and the rest was unused Berkeley DB
>logs.
>
>
Now _that_ is what we designed for. :-)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by Michael W Thelen <th...@cs.utah.edu>.
* Michael W Thelen <th...@cs.utah.edu> [2004-05-25 15:43]:
> I'm currently in the process of converting my company's CVS repository
> to Subversion (hooray!). Using cvs2svn, the 4.4GB CVS repository
> becomes an approximately 18GB Subversion repository
Hmm, never mind. The strings file ended up being just 1.6GB (smaller
than the original CVS repository!), and the rest was unused Berkeley DB
logs.
-- Mike
--
Michael W. Thelen
Nothing is more dangerous than an idea if it's the only one you have.
-- Emil-Auguste Chartier, "Propos sur la religion", 1938
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by Michael W Thelen <th...@cs.utah.edu>.
* kfogel@collab.net <kf...@collab.net> [2004-05-24 10:54]:
> "Max Bowsher" <ma...@ukf.net> writes:
> > >> This becomes very visible when people convert from CVS to subversion,
> > >> and their repositories increase significantly in size.
> > >
> > > Are you sure that this is the reason people's repositories are
> > > increasing in overall size?
> >
> > I am sure that it is a major contributing factor. There may be other issues
> > with the same symptom, but this is a real problem.
>
> Hmmm. I guess I'm surprised. We see repository size increases
> (compared to CVS) even in repositories that don't have a lot of active
> development on branches, or so I thought. I guess I'd want to do some
> careful measuring -- or maybe you have already? What were the stats?
I'm currently in the process of converting my company's CVS repository
to Subversion (hooray!). Using cvs2svn, the 4.4GB CVS repository
becomes an approximately 18GB Subversion repository (approximate because
it's still loading as I type this, but nearing completion). It's about
22,000 revisions over four years. We've hardly done any CVS branching
at all, but lots of tagging.
I hadn't heard of vcp before. I can try that out too and see how it
goes. If I can provide more information that's useful, feel free to ask
and I'll see what I can do to help.
-- Mike
--
Michael W. Thelen
... spell 'creat' with an E.
-- Ken Thompson
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by kf...@collab.net.
"Max Bowsher" <ma...@ukf.net> writes:
> >> This becomes very visible when people convert from CVS to subversion,
> >> and their repositories increase significantly in size.
> >
> > Are you sure that this is the reason people's repositories are
> > increasing in overall size?
>
> I am sure that it is a major contributing factor. There may be other issues
> with the same symptom, but this is a real problem.
Hmmm. I guess I'm surprised. We see repository size increases
(compared to CVS) even in repositories that don't have a lot of active
development on branches, or so I thought. I guess I'd want to do some
careful measuring -- or maybe you have already? What were the stats?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by Max Bowsher <ma...@ukf.net>.
kfogel@collab.net wrote:
> "Max Bowsher" <ma...@ukf.net> writes:
>> Subversion stores more fulltexts than CVS, for the same history, if
>> branching is used at all.
>>
>> This becomes very visible when people convert from CVS to subversion,
>> and their repositories increase significantly in size.
>
> Are you sure that this is the reason people's repositories are
> increasing in overall size?
I am sure that it is a major contributing factor. There may be other issues
with the same symptom, but this is a real problem.
Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion branch deltification policy is more space-hungry than CVS
Posted by kf...@collab.net.
"Max Bowsher" <ma...@ukf.net> writes:
> Subversion stores more fulltexts than CVS, for the same history, if
> branching is used at all.
>
> This becomes very visible when people convert from CVS to subversion,
> and their repositories increase significantly in size.
Are you sure that this is the reason people's repositories are
increasing in overall size?
-Karl
> CVS stores 1 fulltext per file.
>
> SVN stores 1 fulltext per file *plus* an additional fulltext for every
> branch on which a file is modified.
>
> This is a tradeoff of space vs. checkout speed which is different to
> that made by CVS, and needs to be carefully considered, as it may well
> dissuade people from switching from CVS to SVN.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org