You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by listman <li...@burble.net> on 2006/08/25 01:13:31 UTC
perforce versus subversion
it seems that people in denial regarding subversion performance? SVN
is very slow for certain kinds of repositories, esp those that
include large binary files. this is a fact and is openly discussed at
most of the companies i've dealt with. my last 3 companies have seen
10-20 min updates in situations, a fresh commit is similiar in
overhead. My point is that Subversion is great, unless you have large
binary files in your working env.
the real question is, when are we going to try and resolve the
redundant diff operations, and even give a switch to turn off the
diff for certain classes of file. then, maybe we'd get close to
perforce performance for large binary data.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: perforce versus subversion
Posted by Talden <ta...@gmail.com>.
> > Being able to supply an "svn:diffasnew" (or a hopefully better one)
> > keyword that instructed the client and the server to treat the file as
> > a complete replacement diff might perhaps be more straightforward.
> > This would not be too severe in terms of effort I expect and wouldn't
> > change anything in the backend storage.
>
> sounds great!
>
> unfortunately my binary files may change only slightly in the
> application that generates them, (eg: i move an inverter on a
> circuit schematic) but the resulting binary is *very* different
> to the previous. there's just no point in diffing these files, the
> delta will be close to just sending the new file over, with no
> processing..
>
> i think this is the case for a lot applications, we can't rely on
> a small change to a user data object showing up as a small
> change to the resulting binary db that stores this change.
I agree that for the vast majority of files diffs may very well not
save much space and are fairly likely to produce very
expensive reconstruct-from-diff operations.
It's likely that, with the relatively low costs of storage that
many people would opt to disable diffing on the binaries (at the
expense of transmission since you're no longer transmitting
only diffs between client and server).
> > ... (someone did mention that another diffing approach
> > was on the cards).
> what are the new diff changes by the way?
That one's beyond my rather recently acquired knowledge of
the Subversion development process (and people involved). It
was mentioned that a cheaper diffing approach existed (called
skip-diffs IIRC). Turn that smile upside-down though - I'm not
even sure that the thought had even begun to speculate about
the merest possibility of this making it's way into a release
any time soon (apologies in advance, for those reading this
backwards, for the blatant borrowing of part of a rather nice
quotation by Douglas Adams).
--
Talden
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: perforce versus subversion
Posted by Steinar Bang <sb...@dod.no>.
>>>>> listman <li...@burble.net>:
[snip!]
> the main discussion is on having a common database format, which is
> still going to be binary..
As long as we're talking about a new format here:
If that binary format is made rsync friendly, and gzip friendly, it
should also be Subversion binary diff friendly.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: perforce versus subversion
Posted by listman <li...@burble.net>.
On Aug 24, 2006, at 10:14 PM, Kenneth Porter wrote:
> --On Thursday, August 24, 2006 7:22 PM -0700 listman
> <li...@burble.net> wrote:
>
>> unfortunately my binary files may change only slightly in the
>> application that
>> generates them, (eg: i move an inverter on a circuit schematic)
>> but the
>> resulting binary is *very* different to the previous. there's just no
>> point in
>> diffing these files, the delta will be close to just sending the new
>> file over, with
>> no processing..
>
> Aren't more CAD systems switching to some kind of text format,
> perhaps stored compressed? Similar to OpenOffice, with its zipped
> XML files. If not, it might be a good time to ask the vendors to do
> that, not only to make Subversion's job easier, but to make sure
> your data isn't locked into a proprietary binary format.
>
nice idea, but the likelyhood of this happening anytime soon is very
minimal.
the main discussion is on having a common database format, which is
still going to be binary..
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: perforce versus subversion
Posted by Talden <ta...@gmail.com>.
That's the most powerful solution I've come up with too.
However I expect that Installing a new diff manager for a given type
or changing the type of a file would require that all of the past
diffs be reprocessed. Perhaps a little painful.
--
Talden
On 8/31/06, Kenneth Porter <sh...@sewingwitch.com> wrote:
> --On Tuesday, August 29, 2006 8:59 AM -0600 Steve Strobel
> <st...@link-comm.com> wrote:
>
> > Even if a small change to the CAD data caused only a small change in the
> > text format, wouldn't the zipped file often still be very different? It
> > seems to me that the only way this would be a significant advantage is if
> > the XML files were unzipped before being checked into Subversion.
>
> Good point. You'd want a standard compression format so that Subversion
> could unpack the archive and store a compressed diff of its contents. Which
> suggests that you need the ability to supply different diff systems for
> different MIME types.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: perforce versus subversion
Posted by Kenneth Porter <sh...@sewingwitch.com>.
--On Tuesday, August 29, 2006 8:59 AM -0600 Steve Strobel
<st...@link-comm.com> wrote:
> Even if a small change to the CAD data caused only a small change in the
> text format, wouldn't the zipped file often still be very different? It
> seems to me that the only way this would be a significant advantage is if
> the XML files were unzipped before being checked into Subversion.
Good point. You'd want a standard compression format so that Subversion
could unpack the archive and store a compressed diff of its contents. Which
suggests that you need the ability to supply different diff systems for
different MIME types.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: perforce versus subversion
Posted by Kenneth Porter <sh...@sewingwitch.com>.
--On Thursday, August 24, 2006 7:22 PM -0700 listman <li...@burble.net>
wrote:
> unfortunately my binary files may change only slightly in the
> application that
> generates them, (eg: i move an inverter on a circuit schematic) but the
> resulting binary is *very* different to the previous. there's just no
> point in
> diffing these files, the delta will be close to just sending the new
> file over, with
> no processing..
Aren't more CAD systems switching to some kind of text format, perhaps
stored compressed? Similar to OpenOffice, with its zipped XML files. If
not, it might be a good time to ask the vendors to do that, not only to
make Subversion's job easier, but to make sure your data isn't locked into
a proprietary binary format.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: perforce versus subversion
Posted by listman <li...@burble.net>.
On Aug 24, 2006, at 7:11 PM, Talden wrote:
> Being able to supply an "svn:diffasnew" (or a hopefully better one)
> keyword that instructed the client and the server to treat the file as
> a complete replacement diff might perhaps be more straightforward.
> This would not be too severe in terms of effort I expect and wouldn't
> change anything in the backend storage.
sounds great!
unfortunately my binary files may change only slightly in the
application that
generates them, (eg: i move an inverter on a circuit schematic) but the
resulting binary is *very* different to the previous. there's just no
point in
diffing these files, the delta will be close to just sending the new
file over, with
no processing..
i think this is the case for a lot applications, we can't rely on a
small change to a
user data object showing up as a small change to the resulting binary
db that
stores this change.
> Auto-adding these keywords based on rules for path and mime-type would
> make this relatively convenient to use until a more permanent solution
> comes along.
> It wouldn't resolve issues for people with large slow updating
> binaries in their repositories now, but then the approach also
> wouldn't prevent a better solution being introduced later (someone did
> mention that another diffing approach was on the cards).
>
it seems like an improvement to me, any developers care to comment?
what are the new diff changes by the way?
thx!
> --
> Talden
>
> On 8/25/06, listman <li...@burble.net> wrote:
>>
>> it seems that people in denial regarding subversion performance? SVN
>> is very slow for certain kinds of repositories, esp those that
>> include large binary files. this is a fact and is openly discussed at
>> most of the companies i've dealt with. my last 3 companies have seen
>> 10-20 min updates in situations, a fresh commit is similiar in
>> overhead. My point is that Subversion is great, unless you have large
>> binary files in your working env.
>>
>> the real question is, when are we going to try and resolve the
>> redundant diff operations, and even give a switch to turn off the
>> diff for certain classes of file. then, maybe we'd get close to
>> perforce performance for large binary data.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
>> For additional commands, e-mail: users-help@subversion.tigris.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: perforce versus subversion
Posted by Talden <ta...@gmail.com>.
Being able to supply an "svn:diffasnew" (or a hopefully better one)
keyword that instructed the client and the server to treat the file as
a complete replacement diff might perhaps be more straightforward.
This would not be too severe in terms of effort I expect and wouldn't
change anything in the backend storage.
Auto-adding these keywords based on rules for path and mime-type would
make this relatively convenient to use until a more permanent solution
comes along.
It wouldn't resolve issues for people with large slow updating
binaries in their repositories now, but then the approach also
wouldn't prevent a better solution being introduced later (someone did
mention that another diffing approach was on the cards).
--
Talden
On 8/25/06, listman <li...@burble.net> wrote:
>
> it seems that people in denial regarding subversion performance? SVN
> is very slow for certain kinds of repositories, esp those that
> include large binary files. this is a fact and is openly discussed at
> most of the companies i've dealt with. my last 3 companies have seen
> 10-20 min updates in situations, a fresh commit is similiar in
> overhead. My point is that Subversion is great, unless you have large
> binary files in your working env.
>
> the real question is, when are we going to try and resolve the
> redundant diff operations, and even give a switch to turn off the
> diff for certain classes of file. then, maybe we'd get close to
> perforce performance for large binary data.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: perforce versus subversion
Posted by Daniel Berlin <db...@dberlin.org>.
listman wrote:
> it seems that people in denial regarding subversion performance? SVN
> is very slow for certain kinds of repositories, esp those that
> include large binary files. this is a fact and is openly discussed at
> most of the companies i've dealt with. my last 3 companies have seen
> 10-20 min updates in situations, a fresh commit is similiar in
> overhead. My point is that Subversion is great, unless you have large
> binary files in your working env.
>
>
Without repositories we can use to reproduce the problem, we can't
really fix it.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org