You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Hadmut Danisch <ha...@danisch.de> on 2005/12/07 15:43:20 UTC
Autoexpanding ZIP archives?
Hi,
sorry, if this was asked before, but I could not yet find anything
about that matter.
There are plenty of open source file formats (e.g. OpenDocument),
which look like a binary format, but are actually just a zip archive
of several files, mostly XML, but binaries like images etc. as well.
How would subversion treat these archives? As plain binaries?
If I correct just a typo in a large presentation, which is actually
just a changed line in one of the XML files, this impacs the whole zip
archive.
How would subversion deal with that? Would it store the complete
binary as a diff, just because on XML line had been changed?
What about auto-expanding zip archives and storing them as svn files,
and auto-compressing when exporting the repository?
regards
Hadmut
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Michael Sinz <Mi...@sinz.org>.
Peter N. Lundblad wrote:
> On Wed, 7 Dec 2005, Malcolm Rowe wrote:
>
>
>>On Wed, Dec 07, 2005 at 12:49:20PM -0500, John Peacock wrote:
>>Note that the same argument essentially applies to any structured
>>document for which there is a 'better' (more space efficient, better able
>>to represent changes) delta mechanism than a binary diff: zip files,
>>compound OLE documents (e.g., Microsoft Office files), XML files,
>>potentially even C code.
>>
>
> Why wouldn't the binary diff algorithm be suitable for XML documents?
I think people are talking by/past each other here. There are really
two different "diff" methods/algorithms involved here.
The one that is very visible is that of "svn diff" which currently only
works in line mode on text files. This is also the diff and merge/patch
mechanism within Subversion's merging feature.
There is also the diff "svndiff" mechanism(s) that are used in the backend
storage system. This algorithm is binary in nature and treats all files
the same. Its main purpose is to provide some level of delta compression
for the revision histories of files, be they text or xml or whatever.
On the front-end side of things, having better and more plugable diff
mechanisms (and merge/patch) would be a major win, especially when you
start dealing with specialized methods for certain file types.
The backend just needs to provide reasonable performance, reasonable
storage efficiency, and absolute reliability/reproduceability.
So, be careful when one talkes about XML documents vs binary diff vs
the line-by-line diff of "svn diff" since they are not all within the
same domain of the Subversion system/problem set.
--
Michael Sinz Technology and Engineering Director/Consultant
"Starting Startups" mailto:michael.sinz@sinz.org
My place on the web http://www.sinz.org/Michael.Sinz
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Wed, Dec 07, 2005 at 10:16:15PM +0100, Peter N. Lundblad wrote:
> On Wed, 7 Dec 2005, Malcolm Rowe wrote:
>
> > On Wed, Dec 07, 2005 at 12:49:20PM -0500, John Peacock wrote:
> > Note that the same argument essentially applies to any structured
> > document for which there is a 'better' (more space efficient, better able
> > to represent changes) delta mechanism than a binary diff: zip files,
> > compound OLE documents (e.g., Microsoft Office files), XML files,
> > potentially even C code.
> >
> Why wouldn't the binary diff algorithm be suitable for XML documents?
>
Oh, it is, of course, though there probably is a slightly 'better'
algorithm that can represent deltas in terms of the underlying DOM,
rather than just streams of bytes, that's all.
The XML and C examples were very much edge-case examples, since our
current representation is for all intents and purposes, fine.
Regards,
Malcolm
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2005-12-07 at 17:11 -0500, John Peacock wrote:
> Greg Hudson wrote:
> > Would these generate the exact same target file, byte for byte? If not,
> > they're probably not suitable for the job.
>
> No, I believe they would produce the corresponding changes to transform
> the source to the target. Any xmldiff program that could be used within
> Subversion would be a "XML context-sensitive diff" as opposed to GNU
> diff or Subversion's internal diff which is a "line sensitive diff" right?
No, you're wandering off-topic. This discussion is about content-aware
alternatives to svndiff for more efficient delta compression.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by John Peacock <jp...@rowman.com>.
Greg Hudson wrote:
> Would these generate the exact same target file, byte for byte? If not,
> they're probably not suitable for the job.
No, I believe they would produce the corresponding changes to transform
the source to the target. Any xmldiff program that could be used within
Subversion would be a "XML context-sensitive diff" as opposed to GNU
diff or Subversion's internal diff which is a "line sensitive diff" right?
> Even if we did allow fudging of the target file, I'm not sure it's safe
> to reorder attributes or sub-elements of an XML file,
My understanding (and I'd be more than happy to be corrected) was that
depending on the XML, there were certain bits that could be rearranged
and still be "equivalent" and other bits that couldn't. This was above
and beyond the "</tag> vs <tag></tag>" equivalence.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2005-12-07 at 16:28 -0500, John Peacock wrote:
> Peter N. Lundblad wrote:
> > Why wouldn't the binary diff algorithm be suitable for XML documents?
> It is, but that isn't the smallest possible diff for that document type,
> which could be generated by some magical xmldiff like one of these:
Would these generate the exact same target file, byte for byte? If not,
they're probably not suitable for the job.
Even if we did allow fudging of the target file, I'm not sure it's safe
to reorder attributes or sub-elements of an XML file, so I think all
you'd really get is whitespace-independence. Whitespace independence
can net you some substantial gains relative to a line-based diff
algorithm in pathological cases, but relative to a binary algorithm it
doesn't seem like such a win.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Marcus Rueckert <da...@web.de>.
On 2005-12-08 07:54:32 +0100, Andreas Magnusson wrote:
> OK, time for the stupid question.
> Wouldn't it be easier to teach the client program (for example
> OpenOffice) to be able to save in in a non-compressed way? An example
> (although NOT open in any way like OO) is Microsoft InfoPath whose doc
> is a cab file, consisting of xml files. This program can save
> uncompressed, probably so that you can store it in a VCS and really see
> the differences. Why not pose this req. on OpenOffice?
+1
darix
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Phillip Susi <ps...@cfl.rr.com>.
The reason that OO.o compresses the documents is because xml tends to be
very verbose in text form, so compressing it saves a TON of space. It
would be nice to be able to save all that space in the working copy by
leaving the files compressed, without having it foul up the deltas in
the repository. Best of both worlds that way.
Of course, if binary xml ( ASN.1 probably ) is ever adopted, this
problem could go away.
Andreas Magnusson wrote:
>
> OK, time for the stupid question.
> Wouldn't it be easier to teach the client program (for example
> OpenOffice) to be able to save in in a non-compressed way? An example
> (although NOT open in any way like OO) is Microsoft InfoPath whose doc
> is a cab file, consisting of xml files. This program can save
> uncompressed, probably so that you can store it in a VCS and really see
> the differences. Why not pose this req. on OpenOffice?
>
> /Andreas
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Andreas Magnusson <an...@home.se>.
Jim Blandy wrote:
> I dunno. I sympathize with the people who are complaining about the
> complexity. People have a right to expect that a VCS won't munge your
> stuff. But I think this issue is going to come up again. We're going
> to see a lot of people putting OO documents in Subversion
> repositories, and depending on the size of their project, it's not
> hard to imagine that some of them will complain about space usage.
>
> I'd be surprised if other version control systems did much better, though.
OK, time for the stupid question.
Wouldn't it be easier to teach the client program (for example
OpenOffice) to be able to save in in a non-compressed way? An example
(although NOT open in any way like OO) is Microsoft InfoPath whose doc
is a cab file, consisting of xml files. This program can save
uncompressed, probably so that you can store it in a VCS and really see
the differences. Why not pose this req. on OpenOffice?
/Andreas
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Jim Blandy <ji...@red-bean.com>.
I dunno. I sympathize with the people who are complaining about the
complexity. People have a right to expect that a VCS won't munge your
stuff. But I think this issue is going to come up again. We're going
to see a lot of people putting OO documents in Subversion
repositories, and depending on the size of their project, it's not
hard to imagine that some of them will complain about space usage.
I'd be surprised if other version control systems did much better, though.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by John Peacock <jp...@rowman.com>.
Peter N. Lundblad wrote:
> Hmmm, are we entering micro-optimization land now. Maybe the binary diff
> doesn't produce the smallest diff possible, but it must be much better
> than a context diff in this situation. I can understand why people want
> us to be more intelligent for .zip files, but not in this case.
Except that XML files are, typically, human readable code, yet a binary
diff would not be. You can use the '--diff-cmd' and '--extensions' to
produce diffs using an external tool, but if I remember correctly, this
is primarily intended to be used with GNU diff instead of the internal
diff. Being able to more easily use an external diff tool for merging
purposes has some utility.
All of this is orthogonal to the original discussion, which was how to
add sufficient pixie dust to Subversion to magically understand
different file formats and store the constituent elements separately
within the repository. Things have wandered a bit since then... ;)
John
p.s. please don't take offense, anyone who has been advocating for
client side hooks. I'm tired and just trying to add a little levity to
the discussion.
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Wed, 7 Dec 2005, John Peacock wrote:
> Peter N. Lundblad wrote:
> > Why wouldn't the binary diff algorithm be suitable for XML documents?
>
> It is, but that isn't the smallest possible diff for that document type,
> which could be generated by some magical xmldiff like one of these:
>
...
> all of which provide a context sensitive diff, based on the underlying
> XML structure. The binary diff algorithm doesn't know nothin' 'bout
> parsin' no XML files, so it is just a binary mapping between two
> collections of bytes.
>
Hmmm, are we entering micro-optimization land now. Maybe the binary diff
doesn't produce the smallest diff possible, but it must be much better
than a context diff in this situation. I can understand why people want
us to be more intelligent for .zip files, but not in this case.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by John Peacock <jp...@rowman.com>.
Peter N. Lundblad wrote:
> Why wouldn't the binary diff algorithm be suitable for XML documents?
It is, but that isn't the smallest possible diff for that document type,
which could be generated by some magical xmldiff like one of these:
http://www.logilab.org/projects/xmldiff/
http://apps.gotdotnet.com/xmltools/xmldiff/overview.html
http://software.decisionsoft.com/tools.html
http://forge.nesc.ac.uk/projects/xmldiff/
all of which provide a context sensitive diff, based on the underlying
XML structure. The binary diff algorithm doesn't know nothin' 'bout
parsin' no XML files, so it is just a binary mapping between two
collections of bytes.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Wed, 7 Dec 2005, Malcolm Rowe wrote:
> On Wed, Dec 07, 2005 at 12:49:20PM -0500, John Peacock wrote:
> Note that the same argument essentially applies to any structured
> document for which there is a 'better' (more space efficient, better able
> to represent changes) delta mechanism than a binary diff: zip files,
> compound OLE documents (e.g., Microsoft Office files), XML files,
> potentially even C code.
>
Why wouldn't the binary diff algorithm be suitable for XML documents?
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Marcus Rueckert <da...@web.de>.
On 2005-12-07 14:39:00 -0500, Phillip Susi wrote:
> Good point. Sometimes I will change a block of C code by wrapping it in
> a conditional or loop. The real change is to add maybe one or two
> lines, and add another tab to indent 50 lines one more level.
> Subversion diffs out and stores all 52 lines rather than just the 2 new
> lines, and recognizing that the other 50 lines just got another tab
> prefixed ( a one byte change per line, which is exactly the same on all
> 50 lines ). A smarter diff engine would be nice for that.
omg. yes please complicate all that more and more so we can finally
crunch it down to a "change description language", which will record
insert 2 lines and indent the lines in between ... i mean pulling in vim
or emacs into SVN would give that feature quite easily.
</ironic>
if you have really such disk space problems ... you should either
consider if need some more disks or think if you _really_ need all
your stuff under version control. otherwise i can just say:
keep it simple stupid.
just my 2 cents,
darix
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Max Bowsher <ma...@ukf.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Phillip Susi wrote:
> Good point. Sometimes I will change a block of C code by wrapping it in
> a conditional or loop. The real change is to add maybe one or two
> lines, and add another tab to indent 50 lines one more level. Subversion
> diffs out and stores all 52 lines rather than just the 2 new lines, and
> recognizing that the other 50 lines just got another tab prefixed ( a
> one byte change per line, which is exactly the same on all 50 lines ).
> A smarter diff engine would be nice for that.
*sigh*
It's quite irritating to see someone proposing a 'new feature', which is
actually almost exactly what Subversion has been doing since before 1.0.
For goodness sake, before you try to re-design something, understand how
it currently works!!!
Max.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Cygwin)
iD8DBQFDl2FdfFNSmcDyxYARAvD5AJ9KvqK4oKSX+nEVpEbmwM00d8+pfQCgmfgx
yjuur+hzmd6EqVyegVfwhXg=
=Fe0C
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Phillip Susi <ps...@cfl.rr.com>.
Good point. Sometimes I will change a block of C code by wrapping it in
a conditional or loop. The real change is to add maybe one or two
lines, and add another tab to indent 50 lines one more level.
Subversion diffs out and stores all 52 lines rather than just the 2 new
lines, and recognizing that the other 50 lines just got another tab
prefixed ( a one byte change per line, which is exactly the same on all
50 lines ). A smarter diff engine would be nice for that.
Malcolm Rowe wrote:
>
> The original poster was, I believe, talking about OpenOffice.org
> documents, which sound like a fine thing to store in a version control
> system.
>
> Note that the same argument essentially applies to any structured
> document for which there is a 'better' (more space efficient, better able
> to represent changes) delta mechanism than a binary diff: zip files,
> compound OLE documents (e.g., Microsoft Office files), XML files,
> potentially even C code.
>
> This is just the storage equivalent to the 'custom diff tool for different
> document types' issue.
>
> Regards,
> Malcolm
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Wed, Dec 07, 2005 at 12:49:20PM -0500, John Peacock wrote:
> Hadmut Danisch wrote:
> >The zip archives store everything with a modification date. So just
> >slight changes can cause large differenes. Keep in mind that LZ
> >compression is not byte aligned. A change of just a single bit in the
> >uncompressed source (e.g. the mod date of a directory) could leave the
> >10 MB unmodified in principle, but shift it by one bit. The full
> >archive is modified, 10 MB of storage wasted.
>
> The obvious question then being why are you storing this information in
> a version control system in the first place? Perhaps Subversion isn't
> the right tool for your problem?
>
The original poster was, I believe, talking about OpenOffice.org
documents, which sound like a fine thing to store in a version control
system.
Note that the same argument essentially applies to any structured
document for which there is a 'better' (more space efficient, better able
to represent changes) delta mechanism than a binary diff: zip files,
compound OLE documents (e.g., Microsoft Office files), XML files,
potentially even C code.
This is just the storage equivalent to the 'custom diff tool for different
document types' issue.
Regards,
Malcolm
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by John Peacock <jp...@rowman.com>.
Hadmut Danisch wrote:
> The zip archives store everything with a modification date. So just
> slight changes can cause large differenes. Keep in mind that LZ
> compression is not byte aligned. A change of just a single bit in the
> uncompressed source (e.g. the mod date of a directory) could leave the
> 10 MB unmodified in principle, but shift it by one bit. The full
> archive is modified, 10 MB of storage wasted.
The obvious question then being why are you storing this information in
a version control system in the first place? Perhaps Subversion isn't
the right tool for your problem?
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2005-12-07 at 11:47 -0600, Ben Collins-Sussman wrote:
> Yes, you've successfully described a possible pathological worst-case
> scenario. Nobody disagrees. What ghudson and I are saying is that on
> *average*, it will probably only be a 5MB delta, not a 10MB one.
No, that's not what I'm saying. :) I was disagreeing with you.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Ben Collins-Sussman <su...@red-bean.com>.
On 12/7/05, Hadmut Danisch <ha...@danisch.de> wrote:
> On Wed, Dec 07, 2005 at 10:17:22AM -0600, Ben Collins-Sussman wrote:
> >
> > Try it yourself: make a .zip presentation with 10MB of graphics and
> > commit it to an svn repository. Change one word in the presentation.
> > Now commit that new .zip to the repository. The repository's size
> > should only grow by the binary difference between the 2 zip files. I
> > don't believe that every single byte will be different, maybe 50% of
> > them will. So I would expect the repository to grow to 15MB, not
> > 20MB.
>
>
> The zip archives store everything with a modification date. So just
> slight changes can cause large differenes. Keep in mind that LZ
> compression is not byte aligned. A change of just a single bit in the
> uncompressed source (e.g. the mod date of a directory) could leave the
> 10 MB unmodified in principle, but shift it by one bit. The full
> archive is modified, 10 MB of storage wasted.
Yes, you've successfully described a possible pathological worst-case
scenario. Nobody disagrees. What ghudson and I are saying is that on
*average*, it will probably only be a 5MB delta, not a 10MB one.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Hadmut Danisch <ha...@danisch.de>.
On Wed, Dec 07, 2005 at 10:17:22AM -0600, Ben Collins-Sussman wrote:
>
> Try it yourself: make a .zip presentation with 10MB of graphics and
> commit it to an svn repository. Change one word in the presentation.
> Now commit that new .zip to the repository. The repository's size
> should only grow by the binary difference between the 2 zip files. I
> don't believe that every single byte will be different, maybe 50% of
> them will. So I would expect the repository to grow to 15MB, not
> 20MB.
The zip archives store everything with a modification date. So just
slight changes can cause large differenes. Keep in mind that LZ
compression is not byte aligned. A change of just a single bit in the
uncompressed source (e.g. the mod date of a directory) could leave the
10 MB unmodified in principle, but shift it by one bit. The full
archive is modified, 10 MB of storage wasted.
regards
Hadmut
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2005-12-07 at 19:14 +0100, Hadmut Danisch wrote:
> I think so. But as a person who mainly works in the field of security
> for about 15 years, I do not yet see why such transformations should
> directly imply security problems. Of course, poorly implemented
> scripts could, but that's not an argument. If an attacker has access
> to the repository, he could also modify source code and wait until
> someone checks out, compiles, and runs it.
Are you arguing on this basis that the server should be able to tell the
client to run arbitrary code? That's not acceptable. People use
Subversion to version things other than source code, and by running "svn
co" they aren't consenting to give the server full access to their
client machines.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2005-12-07 at 13:27 -0500, John Peacock wrote:
> And, as Greg points out, there are serious security issues outside of
> just the scope of how well written any client scripting may be. Any
> solution must permit the client code to ignore the server-provided
> scripts.
I'd go further, and say that the default configuration must not have
clients blindly running server-provided scripts.
I'm not wholly opposed to client-side hooks for this sort of thing, but
I think people who want them need to recognize that the best we could do
is a solution which (a) requires configuration on every client to make
work, and (b) is a nightmare in cross-platform environments.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by John Peacock <jp...@rowman.com>.
Hadmut Danisch wrote:
> On Wed, Dec 07, 2005 at 01:01:40PM -0500, John Peacock wrote:
>> 1) security;
>> 4) did I mention security?
>
> I think so. But as a person who mainly works in the field of security
> for about 15 years, I do not yet see why such transformations should
> directly imply security problems. Of course, poorly implemented
> scripts could, but that's not an argument.
The issue is that if the Subversion project supports client side
scripting then it is inherent in any such scheme that Subversion is
dependent on the various repository administrators to write competent
code and distribute it to their users.
I have to agree with the consensus opinion of the Subversion developers
that client side scripting is not likely to come about without a great
deal of consideration and planning. At the very least, server-mediated
configuration files (which would typically be used to distribute
something like client scripts) would have to be functional before any
design for client-side hooks could be considered.
And, as Greg points out, there are serious security issues outside of
just the scope of how well written any client scripting may be. Any
solution must permit the client code to ignore the server-provided
scripts. Plus there is the argument over whether to embed Python, Java,
or Perl in the Subversion client. ;)
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Hadmut Danisch <ha...@danisch.de>.
On Wed, Dec 07, 2005 at 01:01:40PM -0500, John Peacock wrote:
>
> 1) security;
> 4) did I mention security?
I think so. But as a person who mainly works in the field of security
for about 15 years, I do not yet see why such transformations should
directly imply security problems. Of course, poorly implemented
scripts could, but that's not an argument. If an attacker has access
to the repository, he could also modify source code and wait until
someone checks out, compiles, and runs it.
regards
Hadmut
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by John Peacock <jp...@rowman.com>.
Phillip Susi wrote:
> 2) The client configuration file would list acceptable filters, possibly
> on a per repository basis, and map them to the local executable that
> performs the filtering
And how are you establishing what "type" of file you are talking about,
since filenames are fungible. I could be perverse and force my
OpenOffice files to have the extension .ZIP (or .OOOZIP), I can do that,
but the built in handling you propose couldn't know that (yes, I know
that I have to go through special effort to then read in that file again).
If we were all using Amiga's and had the AIFF file format (IFF was
invented by Electronic Arts, FWIW), we could just call the system
library that checked for the AIFF header and determine what application
this binary belongs to (everything that didn't have a header was
automatically text). We don't have that luxury, and file signature
mapping is a black art at best (since we have already noted that
OpenOffice files are really zipfiles).
> 3) svn add already does some kind of checking to decide if the file
> is binary or text. This could be extended to recognize the mime type
> of the file in more detail, and based on settings in the client
> configuration file, automatically set the filter property based on
> the mime type.
And this has already been noted as being a less than complete solution
already. All in all, it seems like a very fragile feature that can be
handled more easily by a wrapper script on a site by site basis.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Phillip Susi <ps...@cfl.rr.com>.
I really like the idea of being able to configure the subversion client
to filter files through arbitrary programs, which could include
unzipping. I also don't see any security problems if it is done in a
sane way. I propose this:
1) Use a property to inform the client that it should use a given filter
on the file
2) The client configuration file would list acceptable filters, possibly
on a per repository basis, and map them to the local executable that
performs the filtering
3) svn add already does some kind of checking to decide if the file is
binary or text. This could be extended to recognize the mime type of
the file in more detail, and based on settings in the client
configuration file, automatically set the filter property based on the
mime type.
This would prevent the client from executing arbitrary code at the
direction of the server without the user's consent, but allow automatic
filtering. Obviously the client would need to be configured to handle
each filter used in the repository, but eventually commonly used filters
would probably come preconfigred with the subversion install.
In the event that a repository uses filters that do not come out of the
box, then the administrator would need to direct the clients of the
repository how to configure their system to use the filters. If the
client runs into a filter it does not know how to handle, then it could
fail and inform the user of the problem.
John Peacock wrote:
> Hadmut Danisch wrote:
>> Allow to configure arbitrary transformation programs (e.g. just a
>> shell script) on the client side, which recognize files on a pattern
>> base and run them through that filter instead of putting them directly
>> into that archive.
>
> Client side scripting has been discussed in the past; there are a couple
> of obvious problems:
>
> 1) security;
> 2) Subversion is a fully cross-platform tool and some people's computers
> are infested with OS's with incompetent shells and no robust scripting
> capability;
> 3) distribution of repository standard scripts to clients isn't
> available (though at least this is expected to change at some point);
> 4) did I mention security?
>
> There is nothing stopping you from producing a wrapper to the svn binary
> which does whatever massaging you want to the data before calling svn
> itself. If it is really useful in the general case, you can submit it
> to the Subversion project and it could live along side the other
> contrib/client-side tools.
>
> John
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by John Peacock <jp...@rowman.com>.
Hadmut Danisch wrote:
> Allow to configure arbitrary transformation programs (e.g. just a
> shell script) on the client side, which recognize files on a pattern
> base and run them through that filter instead of putting them directly
> into that archive.
Client side scripting has been discussed in the past; there are a couple
of obvious problems:
1) security;
2) Subversion is a fully cross-platform tool and some people's computers
are infested with OS's with incompetent shells and no robust scripting
capability;
3) distribution of repository standard scripts to clients isn't
available (though at least this is expected to change at some point);
4) did I mention security?
There is nothing stopping you from producing a wrapper to the svn binary
which does whatever massaging you want to the data before calling svn
itself. If it is really useful in the general case, you can submit it
to the Subversion project and it could live along side the other
contrib/client-side tools.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Hadmut Danisch <ha...@danisch.de>.
On Wed, Dec 07, 2005 at 12:26:10PM -0500, Greg Hudson wrote:
>
> Still, Hadmut, just because it's possible to do better here doesn't mean
> it's worthwhile. How would you feel if Subversion corrupted your data
> because it went through an obscure, poorly-tested code path to optimize
> storing it?
Bad, because that would be a poor way to implement it.
I'd propose a different way:
Allow to configure arbitrary transformation programs (e.g. just a
shell script) on the client side, which recognize files on a pattern
base and run them through that filter instead of putting them directly
into that archive.
As an example, I could have all files of pattern *.odp (opendocument)
not put in SVN directly, but zip-extracted. Simple task to write a
shell script to transform this. This would allow to store the contents
uncompressed (or compressed by file), while having the regular
opendocument file on the client side.
I could imagine even more examples for such transformers:
- Character Code conversion. E.g. svn server has UTF-8, I have
ISO-8859-1
- I'd love to put my Firefox bookmarks in a repository. Unfortunately,
it does not just contain the bookmarks, but access dates as well,
which causes plenty of changes just by using Firefox and the
bookmarks. I'd have a transformer which just removes that access
dates and leaves the bookmark information.
- Encryption (which is the opposite of my zip proposal):
Have the encrypted data on the SVN server, decrypt when downloading,
encrypt when committing.
- Digital Signatures
- Further scripting, e.g. postprocess configuration files stored in
SVN, restart a daemon, etc.
That way SVN wouldn't know anything about zip archives or
presentations. It's just my client who is configured to call a
particular program to do any magic transformation.
regards
Hadmut
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Peter Samuelson <pe...@p12n.org>.
[Greg Hudson]
> The compression result of two similar plaintexts doesn't generally
> look similar after the first difference, and sometimes before the
> first difference.
Yeah but we're talking about .zip here. The member files within a zip
archive are compressed individually. That's why .tar.gz, which uses
the same compression algorithm, produces slightly smaller files.
(OTOH, because of this, zip archives are random-access, and it's
possible to recover files stored *after* a point of file corruption.)
Re: Autoexpanding ZIP archives?
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2005-12-07 at 10:17 -0600, Ben Collins-Sussman wrote:
> On 12/7/05, Hadmut Danisch <ha...@danisch.de> wrote:
>
> > If I have a presentation with 10 MB of graphics, and enter just a
> > single word, SVN wastes another 10 MB just for every little change.
>
> I think you're exaggerating, this is the extreme worst case it could
> possibly ever be. It's what CVS does, in fact; it doesn't try to
> examine differences between binary files, it just stores each one in
> full.
The compression result of two similar plaintexts doesn't generally look
similar after the first difference, and sometimes before the first
difference. So I would expect at least a 5MB delta on average; quite
likely there would be some changed metadata at the beginning which would
make the entire new zipfile dissimilar to the old one.
Still, Hadmut, just because it's possible to do better here doesn't mean
it's worthwhile. How would you feel if Subversion corrupted your data
because it went through an obscure, poorly-tested code path to optimize
storing it? Some version control systems are moving in the direction of
not delta-compressing data at all, because disk space is cheap and code
complexity is expensive. I think that's too extreme (and git, the most
widely-used system which initially did this, has to the best of my
knowledge backtracked and started doing delta compression), but
"optimize every use case and damn the code size" is not the best
philosophy.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Daniel Berlin <db...@dberlin.org>.
On Wed, 2005-12-07 at 10:17 -0600, Ben Collins-Sussman wrote:
> On 12/7/05, Hadmut Danisch <ha...@danisch.de> wrote:
>
> > If I have a presentation with 10 MB of graphics, and enter just a
> > single word, SVN wastes another 10 MB just for every little change.
>
> I think you're exaggerating, this is the extreme worst case it could
> possibly ever be. It's what CVS does, in fact; it doesn't try to
> examine differences between binary files, it just stores each one in
> full.
>
> Try it yourself: make a .zip presentation with 10MB of graphics and
> commit it to an svn repository. Change one word in the presentation.
> Now commit that new .zip to the repository. The repository's size
> should only grow by the binary difference between the 2 zip files. I
> don't believe that every single byte will be different, maybe 50% of
> them will.
Let me go one further.
If you produce for me a testcase where the changed and original .zip
files are not actually different by more than 50%, and yet the delta
produced by the vdelta-test program in
subversion/tests/libsvn_delta/vdelta-test is generating diffs that are
ridiculously large, i will attempt to fix your problem in a sane way.
Note that stream alignment is not an easy issue. It's also probably
*not* worth it to do it from a maintenance, etc perspective.
But i'll happily take a look.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Ben Collins-Sussman <su...@red-bean.com>.
On 12/7/05, Hadmut Danisch <ha...@danisch.de> wrote:
> If I have a presentation with 10 MB of graphics, and enter just a
> single word, SVN wastes another 10 MB just for every little change.
I think you're exaggerating, this is the extreme worst case it could
possibly ever be. It's what CVS does, in fact; it doesn't try to
examine differences between binary files, it just stores each one in
full.
Try it yourself: make a .zip presentation with 10MB of graphics and
commit it to an svn repository. Change one word in the presentation.
Now commit that new .zip to the repository. The repository's size
should only grow by the binary difference between the 2 zip files. I
don't believe that every single byte will be different, maybe 50% of
them will. So I would expect the repository to grow to 15MB, not
20MB.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Hadmut Danisch <ha...@danisch.de>.
On Wed, Dec 07, 2005 at 10:55:37AM -0500, John Peacock wrote:
>
> XML files are not, /per se/, mergeable as text files, so they are
> treated as binary. Certain XML files are mergeable, but others can be
> reordered without changing the meaning at the XML level. There has been
> some discussion about using external XML diff tools to make automatic
> XML merging possible, but it hasn't gone anywhere yet (AFAICT).
>
> >How would subversion deal with that? Would it store the complete
> >binary as a diff, just because on XML line had been changed?
>
> Yes, because it is a binary file.
Even if XML files are treated as binary files, this would not
necessarily mean to treat the whole zip archive as a unit.
If I have a presentation with 10 MB of graphics, and enter just a
single word, SVN wastes another 10 MB just for every little change.
Since these file formats are not proprietary but open source, it would
be desirable to have better support.
regards
Hadmut
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by John Peacock <jp...@rowman.com>.
Hadmut Danisch wrote:
> If I correct just a typo in a large presentation, which is actually
> just a changed line in one of the XML files, this impacs the whole zip
> archive.
XML files are not, /per se/, mergeable as text files, so they are
treated as binary. Certain XML files are mergeable, but others can be
reordered without changing the meaning at the XML level. There has been
some discussion about using external XML diff tools to make automatic
XML merging possible, but it hasn't gone anywhere yet (AFAICT).
> How would subversion deal with that? Would it store the complete
> binary as a diff, just because on XML line had been changed?
Yes, because it is a binary file.
> What about auto-expanding zip archives and storing them as svn files,
> and auto-compressing when exporting the repository?
ETOOMAGICAL
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Yannick Menager <ya...@fastmail.fm>.
Hadmut Danisch <hadmut <at> danisch.de> writes:
> Hi,
> There are plenty of open source file formats (e.g. OpenDocument),
> which look like a binary format, but are actually just a zip archive
> of several files, mostly XML, but binaries like images etc. as well.
> How would subversion treat these archives? As plain binaries?
> What about auto-expanding zip archives and storing them as svn files,
> and auto-compressing when exporting the repository?
Hmm... That would indeed be a *very* nice feature. Automatic support for
compression/uncompression of that major compressions schemes that can be easily
incorporated. I'd love to see that happening
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Autoexpanding ZIP archives?
Posted by Ben Collins-Sussman <su...@red-bean.com>.
On 12/7/05, Hadmut Danisch <ha...@danisch.de> wrote:
> Hi,
>
> sorry, if this was asked before, but I could not yet find anything
> about that matter.
>
> There are plenty of open source file formats (e.g. OpenDocument),
> which look like a binary format, but are actually just a zip archive
> of several files, mostly XML, but binaries like images etc. as well.
>
>
> How would subversion treat these archives? As plain binaries?
>
Yes.
The server treats all files as opaque binary blobs of data, and stores
binary differences between successive versions of files.
The client treats files as either binary blobs, or as text, based on
the metadata you attach to a file. If the client thinks a file is
textual, it will attempt to display diffs and perform merges using
line-based contextual differencing algorithms. If the client thinks a
file is binary data, then it won't attempt to display diffs or perform
merges.
> If I correct just a typo in a large presentation, which is actually
> just a changed line in one of the XML files, this impacs the whole zip
> archive.
Yes, possibly changing more bytes then than the original 1-line change...
>
> How would subversion deal with that? Would it store the complete
> binary as a diff, just because on XML line had been changed?
>
Yes, the server would compare the 2 zip files, and store a binary diff.
>
> What about auto-expanding zip archives and storing them as svn files,
> and auto-compressing when exporting the repository?
>
Many would argue that this is a bad idea, that it's not the server's
job to understand file formats, that that way lies madness. Would
would be the purpose of all that complexity? To save a small bit of
repository disk space when versioning compressed archives? Is it
worth it?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org