You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Mike Samuel <mi...@gmail.com> on 2009/11/17 20:56:44 UTC
Merge mode (Was Classifying files as binary or text)
I thought I'd start a new thread about issue 1002 with a writeup of
Branko's merge mode property. The proposal is first, followed by the
same background as before and slightly updated goals.
Proposal:
========
(1) Add documentation on the svn:merge-mode property that lists the
allowed values as ("simple" and "none")
(2) Add example autoprops rules to the documentation that sets
svn:merge-mode to "simple" for the following file types
application/ecmascript
application/json
application/xml
image/svg+xml
(3) Change the text quoted from the SVN manual under Background to
read as below.
(4) Update the implementation to agree.
Subversion treats the following files as [[mergable]]:
* Files with no svn:mime-type [[and no svn:merge-mode]]
* Files with a svn:mime-type starting "text/"
* Files with a svn:mime-type equal to "image/x-xbitmap"
* Files with a svn:mime-type equal to "image/x-xpixmap"
* [[Files with a svn:merge-mode that is equal to "simple"]]
All other files are treated as [[unmergeable]], meaning that
Subversion will:
* Not attempt to automatically merge received changes with
local changes during svn update or svn merge
* Not show the differences as part of svn diff
* Not show line-by-line attribution for svn blame
In all other respects, Subversion treats [[mergable]] files the
same as [[unmergeable]] files, e.g. if you set
the svn:keywords or svn:eol-style properties, Subversion will
perform keyword substitution
or newline conversion on [[unmergeable]] files.
Goal:
====
To update the scheme by which svn {update,diff,merge,blame} to allow
merging of files
with svn:mime-type outside the hard-coded list currently used.
This determination should be independent of the platform svn
is running on, so independent of the set of supported character sets.
This scheme should not complicate future extensions to the merge
system which might wish to use a different merge policy, e.g. for XML
than for source code files.
This scheme should work with autoprops, and other mechanisms repository
administrators use to manage files. Specifically, some kinds of XML can
be meaningfully meged, and others cannot.
This scheme should work within existing limitations, such as the inability
to merge UTF-16 and UTF-32.
Background:
==========
The current behavior is described at
http://subversion.tigris.org/faq.html#binary-files
Subversion treats the following files as text:
* Files with no svn:mime-type
* Files with a svn:mime-type starting "text/"
* Files with a svn:mime-type equal to "image/x-xbitmap"
* Files with a svn:mime-type equal to "image/x-xpixmap"
All other files are treated as binary, meaning that Subversion will:
* Not attempt to automatically merge received changes with
local changes during svn update or svn merge
* Not show the differences as part of svn diff
* Not show line-by-line attribution for svn blame
In all other respects, Subversion treats binary files the same as
text files, e.g. if you set
the svn:keywords or svn:eol-style properties, Subversion will
perform keyword substitution
or newline conversion on binary files.
Common source code mime-types are misclassified, and that problem is
likely to grow because of current IANA policy.
Mime-types are handed out by the IANA, which only assigns text/*
mime-types for file-types that are meant to be human readable. Source
code is explicitly not considered human readable. This is why many
source code and data mime-types are in the application/* group or
other non text/* groups: application/json, application/ecmascript,
application/xml, image/svg+xml.
RFC 4288 ( ftp://ftp.rfc-editor.org/in-notes/rfc4288.txt ) says this
Expected uses for the "application" media type
include but are not limited to file transfer, spreadsheets,
presentations, scheduling data, and languages for "active"
(computational) material.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419155
Re: Merge mode (Was Classifying files as binary or text)
Posted by Mike Samuel <mi...@gmail.com>.
2009/11/17 B Smith-Mannschott <bs...@gmail.com>:
> On Tue, Nov 17, 2009 at 21:56, Mike Samuel <mi...@gmail.com> wrote:
>> I thought I'd start a new thread about issue 1002 with a writeup of
>> Branko's merge mode property. The proposal is first, followed by the
>> same background as before and slightly updated goals.
>
>> Proposal:
>> ========
>> (1) Add documentation on the svn:merge-mode property that lists the
>> allowed values as ("simple" and "none")
>> (2) Add example autoprops rules to the documentation that sets
>> svn:merge-mode to "simple" for the following file types
>> application/ecmascript
>> application/json
>> application/xml
>> image/svg+xml
>
> Strictly speaking, autoprops apply are triggered by file name
> patterns, not file [mime] types.
Quite right.
> [...snip...]
>
> Generally, I like the proposal:
>
> 1. It keeps svn:merge-mode orthogonal to related svn:keywords,
> svn:eol-style svn:mime-type properties.
>
> 2. The goals section strikes me as both simple and sufficient. (Less
> chance of unintended consequences.)
>
> // Ben
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419171
Re: Merge mode (Was Classifying files as binary or text)
Posted by "B. Smith-Mannschott" <bs...@gmail.com>.
On Tue, Nov 17, 2009 at 21:56, Mike Samuel <mi...@gmail.com> wrote:
> I thought I'd start a new thread about issue 1002 with a writeup of
> Branko's merge mode property. The proposal is first, followed by the
> same background as before and slightly updated goals.
> Proposal:
> ========
> (1) Add documentation on the svn:merge-mode property that lists the
> allowed values as ("simple" and "none")
> (2) Add example autoprops rules to the documentation that sets
> svn:merge-mode to "simple" for the following file types
> application/ecmascript
> application/json
> application/xml
> image/svg+xml
Strictly speaking, autoprops apply are triggered by file name
patterns, not file [mime] types.
[...snip...]
Generally, I like the proposal:
1. It keeps svn:merge-mode orthogonal to related svn:keywords,
svn:eol-style svn:mime-type properties.
2. The goals section strikes me as both simple and sufficient. (Less
chance of unintended consequences.)
// Ben
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419169
Re: Merge mode (Was Classifying files as binary or text)
Posted by Mark Phippard <ma...@gmail.com>.
On Tue, Nov 17, 2009 at 3:56 PM, Mike Samuel <mi...@gmail.com> wrote:
> I thought I'd start a new thread about issue 1002 with a writeup of
> Branko's merge mode property. The proposal is first, followed by the
> same background as before and slightly updated goals.
>
> Proposal:
> ========
> (1) Add documentation on the svn:merge-mode property that lists the
> allowed values as ("simple" and "none")
> (2) Add example autoprops rules to the documentation that sets
> svn:merge-mode to "simple" for the following file types
> application/ecmascript
> application/json
> application/xml
> image/svg+xml
> (3) Change the text quoted from the SVN manual under Background to
> read as below.
> (4) Update the implementation to agree.
>
> Subversion treats the following files as [[mergable]]:
>
> * Files with no svn:mime-type [[and no svn:merge-mode]]
> * Files with a svn:mime-type starting "text/"
> * Files with a svn:mime-type equal to "image/x-xbitmap"
> * Files with a svn:mime-type equal to "image/x-xpixmap"
> * [[Files with a svn:merge-mode that is equal to "simple"]]
>
> All other files are treated as [[unmergeable]], meaning that
> Subversion will:
>
> * Not attempt to automatically merge received changes with
> local changes during svn update or svn merge
> * Not show the differences as part of svn diff
> * Not show line-by-line attribution for svn blame
>
> In all other respects, Subversion treats [[mergable]] files the
> same as [[unmergeable]] files, e.g. if you set
> the svn:keywords or svn:eol-style properties, Subversion will
> perform keyword substitution
> or newline conversion on [[unmergeable]] files.
I am good with this. Proposal seems clear and well thought out.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419166
Re: Merge mode (Was Classifying files as binary or text)
Posted by Branko Cibej <br...@xbc.nu>.
Mike Samuel wrote:
> 2009/11/17 Martin Furter <mf...@rola.ch>:
>
>> On Wed, 18 Nov 2009, Branko Cibej wrote:
>>
>>
>>> Mike Samuel wrote:
>>>
>>>> 2009/11/17 Branko Čibej <br...@xbc.nu>:
>>>>
>>>>
>>>>> Mike Samuel wrote:
>>>>>
>>>>>
>>>>>> Source code is explicitly not considered human readable.
>>>>>>
>>>>>>
>>>>> Recalling some horrors I've had to review, I even tend to agree ... but
>>>>> it's a strange policy nonetheless. I wonder what kind of rationale they
>>>>> have for it.
>>>>>
>>>>>
>>>> I think making a distinction between human readable content and source
>>>> code is fine. But the distinction was not made consistently from the
>>>> beginning so it's led to confusion.
>>>>
>>>> And if they're going to make such a distinction, usually it helps to
>>>> distinguish between source code that's meant to be read and source
>>>> code for a language that has no separate compilation stage that is
>>>> meant to be delivered to an interpreter. I think this echoes bsmith's
>>>> two flavors of XML in the previous thread.
>>>>
>>>> What do you think of the property name, "svn:merge-mode," and the
>>>> values ("none" "simple")?
>>>>
>>>>
>>> The property name is descriptive enough, so is the "none"; don't have a
>>> quibble there. I do think "simple" is a bit too simple. :) It's a
>>> line-based contextual merge; dunno what would be a better short name for
>>> that, maybe "patch"?
>>>
>> Yeah, I thought the same.
>> What about "line-based"?
>> To me that sounds descriptive enough and not too long :)
>>
>
> Should we collect a bunch of names and put it to a vote?
> So far we have "line-based," "patch," and "simple."
>
I'm not too fond of putting UI design issues to a vote ... it's much
better if everyone just does what I say. :-D
> I guess I can start implementing in the meantime.
>
> Is svn_mime_type_is_binary in subversion/include/svn_types.h the right
> place to start?
>
It is.
> If so, I'll try renaming that
Before you start, I suggest you read:
http://svn.apache.org/repos/asf/subversion/trunk/www/hacking.html
Especially the part about "Release numbering, compatibility and
deprecation." That is a public function, you can't just rename it and
stay within our compatibility guidelines.
> and adding a second merge mode parameter
> and then threading the merge mode through from clients.
>
> I was looking under subversion/tests/libsvn_subr for tests of that
> function, but the word binary doesn't seem to exists anywhere there
> outside comments. Should I just create a new validate_test.c in that
> directory?
>
Try looking at subversion/tests/cmdline: autoprop_tests; blame_tests #2;
commit_tests #3; diff_tests #11, #26; merge_tests #12, #13, #22;
prop_tests #14; svnlook_tests #10; update_tests #1, #2.
Lots of such. :) For the cmdline tests, I suggest adding new ones so
that we continue testing the old behaviour of svn:mime-type without the
new property. Writing specific test s for that ln libsvn_subr would be
really nice, too.
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419350
Re: Merge mode (Was Classifying files as binary or text)
Posted by Mike Samuel <mi...@gmail.com>.
2009/11/17 Martin Furter <mf...@rola.ch>:
>
>
> On Wed, 18 Nov 2009, Branko Cibej wrote:
>
>> Mike Samuel wrote:
>>>
>>> 2009/11/17 Branko Čibej <br...@xbc.nu>:
>>>
>>>> Mike Samuel wrote:
>>>>
>>>>> Source code is explicitly not considered human readable.
>>>>>
>>>> Recalling some horrors I've had to review, I even tend to agree ... but
>>>> it's a strange policy nonetheless. I wonder what kind of rationale they
>>>> have for it.
>>>>
>>>
>>> I think making a distinction between human readable content and source
>>> code is fine. But the distinction was not made consistently from the
>>> beginning so it's led to confusion.
>>>
>>> And if they're going to make such a distinction, usually it helps to
>>> distinguish between source code that's meant to be read and source
>>> code for a language that has no separate compilation stage that is
>>> meant to be delivered to an interpreter. I think this echoes bsmith's
>>> two flavors of XML in the previous thread.
>>>
>>> What do you think of the property name, "svn:merge-mode," and the
>>> values ("none" "simple")?
>>>
>>
>> The property name is descriptive enough, so is the "none"; don't have a
>> quibble there. I do think "simple" is a bit too simple. :) It's a
>> line-based contextual merge; dunno what would be a better short name for
>> that, maybe "patch"?
>
> Yeah, I thought the same.
> What about "line-based"?
> To me that sounds descriptive enough and not too long :)
Should we collect a bunch of names and put it to a vote?
So far we have "line-based," "patch," and "simple."
I guess I can start implementing in the meantime.
Is svn_mime_type_is_binary in subversion/include/svn_types.h the right
place to start?
If so, I'll try renaming that and adding a second merge mode parameter
and then threading the merge mode through from clients.
I was looking under subversion/tests/libsvn_subr for tests of that
function, but the word binary doesn't seem to exists anywhere there
outside comments. Should I just create a new validate_test.c in that
directory?
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419251
Re: Merge mode (Was Classifying files as binary or text)
Posted by Martin Furter <mf...@rola.ch>.
On Wed, 18 Nov 2009, Branko Cibej wrote:
> Mike Samuel wrote:
>> 2009/11/17 Branko Čibej <br...@xbc.nu>:
>>
>>> Mike Samuel wrote:
>>>
>>>> Source code is explicitly not considered human readable.
>>>>
>>> Recalling some horrors I've had to review, I even tend to agree ... but
>>> it's a strange policy nonetheless. I wonder what kind of rationale they
>>> have for it.
>>>
>>
>> I think making a distinction between human readable content and source
>> code is fine. But the distinction was not made consistently from the
>> beginning so it's led to confusion.
>>
>> And if they're going to make such a distinction, usually it helps to
>> distinguish between source code that's meant to be read and source
>> code for a language that has no separate compilation stage that is
>> meant to be delivered to an interpreter. I think this echoes bsmith's
>> two flavors of XML in the previous thread.
>>
>> What do you think of the property name, "svn:merge-mode," and the
>> values ("none" "simple")?
>>
>
> The property name is descriptive enough, so is the "none"; don't have a
> quibble there. I do think "simple" is a bit too simple. :) It's a
> line-based contextual merge; dunno what would be a better short name for
> that, maybe "patch"?
Yeah, I thought the same.
What about "line-based"?
To me that sounds descriptive enough and not too long :)
Martin
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419232
Re: Merge mode (Was Classifying files as binary or text)
Posted by Branko Cibej <br...@xbc.nu>.
Mike Samuel wrote:
> 2009/11/17 Branko Čibej <br...@xbc.nu>:
>
>> Mike Samuel wrote:
>>
>>> Source code is explicitly not considered human readable.
>>>
>> Recalling some horrors I've had to review, I even tend to agree ... but
>> it's a strange policy nonetheless. I wonder what kind of rationale they
>> have for it.
>>
>
> I think making a distinction between human readable content and source
> code is fine. But the distinction was not made consistently from the
> beginning so it's led to confusion.
>
> And if they're going to make such a distinction, usually it helps to
> distinguish between source code that's meant to be read and source
> code for a language that has no separate compilation stage that is
> meant to be delivered to an interpreter. I think this echoes bsmith's
> two flavors of XML in the previous thread.
>
> What do you think of the property name, "svn:merge-mode," and the
> values ("none" "simple")?
>
The property name is descriptive enough, so is the "none"; don't have a
quibble there. I do think "simple" is a bit too simple. :) It's a
line-based contextual merge; dunno what would be a better short name for
that, maybe "patch"?
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419215
Re: Merge mode (Was Classifying files as binary or text)
Posted by Mike Samuel <mi...@gmail.com>.
2009/11/17 Branko Čibej <br...@xbc.nu>:
> Mike Samuel wrote:
>> Source code is explicitly not considered human readable.
>
> Recalling some horrors I've had to review, I even tend to agree ... but
> it's a strange policy nonetheless. I wonder what kind of rationale they
> have for it.
I think making a distinction between human readable content and source
code is fine. But the distinction was not made consistently from the
beginning so it's led to confusion.
And if they're going to make such a distinction, usually it helps to
distinguish between source code that's meant to be read and source
code for a language that has no separate compilation stage that is
meant to be delivered to an interpreter. I think this echoes bsmith's
two flavors of XML in the previous thread.
What do you think of the property name, "svn:merge-mode," and the
values ("none" "simple")?
> -- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419164
Re: Merge mode (Was Classifying files as binary or text)
Posted by Branko Cibej <br...@xbc.nu>.
Mike Samuel wrote:
> Source code is explicitly not considered human readable.
Recalling some horrors I've had to review, I even tend to agree ... but
it's a strange policy nonetheless. I wonder what kind of rationale they
have for it.
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419159
Re: Merge mode (Was Classifying files as binary or text)
Posted by Branko Cibej <br...@xbc.nu>.
Julian Foad wrote:
> Here is a vision of a solution that doesn't treat MIME-type as "special"
> but can still take advantage of it, both for 99% "Do The Right Thing"
> without manual setting of another property, and also for backward
> compatibility.
>
> (1) Use of line-wise merge shall be determined by a set of rules of the
> following kinds:
>
> * a specified property not existing, or existing and its value
> matching a specified pattern
>
> * the file name matching a specified pattern
>
> * a content scan determining whether the content is predominantly
> plain text
>
I'll point out that we already have all of that. :)
* "a specified property", currently svn:mime-type;
* "the file name", currently autoprops;
* "a content scan", see svn_io_detect_mimetype, for whatever that's
worth.
Though there's room for improvement of course.
I do not at disagree with these principles; but as usual, the devil is
in the details.
First a word about using file names for content type determination. As
others have pointed out, that's shaky ground, based on, e.g., XML
troubles. My own recent experience has to do with video files -- not
really a good example for "source control" -- and has taught me to
*never* take the file name (or extension) into account if you want to
preserve sanity.
This implies that the file name should be the *least* important
determinator of content type; an assumption that we happily break with
autoprops.
A content scan is a much better choice, and of course our current
content scan is "less than perfect." There are a lot of things we can
improve there, including using commonly available algorithms for
determining a file's content type (although for example the current
libmagic, which someone mentioned, is deficient:
$ file foo.txt
foo.txt: data
$ cat foo.txt
This is UTF-16BE text
One would expect the common UTF encodings to be recognized.)
Which leaves us with a property that takes precedence over all other
determinators. Which is what the proposal is all about, and what your
proposal transmogrified into. :)
Others have pointed out many problems with using and/or extending
svn:mime-type for this purpose. I'll add a couple more:
* IIRC iANA's MIME registry has different types registered to the
same file extensions. Coupled with the ambiguities of content scan
results -- and XML is a prime example of these -- you can use
neither file extension nor content scan to unambiguously set the
correct value, so you'd /have/ to set it manually. If you have to
manually set a property, it might as well be one that is less
ambiguous in the diff/merge/blame context than svn:mime-type.
* The type of a file's contents do not necessarily determine how a
user wants to merge the file. I can imagine having a file that's
essentially text, but must never be automatically merged -- some
kind of package manifest, for example.
> The rules for combining the rules are (something straight-forward, to be
> decided).
>
> The rules can be written to match svn:mime-type and svn:merge-mode, thus
> achieving the best results that mime-type matching can provide and yet
> also the explicit setting when it is wanted.
>
> (2) Use of line-wise diff shall be determined by a set of rules of the
> above kinds.
>
> (3) Use of line-wise blame shall be determined by a set of rules of the
> above kinds.
>
You imply that one might want diff/merge to behave differently from
blame. I don't think that's a reasonable thing to do. The answer to the
question, "who changed this bit of a file?" is very tightly coupled to:
"what changed between these two versions of a file?" If you use
different alogirthms for blame and diff, then blame results become even
fuzzier than they are now.
Take for example the following sequence of changes:
diff -u r1.xml r2.xml
--- r1.xml 2009-11-19 09:10:54.299911291 +0100
+++ r2.xml 2009-11-19 09:11:09.235528762 +0100
@@ -1,3 +1,3 @@
<p>
-this is text and more text
+this is text<br />and more text
</p>
$ diff -u r2.xml r3.xml
--- r2.xml 2009-11-19 09:11:09.235528762 +0100
+++ r3.xml 2009-11-19 09:11:19.223518197 +0100
@@ -1,3 +1,3 @@
<p>
-this is text<br />and more text
+this is text<br/>and more text
</p>
A line-based blame will flag the whole line as changed between r2 and
r3, but an XML-based diff will tell you that only the line-break was
changed, and also that the change has no semantic effect.
> (4) Default rules should be supplied.
>
> (5) The rules can be customised in the client's config. (Not that
> client-side config is ideal, but that's the kind of config we have and
> it is manageable.)
>
Have suitcase, will travel.
> (6) The other issue is whether to store the results of that
> determination in versioned properties on the file (svn:merge-mode,
> svn:diff-mode, svn:blame-mode). If we leave the determination till run
> time, then a client that has a better merge (or diff or blame) tool for
> certain file types will be able to use it.
>
I fail to see how that last bit is relevant. A client that has a better
merge or diff or blame tool will be able to use it regardless of what
properties we set or don't set on a file, or when or if we scan the
file's contents. It's safe to assume that any really smart diff tool
will have its own set of criteria for determining how to deal with a
file, and/or will allow the user to change its mode interactively. So it
can safely ignore whatever Subversion tells it about a file's type.
Unless you're thinking of teaching SVN to launch different diff tools
based on determined file type. Somehwat overengineered, IMHO, and anyone
can acheive the same effect today with a suitable wrapper.
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419953
Re: Merge mode (Was Classifying files as binary or text)
Posted by Branko Čibej <br...@xbc.nu>.
Julian Foad wrote:
> Here is a vision of a solution that doesn't treat MIME-type as "special"
> but can still take advantage of it, both for 99% "Do The Right Thing"
> without manual setting of another property, and also for backward
> compatibility.
>
> (1) Use of line-wise merge shall be determined by a set of rules of the
> following kinds:
>
> * a specified property not existing, or existing and its value
> matching a specified pattern
>
> * the file name matching a specified pattern
>
> * a content scan determining whether the content is predominantly
> plain text
>
I'll point out that we already have all of that. :)
* "a specified property", currently svn:mime-type;
* "the file name", currently autoprops;
* "a content scan", see svn_io_detect_mimetype, for whatever that's
worth.
Though there's room for improvement of course.
I do not at disagree with these principles; but as usual, the devil is
in the details.
First a word about using file names for content type determination. As
others have pointed out, that's shaky ground, based on, e.g., XML
troubles. My own recent experience has to do with video files -- not
really a good example for "source control" -- and has taught me to
*never* take the file name (or extension) into account if you want to
preserve sanity.
This implies that the file name should be the *least* important
determinator of content type; an assumption that we happily break with
autoprops.
A content scan is a much better choice, and of course our current
content scan is "less than perfect." There are a lot of things we can
improve there, including using commonly available algorithms for
determining a file's content type (although for example the current
libmagic, which someone mentioned, is deficient:
$ file foo.txt
foo.txt: data
$ cat foo.txt
This is UTF-16BE text
One would expect the common UTF encodings to be recognized.)
Which leaves us with a property that takes precedence over all other
determinators. Which is what the proposal is all about, and what your
proposal transmogrified into. :)
Others have pointed out many problems with using and/or extending
svn:mime-type for this purpose. I'll add a couple more:
* IIRC iANA's MIME registry has different types registered to the
same file extensions. Coupled with the ambiguities of content scan
results -- and XML is a prime example of these -- you can use
neither file extension nor content scan to unambiguously set the
correct value, so you'd /have/ to set it manually. If you have to
manually set a property, it might as well be one that is less
ambiguous in the diff/merge/blame context than svn:mime-type.
* The type of a file's contents do not necessarily determine how a
user wants to merge the file. I can imagine having a file that's
essentially text, but must never be automatically merged -- some
kind of package manifest, for example.
> The rules for combining the rules are (something straight-forward, to be
> decided).
>
> The rules can be written to match svn:mime-type and svn:merge-mode, thus
> achieving the best results that mime-type matching can provide and yet
> also the explicit setting when it is wanted.
>
> (2) Use of line-wise diff shall be determined by a set of rules of the
> above kinds.
>
> (3) Use of line-wise blame shall be determined by a set of rules of the
> above kinds.
>
You imply that one might want diff/merge to behave differently from
blame. I don't think that's a reasonable thing to do. The answer to the
question, "who changed this bit of a file?" is very tightly coupled to:
"what changed between these two versions of a file?" If you use
different alogirthms for blame and diff, then blame results become even
fuzzier than they are now.
Take for example the following sequence of changes:
diff -u r1.xml r2.xml
--- r1.xml 2009-11-19 09:10:54.299911291 +0100
+++ r2.xml 2009-11-19 09:11:09.235528762 +0100
@@ -1,3 +1,3 @@
<p>
-this is text and more text
+this is text<br />and more text
</p>
$ diff -u r2.xml r3.xml
--- r2.xml 2009-11-19 09:11:09.235528762 +0100
+++ r3.xml 2009-11-19 09:11:19.223518197 +0100
@@ -1,3 +1,3 @@
<p>
-this is text<br />and more text
+this is text<br/>and more text
</p>
A line-based blame will flag the whole line as changed between r2 and
r3, but an XML-based diff will tell you that only the line-break was
changed, and also that the change has no semantic effect.
> (4) Default rules should be supplied.
>
> (5) The rules can be customised in the client's config. (Not that
> client-side config is ideal, but that's the kind of config we have and
> it is manageable.)
>
Have suitcase, will travel.
> (6) The other issue is whether to store the results of that
> determination in versioned properties on the file (svn:merge-mode,
> svn:diff-mode, svn:blame-mode). If we leave the determination till run
> time, then a client that has a better merge (or diff or blame) tool for
> certain file types will be able to use it.
>
I fail to see how that last bit is relevant. A client that has a better
merge or diff or blame tool will be able to use it regardless of what
properties we set or don't set on a file, or when or if we scan the
file's contents. It's safe to assume that any really smart diff tool
will have its own set of criteria for determining how to deal with a
file, and/or will allow the user to change its mode interactively. So it
can safely ignore whatever Subversion tells it about a file's type.
Unless you're thinking of teaching SVN to launch different diff tools
based on determined file type. Somehwat overengineered, IMHO, and anyone
can acheive the same effect today with a suitable wrapper.
-- Brane
Re: Merge mode (Was Classifying files as binary or text)
Posted by Julian Foad <ju...@btopenworld.com>.
B. Smith-Mannschott wrote:
> After many years of using Subversion, I've come to the conclusion that
> svn:mime-type is best understood as a way to keep Apache happy and
> able to support read-only repository browsing. The fact that this
> property is also used to determine ability to merge, diff, and blame
> is an unfortunate wart.
Ah, yes, it is illogical to design the system such that mergeability
shall be determined from the MIME type alone. That is certainly not, and
never can be, a 100% indicator of mergeability.
I think there are two issues here.
(1) Subversion's MIME type recognition is severely deficient (lots of
common textual types not recognized), causing the use of MIME type for
this purpose to be seen as a poor choice of design. Would you ever have
considered it a "wart" if all registered MIME types were recognized so
that the current mechanism worked well in nearly all cases?
(2) MIME type can never be a 100% indicator of line-based text, much
less of line-wise mergability. It is just a useful clue that can get us
maybe 98% there. To get 100% there, other indicators are needed too,
including a way to independently and specifically indicate
blame/diff/merge behaviour to Subversion.
Here is a vision of a solution that doesn't treat MIME-type as "special"
but can still take advantage of it, both for 99% "Do The Right Thing"
without manual setting of another property, and also for backward
compatibility.
(1) Use of line-wise merge shall be determined by a set of rules of the
following kinds:
* a specified property not existing, or existing and its value
matching a specified pattern
* the file name matching a specified pattern
* a content scan determining whether the content is predominantly
plain text
The rules for combining the rules are (something straight-forward, to be
decided).
The rules can be written to match svn:mime-type and svn:merge-mode, thus
achieving the best results that mime-type matching can provide and yet
also the explicit setting when it is wanted.
(2) Use of line-wise diff shall be determined by a set of rules of the
above kinds.
(3) Use of line-wise blame shall be determined by a set of rules of the
above kinds.
(4) Default rules should be supplied.
(5) The rules can be customised in the client's config. (Not that
client-side config is ideal, but that's the kind of config we have and
it is manageable.)
(6) The other issue is whether to store the results of that
determination in versioned properties on the file (svn:merge-mode,
svn:diff-mode, svn:blame-mode). If we leave the determination till run
time, then a client that has a better merge (or diff or blame) tool for
certain file types will be able to use it.
- Julian
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419446
Re: Merge mode (Was Classifying files as binary or text)
Posted by Julian Foad <ju...@btopenworld.com>.
B. Smith-Mannschott wrote:
> After many years of using Subversion, I've come to the conclusion that
> svn:mime-type is best understood as a way to keep Apache happy and
> able to support read-only repository browsing. The fact that this
> property is also used to determine ability to merge, diff, and blame
> is an unfortunate wart.
Ah, yes, it is illogical to design the system such that mergeability
shall be determined from the MIME type alone. That is certainly not, and
never can be, a 100% indicator of mergeability.
I think there are two issues here.
(1) Subversion's MIME type recognition is severely deficient (lots of
common textual types not recognized), causing the use of MIME type for
this purpose to be seen as a poor choice of design. Would you ever have
considered it a "wart" if all registered MIME types were recognized so
that the current mechanism worked well in nearly all cases?
(2) MIME type can never be a 100% indicator of line-based text, much
less of line-wise mergability. It is just a useful clue that can get us
maybe 98% there. To get 100% there, other indicators are needed too,
including a way to independently and specifically indicate
blame/diff/merge behaviour to Subversion.
Here is a vision of a solution that doesn't treat MIME-type as "special"
but can still take advantage of it, both for 99% "Do The Right Thing"
without manual setting of another property, and also for backward
compatibility.
(1) Use of line-wise merge shall be determined by a set of rules of the
following kinds:
* a specified property not existing, or existing and its value
matching a specified pattern
* the file name matching a specified pattern
* a content scan determining whether the content is predominantly
plain text
The rules for combining the rules are (something straight-forward, to be
decided).
The rules can be written to match svn:mime-type and svn:merge-mode, thus
achieving the best results that mime-type matching can provide and yet
also the explicit setting when it is wanted.
(2) Use of line-wise diff shall be determined by a set of rules of the
above kinds.
(3) Use of line-wise blame shall be determined by a set of rules of the
above kinds.
(4) Default rules should be supplied.
(5) The rules can be customised in the client's config. (Not that
client-side config is ideal, but that's the kind of config we have and
it is manageable.)
(6) The other issue is whether to store the results of that
determination in versioned properties on the file (svn:merge-mode,
svn:diff-mode, svn:blame-mode). If we leave the determination till run
time, then a client that has a better merge (or diff or blame) tool for
certain file types will be able to use it.
- Julian
Re: Merge mode (Was Classifying files as binary or text)
Posted by B Smith-Mannschott <bs...@gmail.com>.
On Wed, Nov 18, 2009 at 08:25, David Glasser <gl...@davidglasser.net> wrote:
> I'm not a MIME expert, but isn't there a thing where you can add
> key/value pairs after a MIME type which customize its meaning? Maybe
> that could be used for this?
Please, no.
Isn't the fact that charset is munged into svn:mime-type already
sufficiently ugly? Placing these two independent values into the same
property together is a hindrance to scriptability. It clashed badly
with autoprops syntax for *years* (fixed in 1.7, I hear).You're
advocating more of the same?
Maybe it's just me, but I find it easier to work with systems that:
- separate concerns
- keep their features orthogonal
- keep atomic values atomic, rather than munging them together into
arbitrary blobs
(In this, I guess I'm obeying a kind of "normal form").
After many years of using Subversion, I've come to the conclusion that
svn:mime-type is best understood as a way to keep Apache happy and
able to support read-only repository browsing. The fact that this
property is also used to determine ability to merge, diff, and blame
is an unfortunate wart.
// Ben
Re: Merge mode (Was Classifying files as binary or text)
Posted by "B. Smith-Mannschott" <bs...@gmail.com>.
On Wed, Nov 18, 2009 at 08:25, David Glasser <gl...@davidglasser.net> wrote:
> I'm not a MIME expert, but isn't there a thing where you can add
> key/value pairs after a MIME type which customize its meaning? Maybe
> that could be used for this?
Please, no.
Isn't the fact that charset is munged into svn:mime-type already
sufficiently ugly? Placing these two independent values into the same
property together is a hindrance to scriptability. It clashed badly
with autoprops syntax for *years* (fixed in 1.7, I hear).You're
advocating more of the same?
Maybe it's just me, but I find it easier to work with systems that:
- separate concerns
- keep their features orthogonal
- keep atomic values atomic, rather than munging them together into
arbitrary blobs
(In this, I guess I'm obeying a kind of "normal form").
After many years of using Subversion, I've come to the conclusion that
svn:mime-type is best understood as a way to keep Apache happy and
able to support read-only repository browsing. The fact that this
property is also used to determine ability to merge, diff, and blame
is an unfortunate wart.
// Ben
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419378
Re: Merge mode (Was Classifying files as binary or text)
Posted by Branko Čibej <br...@xbc.nu>.
David Glasser wrote:
> I'm not a MIME expert, but isn't there a thing where you can add
> key/value pairs after a MIME type which customize its meaning? Maybe
> that could be used for this?
>
I understand those attributes are defined by the RFC, and you can't just
add your own. You can add comments, but eek; we don't need to parse MIME
commends, we have other properties.
-- Brane
Re: Merge mode (Was Classifying files as binary or text)
Posted by Branko Cibej <br...@xbc.nu>.
David Glasser wrote:
> I'm not a MIME expert, but isn't there a thing where you can add
> key/value pairs after a MIME type which customize its meaning? Maybe
> that could be used for this?
>
I understand those attributes are defined by the RFC, and you can't just
add your own. You can add comments, but eek; we don't need to parse MIME
commends, we have other properties.
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419351
Re: Merge mode (Was Classifying files as binary or text)
Posted by David Glasser <gl...@davidglasser.net>.
I'm not a MIME expert, but isn't there a thing where you can add
key/value pairs after a MIME type which customize its meaning? Maybe
that could be used for this?
--dave
On Tue, Nov 17, 2009 at 7:05 PM, Julian Foad <ju...@btopenworld.com> wrote:
> I need to spend some time replying, late at night though it is.
>
> Let me try to explain why I think a "how to merge" property should not
> be the primary indicator of how subversion should merge each file.
>
>
> Principle
> =========
>
> I have read that, in the realm of data handling, there is a principle
> that it is a bad idea to tag data with annotations that say what kind of
> actions can or should be performed on it. That kind of coupling is
> unscalable. Instead, it is better to tag data with an indication of what
> meaning and/or what syntax the data has, and then let tools decide what
> to do, based on that information.
>
> We already have one data-type indicator: svn:mime-type. Now, MIME type
> is far from a complete data type specifier. It is insufficient for our
> needs, in theory. However, in practice, it is nearly sufficient. (See
> Problem 2 below for an exception.)
>
> We also have another data-type indicator: the file name. A file name is
> also an incomplete source of metadata, and some file names ("README" or
> "CHANGES") give no indication at all of the format, but it is useful in
> many cases ("*.py", "*.c").
>
>
> Problem 1 (limited recognition of MIME types)
> =========
>
> Subversion mis-categorizes a lot of MIME types as "binary" (and
> therefore will not merge or diff or blame them) which really are
> line-based text formats.
>
> The list of such MIME types is continually evolving so it is not
> possible for Subversion to have a built-in complete list. However, it is
> easy for new releases of Subversion to have an updated list.
>
> It is not much harder for Subversion to have a configuarable list of
> which MIME types (or MIME type patterns) should be considered mergeable.
> (The configuration could be extensible: it could say line-wise-mergeable
> or not mergeable or XML-mergeable or ...)
>
>
> Problem 2 (mergeable and non-mergeable XML files)
> =========
>
> A user has some XML files on which a line-based merge is useful, and
> some XML files on which that is not useful, and wishes to label both
> kinds with svn:mime-type=text/xml. Let us suppose one format has each
> XML tag on a separate line, and the other has them all run together and
> line breaks inserted at arbitrary places in the sequence. It may be
> possible to find a different MIME type for one of the file types, but
> that may well not be possible, e.g. if they are both proprietary or
> arbitrary XML formats.
>
> This problem is not limited to XML files. Consider two "plain text"
> files, MIME type text/plain, with different kinds of text in them. One
> has line-based content, such as a shopping list, and changes usually
> leave many lines unchanged. The other contains the text of a newspaper
> article with line breaks at roughly every 70 characters in a stream of
> words, so two similar versions of it may have very few whole lines in
> common.
>
> I believe this is a real but relatively uncommon requirement. It is a
> genuine example of the MIME being insufficient to determine (line-wise)
> mergeability. There are many file formats that can be regarded as being
> (line-wise) mergeable or non-mergeable depending on some aspect of their
> content that cannot be reflected in the MIME type. It is uncommon in the
> sense that most Subversion users' needs can be satisfied by
> distinguishing mergeability based on the MIME type, or better the MIME
> type and file name taken together, of their files.
>
> To solve this problem when it exists, there does indeed need to be
> further metadata about the content type of the file. (Alternatively it
> could be metadata that says how to merge the file, but see "Principle"
> above.)
>
>
> Solution 0 (merge-mode)
> ==========
>
> So we could add a property to each file which says whether the file is
> to be considered line-wise mergeable by Subversion, and say that this
> property will be the primary source of this information. What are the
> pros and cons of this?
>
> Pro: The user can force a line-wise merge on one file and no merge
> attempt on another file even when MIME type and file name are
> insufficient distinguishers.
>
> Pro: The user can forget about providing MIME type at all, and just set
> this property to one of the pre-defined two types of merging (line-based
> or none), if that is all the user cares about.
>
> Con: This property associates the file with one simple kind of merging;
> but the best merge tool available on the client may not be that simple
> kind. If we want to use a better merge tool, say an XML-aware merge
> tool, this property actually gets in the way: it tells us to use a
> simple line-based merge on this file. It would have been better if the
> property had said, "this file contains line-based XML, so you might want
> to use an XML-aware merge rather than a simple line-based merge if you
> can". In other words, we really want to tell the client what the content
> type is, and let the client choose the best merge tool for that content
> type.
>
> Con: This property conveys redundant information. In almost all cases,
> the MIME type and/or file name are sufficient information. It is wrong
> to pretend that MIME type and file name are not good sources of
> how-to-merge information, and to leave their currently weak and
> deficient interpretation as just a deprecated backward-compatibility
> fallback.
>
> Con: Not extensible to diff, blame, etc. An indication that the file is
> line-wise-mergeable is not really a good indication of whether the file
> can be line-wise diffed or blamed.
>
>
> Proposal
> ========
>
> This is the full, long-term proposal. We can choose a subset of this to
> do initially.
>
> (1) Make svn merge/diff/blame take into account the file name as well as
> the svn:mime-type in deciding whether to operate in a "line-wise" mode
> or not operate at all.
>
> (2) Update the built-in MIME type and filename patterns.
>
> * Update the built-in selection based on svn:mime-type to recognise a
> list of MIME types that is reasonably up-to-date right now (even though
> it will be out of date by the time the released software is in use).
>
> * Update the built-in selection based on file names to recognize a
> reasonable list of file name patterns.
>
> (3) Provide a client-side config for extending and overriding the rules
> that map MIME type and file name to a merge/diff/blame mode. This mode
> should be specifiable in the config, not just "line-wise" or "none" but
> any other named mode. Provide config options for specifying the merge
> tool, diff tool and blame tool per mode. Tools should be specifiable as
> none, built-in or external.
>
> (4) Add an optional property for selecting a particular merge mode (and
> diff mode and blame mode) for the cases where (1) and (2) are
> insufficient or inconvenient.
>
>
> Regards,
> - Julian
>
>
> Mike Samuel wrote:
>> Proposal:
>> ========
>> (1) Add documentation on the svn:merge-mode property that lists the
>> allowed values as ("simple" and "none")
>> (2) Add example autoprops rules to the documentation that sets
>> svn:merge-mode to "simple" for the following file types
>> application/ecmascript
>> application/json
>> application/xml
>> image/svg+xml
>> (3) Change the text quoted from the SVN manual under Background to
>> read as below.
>> (4) Update the implementation to agree.
>>
>> Subversion treats the following files as [[mergable]]:
>>
>> * Files with no svn:mime-type [[and no svn:merge-mode]]
>> * Files with a svn:mime-type starting "text/"
>> * Files with a svn:mime-type equal to "image/x-xbitmap"
>> * Files with a svn:mime-type equal to "image/x-xpixmap"
>> * [[Files with a svn:merge-mode that is equal to "simple"]]
>>
>> All other files are treated as [[unmergeable]], meaning that
>> Subversion will:
>>
>> * Not attempt to automatically merge received changes with
>> local changes during svn update or svn merge
>> * Not show the differences as part of svn diff
>> * Not show line-by-line attribution for svn blame
>>
>> In all other respects, Subversion treats [[mergable]] files the
>> same as [[unmergeable]] files, e.g. if you set
>> the svn:keywords or svn:eol-style properties, Subversion will
>> perform keyword substitution
>> or newline conversion on [[unmergeable]] files.
>>
>>
>> Goal:
>> ====
>> To update the scheme by which svn {update,diff,merge,blame} to allow
>> merging of files
>> with svn:mime-type outside the hard-coded list currently used.
>>
>> This determination should be independent of the platform svn
>> is running on, so independent of the set of supported character sets.
>>
>> This scheme should not complicate future extensions to the merge
>> system which might wish to use a different merge policy, e.g. for XML
>> than for source code files.
>>
>> This scheme should work with autoprops, and other mechanisms repository
>> administrators use to manage files. Specifically, some kinds of XML can
>> be meaningfully meged, and others cannot.
>>
>> This scheme should work within existing limitations, such as the inability
>> to merge UTF-16 and UTF-32.
>>
>>
>> Background:
>> ==========
>> The current behavior is described at
>> http://subversion.tigris.org/faq.html#binary-files
>>
>> Subversion treats the following files as text:
>>
>> * Files with no svn:mime-type
>> * Files with a svn:mime-type starting "text/"
>> * Files with a svn:mime-type equal to "image/x-xbitmap"
>> * Files with a svn:mime-type equal to "image/x-xpixmap"
>>
>> All other files are treated as binary, meaning that Subversion will:
>>
>> * Not attempt to automatically merge received changes with
>> local changes during svn update or svn merge
>> * Not show the differences as part of svn diff
>> * Not show line-by-line attribution for svn blame
>>
>> In all other respects, Subversion treats binary files the same as
>> text files, e.g. if you set
>> the svn:keywords or svn:eol-style properties, Subversion will
>> perform keyword substitution
>> or newline conversion on binary files.
>>
>> Common source code mime-types are misclassified, and that problem is
>> likely to grow because of current IANA policy.
>> Mime-types are handed out by the IANA, which only assigns text/*
>> mime-types for file-types that are meant to be human readable. Source
>> code is explicitly not considered human readable. This is why many
>> source code and data mime-types are in the application/* group or
>> other non text/* groups: application/json, application/ecmascript,
>> application/xml, image/svg+xml.
>> RFC 4288 ( ftp://ftp.rfc-editor.org/in-notes/rfc4288.txt ) says this
>> Expected uses for the "application" media type
>> include but are not limited to file transfer, spreadsheets,
>> presentations, scheduling data, and languages for "active"
>> (computational) material.
>>
>> ------------------------------------------------------
>> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419155
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419275
>
--
glasser@davidglasser.net | langtonlabs.org | flickr.com/photos/glasser/
Re: Merge mode (Was Classifying files as binary or text)
Posted by Julian Foad <ju...@btopenworld.com>.
Mike Samuel wrote:
> What are the advantages of taking into account file path and mime-type
> over having autoprops look at mime-type and squirrel information away
> in the mime-type?
Two things:
1. Not coupling the data to knowledge of what tools will available to
the user. So it will allow me to install a special merge tool that
handles XML, for example, or MS-Word documents, for example, and hook
that into Subversion with a svn:mime-type and/or file names, even though
other users of the project may not have those tools available and would
therefore still need to be using the basic text-or-nothing merge modes.
2. Recognizing a stored setting is only half the problem. How to get
that setting set to sensible values consistently on all the files is the
other half of the problem. The proposal lacked a solution to that,
because auto-props is severly under-powered. (Auto-props cannot "look at
mime-type", nor can it examine file content; it can only match filename
patterns.)
> If file name is taken into account, how does diffing with ancestry
> work when a file has been added as a result of an svn mv or svn cp?
You mean, how would we handle the case when diffing a file that has two
different names that might map to two different merge-modes? The answer
would be some simple rule such as "use the later one".
(It is the same issue as how to decide what to do if it had two
different svn:mime-type values, or two different svn:merge-mode
properties. We already deal with similar issues with most of the other
svn:* properties.)
- Julian
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419453
Re: Merge mode (Was Classifying files as binary or text)
Posted by Mike Samuel <mi...@gmail.com>.
What are the advantages of taking into account file path and mime-type
over having autoprops look at mime-type and squirrel information away
in the mime-type?
If file name is taken into account, how does diffing with ancestry
work when a file has been added as a result of an svn mv or svn cp?
2009/11/17 Julian Foad <ju...@btopenworld.com>:
> I need to spend some time replying, late at night though it is.
>
> Let me try to explain why I think a "how to merge" property should not
> be the primary indicator of how subversion should merge each file.
>
>
> Principle
> =========
>
> I have read that, in the realm of data handling, there is a principle
> that it is a bad idea to tag data with annotations that say what kind of
> actions can or should be performed on it. That kind of coupling is
> unscalable. Instead, it is better to tag data with an indication of what
> meaning and/or what syntax the data has, and then let tools decide what
> to do, based on that information.
>
> We already have one data-type indicator: svn:mime-type. Now, MIME type
> is far from a complete data type specifier. It is insufficient for our
> needs, in theory. However, in practice, it is nearly sufficient. (See
> Problem 2 below for an exception.)
>
> We also have another data-type indicator: the file name. A file name is
> also an incomplete source of metadata, and some file names ("README" or
> "CHANGES") give no indication at all of the format, but it is useful in
> many cases ("*.py", "*.c").
>
>
> Problem 1 (limited recognition of MIME types)
> =========
>
> Subversion mis-categorizes a lot of MIME types as "binary" (and
> therefore will not merge or diff or blame them) which really are
> line-based text formats.
>
> The list of such MIME types is continually evolving so it is not
> possible for Subversion to have a built-in complete list. However, it is
> easy for new releases of Subversion to have an updated list.
>
> It is not much harder for Subversion to have a configuarable list of
> which MIME types (or MIME type patterns) should be considered mergeable.
> (The configuration could be extensible: it could say line-wise-mergeable
> or not mergeable or XML-mergeable or ...)
>
>
> Problem 2 (mergeable and non-mergeable XML files)
> =========
>
> A user has some XML files on which a line-based merge is useful, and
> some XML files on which that is not useful, and wishes to label both
> kinds with svn:mime-type=text/xml. Let us suppose one format has each
> XML tag on a separate line, and the other has them all run together and
> line breaks inserted at arbitrary places in the sequence. It may be
> possible to find a different MIME type for one of the file types, but
> that may well not be possible, e.g. if they are both proprietary or
> arbitrary XML formats.
>
> This problem is not limited to XML files. Consider two "plain text"
> files, MIME type text/plain, with different kinds of text in them. One
> has line-based content, such as a shopping list, and changes usually
> leave many lines unchanged. The other contains the text of a newspaper
> article with line breaks at roughly every 70 characters in a stream of
> words, so two similar versions of it may have very few whole lines in
> common.
>
> I believe this is a real but relatively uncommon requirement. It is a
> genuine example of the MIME being insufficient to determine (line-wise)
> mergeability. There are many file formats that can be regarded as being
> (line-wise) mergeable or non-mergeable depending on some aspect of their
> content that cannot be reflected in the MIME type. It is uncommon in the
> sense that most Subversion users' needs can be satisfied by
> distinguishing mergeability based on the MIME type, or better the MIME
> type and file name taken together, of their files.
>
> To solve this problem when it exists, there does indeed need to be
> further metadata about the content type of the file. (Alternatively it
> could be metadata that says how to merge the file, but see "Principle"
> above.)
>
>
> Solution 0 (merge-mode)
> ==========
>
> So we could add a property to each file which says whether the file is
> to be considered line-wise mergeable by Subversion, and say that this
> property will be the primary source of this information. What are the
> pros and cons of this?
This seems to be similar to the proposal at the beginning of this thread.
> Pro: The user can force a line-wise merge on one file and no merge
> attempt on another file even when MIME type and file name are
> insufficient distinguishers.
> Pro: The user can forget about providing MIME type at all, and just set
> this property to one of the pre-defined two types of merging (line-based
> or none), if that is all the user cares about.
> Con: This property associates the file with one simple kind of merging;
> but the best merge tool available on the client may not be that simple
> kind. If we want to use a better merge tool, say an XML-aware merge
> tool, this property actually gets in the way: it tells us to use a
> simple line-based merge on this file. It would have been better if the
> property had said, "this file contains line-based XML, so you might want
> to use an XML-aware merge rather than a simple line-based merge if you
> can". In other words, we really want to tell the client what the content
> type is, and let the client choose the best merge tool for that content
> type.
The goal section of the proposal at the beginning of the thread
includes compatibility with future merge extensions.
I don't understand how it conflicts with that. Per Branko's
suggestion, we are using two values initially, but can extend the
value set to specify different merging schemes.
> Con: This property conveys redundant information. In almost all cases,
> the MIME type and/or file name are sufficient information. It is wrong
> to pretend that MIME type and file name are not good sources of
> how-to-merge information, and to leave their currently weak and
> deficient interpretation as just a deprecated backward-compatibility
> fallback.
"In almost all cases, ... are sufficient" sounds a lot like
"insufficient" to me.
> Con: Not extensible to diff, blame, etc. An indication that the file is
> line-wise-mergeable is not really a good indication of whether the file
> can be line-wise diffed or blamed.
Please provide an example of where line-wise mergable -/-> line-wise
diffable & line-wise blameable.
>
> Proposal
> ========
>
> This is the full, long-term proposal. We can choose a subset of this to
> do initially.
>
> (1) Make svn merge/diff/blame take into account the file name as well as
> the svn:mime-type in deciding whether to operate in a "line-wise" mode
> or not operate at all.
>
> (2) Update the built-in MIME type and filename patterns.
>
> * Update the built-in selection based on svn:mime-type to recognise a
> list of MIME types that is reasonably up-to-date right now (even though
> it will be out of date by the time the released software is in use).
That sounds like a nice idea, but can be proposed and debated
separately from whether a new property, or filename information should
be taken into account. Can you start a separate thread on that?
> * Update the built-in selection based on file names to recognize a
> reasonable list of file name patterns.
>
> (3) Provide a client-side config for extending and overriding the rules
> that map MIME type and file name to a merge/diff/blame mode. This mode
> should be specifiable in the config, not just "line-wise" or "none" but
> any other named mode. Provide config options for specifying the merge
> tool, diff tool and blame tool per mode. Tools should be specifiable as
> none, built-in or external.
>
> (4) Add an optional property for selecting a particular merge mode (and
> diff mode and blame mode) for the cases where (1) and (2) are
> insufficient or inconvenient.
If this proposal requires an extra property above and beyond filename,
can autoprops fill the gaps?
> Regards,
> - Julian
>
>
> Mike Samuel wrote:
>> Proposal:
>> ========
>> (1) Add documentation on the svn:merge-mode property that lists the
>> allowed values as ("simple" and "none")
>> (2) Add example autoprops rules to the documentation that sets
>> svn:merge-mode to "simple" for the following file types
>> application/ecmascript
>> application/json
>> application/xml
>> image/svg+xml
>> (3) Change the text quoted from the SVN manual under Background to
>> read as below.
>> (4) Update the implementation to agree.
>>
>> Subversion treats the following files as [[mergable]]:
>>
>> * Files with no svn:mime-type [[and no svn:merge-mode]]
>> * Files with a svn:mime-type starting "text/"
>> * Files with a svn:mime-type equal to "image/x-xbitmap"
>> * Files with a svn:mime-type equal to "image/x-xpixmap"
>> * [[Files with a svn:merge-mode that is equal to "simple"]]
>>
>> All other files are treated as [[unmergeable]], meaning that
>> Subversion will:
>>
>> * Not attempt to automatically merge received changes with
>> local changes during svn update or svn merge
>> * Not show the differences as part of svn diff
>> * Not show line-by-line attribution for svn blame
>>
>> In all other respects, Subversion treats [[mergable]] files the
>> same as [[unmergeable]] files, e.g. if you set
>> the svn:keywords or svn:eol-style properties, Subversion will
>> perform keyword substitution
>> or newline conversion on [[unmergeable]] files.
>>
>>
>> Goal:
>> ====
>> To update the scheme by which svn {update,diff,merge,blame} to allow
>> merging of files
>> with svn:mime-type outside the hard-coded list currently used.
>>
>> This determination should be independent of the platform svn
>> is running on, so independent of the set of supported character sets.
>>
>> This scheme should not complicate future extensions to the merge
>> system which might wish to use a different merge policy, e.g. for XML
>> than for source code files.
>>
>> This scheme should work with autoprops, and other mechanisms repository
>> administrators use to manage files. Specifically, some kinds of XML can
>> be meaningfully meged, and others cannot.
>>
>> This scheme should work within existing limitations, such as the inability
>> to merge UTF-16 and UTF-32.
>>
>>
>> Background:
>> ==========
>> The current behavior is described at
>> http://subversion.tigris.org/faq.html#binary-files
>>
>> Subversion treats the following files as text:
>>
>> * Files with no svn:mime-type
>> * Files with a svn:mime-type starting "text/"
>> * Files with a svn:mime-type equal to "image/x-xbitmap"
>> * Files with a svn:mime-type equal to "image/x-xpixmap"
>>
>> All other files are treated as binary, meaning that Subversion will:
>>
>> * Not attempt to automatically merge received changes with
>> local changes during svn update or svn merge
>> * Not show the differences as part of svn diff
>> * Not show line-by-line attribution for svn blame
>>
>> In all other respects, Subversion treats binary files the same as
>> text files, e.g. if you set
>> the svn:keywords or svn:eol-style properties, Subversion will
>> perform keyword substitution
>> or newline conversion on binary files.
>>
>> Common source code mime-types are misclassified, and that problem is
>> likely to grow because of current IANA policy.
>> Mime-types are handed out by the IANA, which only assigns text/*
>> mime-types for file-types that are meant to be human readable. Source
>> code is explicitly not considered human readable. This is why many
>> source code and data mime-types are in the application/* group or
>> other non text/* groups: application/json, application/ecmascript,
>> application/xml, image/svg+xml.
>> RFC 4288 ( ftp://ftp.rfc-editor.org/in-notes/rfc4288.txt ) says this
>> Expected uses for the "application" media type
>> include but are not limited to file transfer, spreadsheets,
>> presentations, scheduling data, and languages for "active"
>> (computational) material.
>>
>> ------------------------------------------------------
>> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419155
>
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419302
Re: Merge mode (Was Classifying files as binary or text)
Posted by David Glasser <gl...@davidglasser.net>.
I'm not a MIME expert, but isn't there a thing where you can add
key/value pairs after a MIME type which customize its meaning? Maybe
that could be used for this?
--dave
On Tue, Nov 17, 2009 at 7:05 PM, Julian Foad <ju...@btopenworld.com> wrote:
> I need to spend some time replying, late at night though it is.
>
> Let me try to explain why I think a "how to merge" property should not
> be the primary indicator of how subversion should merge each file.
>
>
> Principle
> =========
>
> I have read that, in the realm of data handling, there is a principle
> that it is a bad idea to tag data with annotations that say what kind of
> actions can or should be performed on it. That kind of coupling is
> unscalable. Instead, it is better to tag data with an indication of what
> meaning and/or what syntax the data has, and then let tools decide what
> to do, based on that information.
>
> We already have one data-type indicator: svn:mime-type. Now, MIME type
> is far from a complete data type specifier. It is insufficient for our
> needs, in theory. However, in practice, it is nearly sufficient. (See
> Problem 2 below for an exception.)
>
> We also have another data-type indicator: the file name. A file name is
> also an incomplete source of metadata, and some file names ("README" or
> "CHANGES") give no indication at all of the format, but it is useful in
> many cases ("*.py", "*.c").
>
>
> Problem 1 (limited recognition of MIME types)
> =========
>
> Subversion mis-categorizes a lot of MIME types as "binary" (and
> therefore will not merge or diff or blame them) which really are
> line-based text formats.
>
> The list of such MIME types is continually evolving so it is not
> possible for Subversion to have a built-in complete list. However, it is
> easy for new releases of Subversion to have an updated list.
>
> It is not much harder for Subversion to have a configuarable list of
> which MIME types (or MIME type patterns) should be considered mergeable.
> (The configuration could be extensible: it could say line-wise-mergeable
> or not mergeable or XML-mergeable or ...)
>
>
> Problem 2 (mergeable and non-mergeable XML files)
> =========
>
> A user has some XML files on which a line-based merge is useful, and
> some XML files on which that is not useful, and wishes to label both
> kinds with svn:mime-type=text/xml. Let us suppose one format has each
> XML tag on a separate line, and the other has them all run together and
> line breaks inserted at arbitrary places in the sequence. It may be
> possible to find a different MIME type for one of the file types, but
> that may well not be possible, e.g. if they are both proprietary or
> arbitrary XML formats.
>
> This problem is not limited to XML files. Consider two "plain text"
> files, MIME type text/plain, with different kinds of text in them. One
> has line-based content, such as a shopping list, and changes usually
> leave many lines unchanged. The other contains the text of a newspaper
> article with line breaks at roughly every 70 characters in a stream of
> words, so two similar versions of it may have very few whole lines in
> common.
>
> I believe this is a real but relatively uncommon requirement. It is a
> genuine example of the MIME being insufficient to determine (line-wise)
> mergeability. There are many file formats that can be regarded as being
> (line-wise) mergeable or non-mergeable depending on some aspect of their
> content that cannot be reflected in the MIME type. It is uncommon in the
> sense that most Subversion users' needs can be satisfied by
> distinguishing mergeability based on the MIME type, or better the MIME
> type and file name taken together, of their files.
>
> To solve this problem when it exists, there does indeed need to be
> further metadata about the content type of the file. (Alternatively it
> could be metadata that says how to merge the file, but see "Principle"
> above.)
>
>
> Solution 0 (merge-mode)
> ==========
>
> So we could add a property to each file which says whether the file is
> to be considered line-wise mergeable by Subversion, and say that this
> property will be the primary source of this information. What are the
> pros and cons of this?
>
> Pro: The user can force a line-wise merge on one file and no merge
> attempt on another file even when MIME type and file name are
> insufficient distinguishers.
>
> Pro: The user can forget about providing MIME type at all, and just set
> this property to one of the pre-defined two types of merging (line-based
> or none), if that is all the user cares about.
>
> Con: This property associates the file with one simple kind of merging;
> but the best merge tool available on the client may not be that simple
> kind. If we want to use a better merge tool, say an XML-aware merge
> tool, this property actually gets in the way: it tells us to use a
> simple line-based merge on this file. It would have been better if the
> property had said, "this file contains line-based XML, so you might want
> to use an XML-aware merge rather than a simple line-based merge if you
> can". In other words, we really want to tell the client what the content
> type is, and let the client choose the best merge tool for that content
> type.
>
> Con: This property conveys redundant information. In almost all cases,
> the MIME type and/or file name are sufficient information. It is wrong
> to pretend that MIME type and file name are not good sources of
> how-to-merge information, and to leave their currently weak and
> deficient interpretation as just a deprecated backward-compatibility
> fallback.
>
> Con: Not extensible to diff, blame, etc. An indication that the file is
> line-wise-mergeable is not really a good indication of whether the file
> can be line-wise diffed or blamed.
>
>
> Proposal
> ========
>
> This is the full, long-term proposal. We can choose a subset of this to
> do initially.
>
> (1) Make svn merge/diff/blame take into account the file name as well as
> the svn:mime-type in deciding whether to operate in a "line-wise" mode
> or not operate at all.
>
> (2) Update the built-in MIME type and filename patterns.
>
> * Update the built-in selection based on svn:mime-type to recognise a
> list of MIME types that is reasonably up-to-date right now (even though
> it will be out of date by the time the released software is in use).
>
> * Update the built-in selection based on file names to recognize a
> reasonable list of file name patterns.
>
> (3) Provide a client-side config for extending and overriding the rules
> that map MIME type and file name to a merge/diff/blame mode. This mode
> should be specifiable in the config, not just "line-wise" or "none" but
> any other named mode. Provide config options for specifying the merge
> tool, diff tool and blame tool per mode. Tools should be specifiable as
> none, built-in or external.
>
> (4) Add an optional property for selecting a particular merge mode (and
> diff mode and blame mode) for the cases where (1) and (2) are
> insufficient or inconvenient.
>
>
> Regards,
> - Julian
>
>
> Mike Samuel wrote:
>> Proposal:
>> ========
>> (1) Add documentation on the svn:merge-mode property that lists the
>> allowed values as ("simple" and "none")
>> (2) Add example autoprops rules to the documentation that sets
>> svn:merge-mode to "simple" for the following file types
>> application/ecmascript
>> application/json
>> application/xml
>> image/svg+xml
>> (3) Change the text quoted from the SVN manual under Background to
>> read as below.
>> (4) Update the implementation to agree.
>>
>> Subversion treats the following files as [[mergable]]:
>>
>> * Files with no svn:mime-type [[and no svn:merge-mode]]
>> * Files with a svn:mime-type starting "text/"
>> * Files with a svn:mime-type equal to "image/x-xbitmap"
>> * Files with a svn:mime-type equal to "image/x-xpixmap"
>> * [[Files with a svn:merge-mode that is equal to "simple"]]
>>
>> All other files are treated as [[unmergeable]], meaning that
>> Subversion will:
>>
>> * Not attempt to automatically merge received changes with
>> local changes during svn update or svn merge
>> * Not show the differences as part of svn diff
>> * Not show line-by-line attribution for svn blame
>>
>> In all other respects, Subversion treats [[mergable]] files the
>> same as [[unmergeable]] files, e.g. if you set
>> the svn:keywords or svn:eol-style properties, Subversion will
>> perform keyword substitution
>> or newline conversion on [[unmergeable]] files.
>>
>>
>> Goal:
>> ====
>> To update the scheme by which svn {update,diff,merge,blame} to allow
>> merging of files
>> with svn:mime-type outside the hard-coded list currently used.
>>
>> This determination should be independent of the platform svn
>> is running on, so independent of the set of supported character sets.
>>
>> This scheme should not complicate future extensions to the merge
>> system which might wish to use a different merge policy, e.g. for XML
>> than for source code files.
>>
>> This scheme should work with autoprops, and other mechanisms repository
>> administrators use to manage files. Specifically, some kinds of XML can
>> be meaningfully meged, and others cannot.
>>
>> This scheme should work within existing limitations, such as the inability
>> to merge UTF-16 and UTF-32.
>>
>>
>> Background:
>> ==========
>> The current behavior is described at
>> http://subversion.tigris.org/faq.html#binary-files
>>
>> Subversion treats the following files as text:
>>
>> * Files with no svn:mime-type
>> * Files with a svn:mime-type starting "text/"
>> * Files with a svn:mime-type equal to "image/x-xbitmap"
>> * Files with a svn:mime-type equal to "image/x-xpixmap"
>>
>> All other files are treated as binary, meaning that Subversion will:
>>
>> * Not attempt to automatically merge received changes with
>> local changes during svn update or svn merge
>> * Not show the differences as part of svn diff
>> * Not show line-by-line attribution for svn blame
>>
>> In all other respects, Subversion treats binary files the same as
>> text files, e.g. if you set
>> the svn:keywords or svn:eol-style properties, Subversion will
>> perform keyword substitution
>> or newline conversion on binary files.
>>
>> Common source code mime-types are misclassified, and that problem is
>> likely to grow because of current IANA policy.
>> Mime-types are handed out by the IANA, which only assigns text/*
>> mime-types for file-types that are meant to be human readable. Source
>> code is explicitly not considered human readable. This is why many
>> source code and data mime-types are in the application/* group or
>> other non text/* groups: application/json, application/ecmascript,
>> application/xml, image/svg+xml.
>> RFC 4288 ( ftp://ftp.rfc-editor.org/in-notes/rfc4288.txt ) says this
>> Expected uses for the "application" media type
>> include but are not limited to file transfer, spreadsheets,
>> presentations, scheduling data, and languages for "active"
>> (computational) material.
>>
>> ------------------------------------------------------
>> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419155
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419275
>
--
glasser@davidglasser.net | langtonlabs.org | flickr.com/photos/glasser/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419346
Re: Merge mode (Was Classifying files as binary or text)
Posted by Mike Samuel <mi...@gmail.com>.
Julian,
Since there seems to be quite a bit of confusion over what you have
proposed, can you repost it. Perhaps you could show the diffs from
the current FAQ as the first post of this thread does for the
svn:merge-mode proposal.
mike
2009/11/18 Julian Foad <ju...@btopenworld.com>:
> On Wed, 2009-11-18 at 10:50 -0500, Mark Phippard wrote:
>> On Wed, Nov 18, 2009 at 10:27 AM, Julian Foad
>> <ju...@btopenworld.com> wrote:
>> > Mark Phippard wrote:
>> >> The reply message I am top-posting to sums up my feelings exactly. I
>> >> think Julian's proposal would be a step-backwards and really not even
>> >> help with the problem at hand.
>> >
>> > Mark, I'm sorry you got that impression, and you don't seem to be the
>> > only one. I must be poor at expressing myself, combined with that I am
>> > only struggling towards the best solution and have not got all the
>> > answers (yet).
>>
>> I just do not think file extensions really helps much. You gave the
>> Perl example in this email where it would, but in your proposal you
>> talked about application/xml and that is an example where it does not
>> really help.
>
> You seem to have got the idea that I was proposing just looking at
> filenames. I was not. I said there should be a mechanism for specifying
> the merge-mode (and also, separately, the diff-mode and the blame-mode)
> as a function of the tuple
>
> (svn:mime-type, filename, content, svn:merge-mode)
>
> Now I'm thinking let's generalize that further and have it a function of
>
> (properties, filename, content)
>
> and supply a good default configuration built-in, and supply an example
> plug-in configuration or two.
>
>> I actually think the original proposal of adding charset
>> is probably a better approach for what you seem to want to accomplish.
>>
>> > I understand only a small part of the reasons why you feel that. If I
>> > may be so cheeky as to paraphrase, I felt it sounded a bit of a "just
>> > give the user another knob to twiddle and he'll shut up" sort of
>> > proposal. Not totally; that's unfair, as it has some merits; but there
>> > was just too much of that feel in it for me to stay quiet.
>>
>> Well, when it comes down to it, I just think the svn:merge-mode
>> proposal does a better job of providing a solution than yours will.
>>
>> >> Despite our penchant for self-flaggelation, it seems like we are
>> >> forgetting that our current system actually works pretty well for most
>> >> cases.
>> >
>> > Exactly... but the proposal had a tase of abandoning it, with a mere
>> > concession to keeping it for backward compatibility rather than
>> > recognizing its value and extending the principle (the principle of
>> > selecting a processing mode based on content type).
>>
>> I don't know, if anything the proposal seems even better for this.
>> That is why the proposal calls for a property named svn:merge-mode.
>> So that it could be possible to inject different modes of processing
>> in the future.
>>
>> > It's the "small subset of file types" that's contentious. That may have
>> > been the case for typical svn users a few years ago but now many
>> > mainstream textual files have types we consider "binary".
>> >
>> > If I use the trunk's new and very useful config option "mime-types-file
>> > = /etc/mime.types" and then "svn add" a .pl Perl script, it gets an
>> > svn:mime-type of "application/x-perl". That means that if I continue
>> > working like that, and we don't update the current svn:mime-type
>> > recognition, then I'll need to go setting the new prop all my Perl
>> > files. If I'm a Perl programmer, that basically does mean I need it on
>> > all my files.
>> >
>> > That is a big concern to me: I don't want everyone to get into the habit
>> > of setting these props all over their data.
>>
>> This probably is an example of our OCD getting in the way. Why did we
>> even add this mime.types feature to begin with? Who thought it would
>> be a brilliant idea to add a "proper" mime-type to every file in the
>> first place? Basically, we added a feature that was not needed that
>> cascades to create a problem.
>>
>> >> B Smith-Mannschott wrote, re why don't we update the built-in recognition:
>> >> > (1) It's an ongoing maintenance headache.
>> >
>> > Sure. So, are we abandoning it in favour of something better? We can't
>> > claim it's still the main method if we aren't prepared to update it.
>> >
>> > Until we replace it, if someone is prepared to update it, should we stop
>> > them just because it's an ongoing maintenance headache? Of course not.
>>
>> I think the ongoing maintenance burden would be trying to codify which
>> mime-types we know we can treat as text. I am not sure it is even
>> reliable. For example, we could not say we handle application/xml
>> because there are definite cases where we cannot.
>
> Just because we can't merge all XML "properly" doesn't mean we can't
> supply a sane default (yes or no) as a starting point for the users to
> customize.
>
>> This is one of the
>> main mime-type that has been a problem. So how does your solution
>> even begin to address this?
>
> By allowing the default to come from svn:mime-type (or a function of
> (svn:mime-type, filename, content)) and allowing the user to override it
> with a special svn:merge-mode property. And of course svn:diff-mode and
> svn:blame-mode.
>
>> In terms of a type like application/x-perl I assume it would be safe
>> to always treat that as text. Given that we have some precedence for
>> doing this already I would be +1 on extending that list to cover some
>> types where we know this is safe.
>
> Great.
>
>> But we still need to something like
>> svn:merge-mode to cover the other types where it is not a binary answer
>
> Yes, I agree. (Modulo my assertion that a yes/no answer is 99% useful
> for most file types: for most people it is not important to be able to
> tell Subversion to try to built-in-merge some XML files but not others,
> I assert.)
>
>> and that is all the proposal was ever about.
>
> No, I don't think it was only about the cases where you can't get a
> definite answer from the svn:mime-type, I think it was aimed at covering
> up the gaping holes in our recognized MIME type list.
>
> The cases where you can't get a definite answer are just an additional
> requirement that mean some people want such an additional mechanism
> anyway.
>
> - Julian
>
>
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419637
Re: Merge mode (Was Classifying files as binary or text)
Posted by Julian Foad <ju...@btopenworld.com>.
On Wed, 2009-11-18 at 10:50 -0500, Mark Phippard wrote:
> On Wed, Nov 18, 2009 at 10:27 AM, Julian Foad
> <ju...@btopenworld.com> wrote:
> > Mark Phippard wrote:
> >> The reply message I am top-posting to sums up my feelings exactly. I
> >> think Julian's proposal would be a step-backwards and really not even
> >> help with the problem at hand.
> >
> > Mark, I'm sorry you got that impression, and you don't seem to be the
> > only one. I must be poor at expressing myself, combined with that I am
> > only struggling towards the best solution and have not got all the
> > answers (yet).
>
> I just do not think file extensions really helps much. You gave the
> Perl example in this email where it would, but in your proposal you
> talked about application/xml and that is an example where it does not
> really help.
You seem to have got the idea that I was proposing just looking at
filenames. I was not. I said there should be a mechanism for specifying
the merge-mode (and also, separately, the diff-mode and the blame-mode)
as a function of the tuple
(svn:mime-type, filename, content, svn:merge-mode)
Now I'm thinking let's generalize that further and have it a function of
(properties, filename, content)
and supply a good default configuration built-in, and supply an example
plug-in configuration or two.
> I actually think the original proposal of adding charset
> is probably a better approach for what you seem to want to accomplish.
>
> > I understand only a small part of the reasons why you feel that. If I
> > may be so cheeky as to paraphrase, I felt it sounded a bit of a "just
> > give the user another knob to twiddle and he'll shut up" sort of
> > proposal. Not totally; that's unfair, as it has some merits; but there
> > was just too much of that feel in it for me to stay quiet.
>
> Well, when it comes down to it, I just think the svn:merge-mode
> proposal does a better job of providing a solution than yours will.
>
> >> Despite our penchant for self-flaggelation, it seems like we are
> >> forgetting that our current system actually works pretty well for most
> >> cases.
> >
> > Exactly... but the proposal had a tase of abandoning it, with a mere
> > concession to keeping it for backward compatibility rather than
> > recognizing its value and extending the principle (the principle of
> > selecting a processing mode based on content type).
>
> I don't know, if anything the proposal seems even better for this.
> That is why the proposal calls for a property named svn:merge-mode.
> So that it could be possible to inject different modes of processing
> in the future.
>
> > It's the "small subset of file types" that's contentious. That may have
> > been the case for typical svn users a few years ago but now many
> > mainstream textual files have types we consider "binary".
> >
> > If I use the trunk's new and very useful config option "mime-types-file
> > = /etc/mime.types" and then "svn add" a .pl Perl script, it gets an
> > svn:mime-type of "application/x-perl". That means that if I continue
> > working like that, and we don't update the current svn:mime-type
> > recognition, then I'll need to go setting the new prop all my Perl
> > files. If I'm a Perl programmer, that basically does mean I need it on
> > all my files.
> >
> > That is a big concern to me: I don't want everyone to get into the habit
> > of setting these props all over their data.
>
> This probably is an example of our OCD getting in the way. Why did we
> even add this mime.types feature to begin with? Who thought it would
> be a brilliant idea to add a "proper" mime-type to every file in the
> first place? Basically, we added a feature that was not needed that
> cascades to create a problem.
>
> >> B Smith-Mannschott wrote, re why don't we update the built-in recognition:
> >> > (1) It's an ongoing maintenance headache.
> >
> > Sure. So, are we abandoning it in favour of something better? We can't
> > claim it's still the main method if we aren't prepared to update it.
> >
> > Until we replace it, if someone is prepared to update it, should we stop
> > them just because it's an ongoing maintenance headache? Of course not.
>
> I think the ongoing maintenance burden would be trying to codify which
> mime-types we know we can treat as text. I am not sure it is even
> reliable. For example, we could not say we handle application/xml
> because there are definite cases where we cannot.
Just because we can't merge all XML "properly" doesn't mean we can't
supply a sane default (yes or no) as a starting point for the users to
customize.
> This is one of the
> main mime-type that has been a problem. So how does your solution
> even begin to address this?
By allowing the default to come from svn:mime-type (or a function of
(svn:mime-type, filename, content)) and allowing the user to override it
with a special svn:merge-mode property. And of course svn:diff-mode and
svn:blame-mode.
> In terms of a type like application/x-perl I assume it would be safe
> to always treat that as text. Given that we have some precedence for
> doing this already I would be +1 on extending that list to cover some
> types where we know this is safe.
Great.
> But we still need to something like
> svn:merge-mode to cover the other types where it is not a binary answer
Yes, I agree. (Modulo my assertion that a yes/no answer is 99% useful
for most file types: for most people it is not important to be able to
tell Subversion to try to built-in-merge some XML files but not others,
I assert.)
> and that is all the proposal was ever about.
No, I don't think it was only about the cases where you can't get a
definite answer from the svn:mime-type, I think it was aimed at covering
up the gaping holes in our recognized MIME type list.
The cases where you can't get a definite answer are just an additional
requirement that mean some people want such an additional mechanism
anyway.
- Julian
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419597
Re: Merge mode (Was Classifying files as binary or text)
Posted by Mike Samuel <mi...@gmail.com>.
2009/11/18 Branko Čibej <br...@xbc.nu>:
> Mark Phippard wrote:
>> This probably is an example of our OCD getting in the way. Why did we
>> even add this mime.types feature to begin with? Who thought it would
>> be a brilliant idea to add a "proper" mime-type to every file in the
>> first place? Basically, we added a feature that was not needed that
>> cascades to create a problem.
>>
>
> Couldn't agree more ... we badly misjudged that one. Using mime.types is
> OK if you're setting svn:mime-type, but not if you're also telling SVN
> how to diff/merge/whatever that file.
>
> (BTW ... we could have a different prop name instead of merge-mode to
> indicate that it can, in fact, cheerfully apply to diff and blame, too.)
I'm still collecting property names so we can compare all at once.
Merging/diffing/etc. are all about breaking a document into smaller
chunks based on semantic or structural cues.
Re: Merge mode (Was Classifying files as binary or text)
Posted by Branko Cibej <br...@xbc.nu>.
Mark Phippard wrote:
> This probably is an example of our OCD getting in the way. Why did we
> even add this mime.types feature to begin with? Who thought it would
> be a brilliant idea to add a "proper" mime-type to every file in the
> first place? Basically, we added a feature that was not needed that
> cascades to create a problem.
>
Couldn't agree more ... we badly misjudged that one. Using mime.types is
OK if you're setting svn:mime-type, but not if you're also telling SVN
how to diff/merge/whatever that file.
(BTW ... we could have a different prop name instead of merge-mode to
indicate that it can, in fact, cheerfully apply to diff and blame, too.)
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419674
Re: Merge mode (Was Classifying files as binary or text)
Posted by Mark Phippard <ma...@gmail.com>.
On Wed, Nov 18, 2009 at 10:27 AM, Julian Foad
<ju...@btopenworld.com> wrote:
> Mark Phippard wrote:
>> The reply message I am top-posting to sums up my feelings exactly. I
>> think Julian's proposal would be a step-backwards and really not even
>> help with the problem at hand.
>
> Mark, I'm sorry you got that impression, and you don't seem to be the
> only one. I must be poor at expressing myself, combined with that I am
> only struggling towards the best solution and have not got all the
> answers (yet).
I just do not think file extensions really helps much. You gave the
Perl example in this email where it would, but in your proposal you
talked about application/xml and that is an example where it does not
really help. I actually think the original proposal of adding charset
is probably a better approach for what you seem to want to accomplish.
> I understand only a small part of the reasons why you feel that. If I
> may be so cheeky as to paraphrase, I felt it sounded a bit of a "just
> give the user another knob to twiddle and he'll shut up" sort of
> proposal. Not totally; that's unfair, as it has some merits; but there
> was just too much of that feel in it for me to stay quiet.
Well, when it comes down to it, I just think the svn:merge-mode
proposal does a better job of providing a solution than yours will.
>> Despite our penchant for self-flaggelation, it seems like we are
>> forgetting that our current system actually works pretty well for most
>> cases.
>
> Exactly... but the proposal had a tase of abandoning it, with a mere
> concession to keeping it for backward compatibility rather than
> recognizing its value and extending the principle (the principle of
> selecting a processing mode based on content type).
I don't know, if anything the proposal seems even better for this.
That is why the proposal calls for a property named svn:merge-mode.
So that it could be possible to inject different modes of processing
in the future.
> It's the "small subset of file types" that's contentious. That may have
> been the case for typical svn users a few years ago but now many
> mainstream textual files have types we consider "binary".
>
> If I use the trunk's new and very useful config option "mime-types-file
> = /etc/mime.types" and then "svn add" a .pl Perl script, it gets an
> svn:mime-type of "application/x-perl". That means that if I continue
> working like that, and we don't update the current svn:mime-type
> recognition, then I'll need to go setting the new prop all my Perl
> files. If I'm a Perl programmer, that basically does mean I need it on
> all my files.
>
> That is a big concern to me: I don't want everyone to get into the habit
> of setting these props all over their data.
This probably is an example of our OCD getting in the way. Why did we
even add this mime.types feature to begin with? Who thought it would
be a brilliant idea to add a "proper" mime-type to every file in the
first place? Basically, we added a feature that was not needed that
cascades to create a problem.
>> B Smith-Mannschott wrote, re why don't we update the built-in recognition:
>> > (1) It's an ongoing maintenance headache.
>
> Sure. So, are we abandoning it in favour of something better? We can't
> claim it's still the main method if we aren't prepared to update it.
>
> Until we replace it, if someone is prepared to update it, should we stop
> them just because it's an ongoing maintenance headache? Of course not.
I think the ongoing maintenance burden would be trying to codify which
mime-types we know we can treat as text. I am not sure it is even
reliable. For example, we could not say we handle application/xml
because there are definite cases where we cannot. This is one of the
main mime-type that has been a problem. So how does your solution
even begin to address this?
In terms of a type like application/x-perl I assume it would be safe
to always treat that as text. Given that we have some precedence for
doing this already I would be +1 on extending that list to cover some
types where we know this is safe. But we still need to something like
svn:merge-mode to cover the other types where it is not a binary
answer and that is all the proposal was ever about.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419559
Re: Merge mode (Was Classifying files as binary or text)
Posted by Julian Foad <ju...@btopenworld.com>.
Mark Phippard wrote:
> The reply message I am top-posting to sums up my feelings exactly. I
> think Julian's proposal would be a step-backwards and really not even
> help with the problem at hand.
Mark, I'm sorry you got that impression, and you don't seem to be the
only one. I must be poor at expressing myself, combined with that I am
only struggling towards the best solution and have not got all the
answers (yet).
I understand only a small part of the reasons why you feel that. If I
may be so cheeky as to paraphrase, I felt it sounded a bit of a "just
give the user another knob to twiddle and he'll shut up" sort of
proposal. Not totally; that's unfair, as it has some merits; but there
was just too much of that feel in it for me to stay quiet.
> Despite our penchant for self-flaggelation, it seems like we are
> forgetting that our current system actually works pretty well for most
> cases.
Exactly... but the proposal had a tase of abandoning it, with a mere
concession to keeping it for backward compatibility rather than
recognizing its value and extending the principle (the principle of
selecting a processing mode based on content type).
> There are a limited number of MIME types where there are
> different ways they could be handled, and users already have a built
> in workaround available (remove the svn:mime-type property).
> Apparently users really like serving files directly from their SVN
> servers or their OCD just will not allow them to remove the property.
> In those limited cases, we are simply talking about adding a new
> property to Subversion to let you tell us to treat the file as text.
> There is no need to make this more complicated.
That's fine if the intent is to accompany this with updating the
mime-type recognition and augmenting it with other recognition methods
that we deem worthwhile, so that most users don't need the special
annotation.
> No one is suggesting that users go out and set the svn:merge-mode
> property on all their existing files. It is just a tool we want to
> give users to handle a small subset of file types.
It's the "small subset of file types" that's contentious. That may have
been the case for typical svn users a few years ago but now many
mainstream textual files have types we consider "binary".
If I use the trunk's new and very useful config option "mime-types-file
= /etc/mime.types" and then "svn add" a .pl Perl script, it gets an
svn:mime-type of "application/x-perl". That means that if I continue
working like that, and we don't update the current svn:mime-type
recognition, then I'll need to go setting the new prop all my Perl
files. If I'm a Perl programmer, that basically does mean I need it on
all my files.
That is a big concern to me: I don't want everyone to get into the habit
of setting these props all over their data.
> B Smith-Mannschott wrote, re why don't we update the built-in recognition:
> > (1) It's an ongoing maintenance headache.
Sure. So, are we abandoning it in favour of something better? We can't
claim it's still the main method if we aren't prepared to update it.
Until we replace it, if someone is prepared to update it, should we stop
them just because it's an ongoing maintenance headache? Of course not.
> > (2) The merge behavior of existing repositories will change,
> > surprising some, no doubt.
It will change, true. Surprising, yes, to those who don't read the
release notes. A problem? In practice, I really don't see a problem.
> > (3) *The mapping from MIME-type to 'mergeable' is ambiguous.*
> > Consider, again, application/xml. No amount of "updating the built-in
> > list of MIME types" is going to help here.
Yes, absolutely. For those FEW cases where the different XML file
formats don't have their own specialized MIME types and the user REALLY
needs or wants to treat them differently, that's where the "yes, have a
dedicated property as well" part of my pseudo-proposal comes in.
> >>> It is not much harder for Subversion to have a configuarable list of
> >>> which MIME types (or MIME type patterns) should be considered mergeable.
> >>> (The configuration could be extensible: it could say line-wise-mergeable
> >>> or not mergeable or XML-mergeable or ...)
> >>
> >> [...]
> >>
> >>> Solution 0 (merge-mode)
> >>> ==========
> >>>
> >>> So we could add a property to each file which says whether the file is
> >>> to be considered line-wise mergeable by Subversion, and say that this
> >>> property will be the primary source of this information. What are the
> >>> pros and cons of this?
> >>
> >> Big con: Depends on auto-props, a mechanism which is not really good
> >> enough for the task. (Strictly speaking it doesn't, because users can
> >> set the property manually, but in a practical sense it does because in a
> >> production environment it is impractical for users to consistently set
> >> the property manually.)
> >
> > I find this criticism misleading. It leaves the impression that the
> > OP's proposal depends exclusively on the merge-mode property to
> > determine merging behavior. This is not the case. Merge-mode is only
> > needed when the existing heuristics -- which remain unchanged by the
> > proposal -- prove insufficient to the task.
Yes, but the implication was that the existing heuristics would become a
second-class citizen, left to rot, and so in practice would become less
and less used.
> > To wit:
> >
> > On Tue, Nov 17, 2009 at 21:56, Mike Samuel <mi...@gmail.com> wrote:
> >>
> >> Subversion treats the following files as [[mergable]]:
> >>
> >> * Files with no svn:mime-type [[and no svn:merge-mode]]
> >> * Files with a svn:mime-type starting "text/"
> >> * Files with a svn:mime-type equal to "image/x-xbitmap"
> >> * Files with a svn:mime-type equal to "image/x-xpixmap"
> >> * [[Files with a svn:merge-mode that is equal to "simple"]]
> >>
> >
> > Furthermore, the need for coordinating autoprops within a team exists
> > without this proposal, independent of issues of merge behavior. I fail
> > to see how the OP's proposal will in any way exacerbate that.
> > I've got about 100 lines of autoprops in the subversion config file I
> > maintain for my team. This is how we try to assure that most of our
> > files get reasonable mime-type, needs-lock, eol-style, from the
> > get-go. Periodically I fix things up with a bash script or two. The
> > need for this won't be affected by the merge-mode proposal.
That's all very well for your team where you have got the management of
auto-props under control. Many people don't use them and so don't have
such a system ready to go.
- Julian
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419549
Re: Merge mode (Was Classifying files as binary or text)
Posted by Mark Phippard <ma...@gmail.com>.
The reply message I am top-posting to sums up my feelings exactly. I
think Julian's proposal would be a step-backwards and really not even
help with the problem at hand.
Despite our penchant for self-flaggelation, it seems like we are
forgetting that our current system actually works pretty well for most
cases. There are a limited number of MIME types where there are
different ways they could be handled, and users already have a built
in workaround available (remove the svn:mime-type property).
Apparently users really like serving files directly from their SVN
servers or their OCD just will not allow them to remove the property.
In those limited cases, we are simply talking about adding a new
property to Subversion to let you tell us to treat the file as text.
There is no need to make this more complicated.
No one is suggesting that users go out and set the svn:merge-mode
property on all their existing files. It is just a tool we want to
give users to handle a small subset of file types.
Mark
On Wed, Nov 18, 2009 at 6:31 AM, B Smith-Mannschott
<bs...@gmail.com> wrote:
> (1) It's an ongoing maintenance headache.
> (2) The merge behavior of existing repositories will change,
> surprising some, no doubt.
> (3) *The mapping from MIME-type to 'mergeable' is ambiguous.*
> Consider, again, application/xml. No amount of "updating the built-in
> list of MIME types" is going to help here.
>
>>> It is not much harder for Subversion to have a configuarable list of
>>> which MIME types (or MIME type patterns) should be considered mergeable.
>>> (The configuration could be extensible: it could say line-wise-mergeable
>>> or not mergeable or XML-mergeable or ...)
>>
>> [...]
>>
>>> Solution 0 (merge-mode)
>>> ==========
>>>
>>> So we could add a property to each file which says whether the file is
>>> to be considered line-wise mergeable by Subversion, and say that this
>>> property will be the primary source of this information. What are the
>>> pros and cons of this?
>>
>> Big con: Depends on auto-props, a mechanism which is not really good
>> enough for the task. (Strictly speaking it doesn't, because users can
>> set the property manually, but in a practical sense it does because in a
>> production environment it is impractical for users to consistently set
>> the property manually.)
>
> I find this criticism misleading. It leaves the impression that the
> OP's proposal depends exclusively on the merge-mode property to
> determine merging behavior. This is not the case. Merge-mode is only
> needed when the existing heuristics -- which remain unchanged by the
> proposal -- prove insufficient to the task.
>
> To wit:
>
> On Tue, Nov 17, 2009 at 21:56, Mike Samuel <mi...@gmail.com> wrote:
>>
>> Subversion treats the following files as [[mergable]]:
>>
>> * Files with no svn:mime-type [[and no svn:merge-mode]]
>> * Files with a svn:mime-type starting "text/"
>> * Files with a svn:mime-type equal to "image/x-xbitmap"
>> * Files with a svn:mime-type equal to "image/x-xpixmap"
>> * [[Files with a svn:merge-mode that is equal to "simple"]]
>>
>
> Furthermore, the need for coordinating autoprops within a team exists
> without this proposal, independent of issues of merge behavior. I fail
> to see how the OP's proposal will in any way exacerbate that.
>
> I've got about 100 lines of autoprops in the subversion config file I
> maintain for my team. This is how we try to assure that most of our
> files get reasonable mime-type, needs-lock, eol-style, from the
> get-go. Periodically I fix things up with a bash script or two. The
> need for this won't be affected by the merge-mode proposal.
>
> // Ben
>
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419498
Re: Merge mode (Was Classifying files as binary or text)
Posted by "B. Smith-Mannschott" <bs...@gmail.com>.
On Wed, Nov 18, 2009 at 11:55, Julian Foad <ju...@btopenworld.com> wrote:
> Julian Foad wrote:
>> I need to spend some time replying, late at night though it is.
>>
>> Let me try to explain why I think a "how to merge" property should not
>> be the primary indicator of how subversion should merge each file.
>
> Some comments on my own posting...
>
>> Principle
>> =========
>>
>> I have read that, in the realm of data handling, there is a principle
>> that it is a bad idea to tag data with annotations that say what kind of
>> actions can or should be performed on it. That kind of coupling is
>> unscalable. Instead, it is better to tag data with an indication of what
>> meaning and/or what syntax the data has, and then let tools decide what
>> to do, based on that information.
>>
>> We already have one data-type indicator: svn:mime-type. Now, MIME type
>> is far from a complete data type specifier. It is insufficient for our
>> needs, in theory. However, in practice, it is nearly sufficient. (See
>> Problem 2 below for an exception.)
>>
>> We also have another data-type indicator: the file name. A file name is
>> also an incomplete source of metadata, and some file names ("README" or
>> "CHANGES") give no indication at all of the format, but it is useful in
>> many cases ("*.py", "*.c").
>
> We also potentially have a third indicator that can be useful: a scan of
> the content to determine whether it is mostly plain text (implemented by
> the mis-named svn_io_detect_mimetype()).
>
>
>> Problem 1 (limited recognition of MIME types)
>> =========
>>
>> Subversion mis-categorizes a lot of MIME types as "binary" (and
>> therefore will not merge or diff or blame them) which really are
>> line-based text formats.
>>
>> The list of such MIME types is continually evolving so it is not
>> possible for Subversion to have a built-in complete list. However, it is
>> easy for new releases of Subversion to have an updated list.
>
> I wonder if I have missed some good reason not to update the built-in
> list of MIME types that are considered "text". Just a constraint on the
> available volunteer effort? I know it's not the 100% solution but it is
> a 99% solution and UI am surprised we are not automatically starting
> here.
(1) It's an ongoing maintenance headache.
(2) The merge behavior of existing repositories will change,
surprising some, no doubt.
(3) *The mapping from MIME-type to 'mergeable' is ambiguous.*
Consider, again, application/xml. No amount of "updating the built-in
list of MIME types" is going to help here.
>> It is not much harder for Subversion to have a configuarable list of
>> which MIME types (or MIME type patterns) should be considered mergeable.
>> (The configuration could be extensible: it could say line-wise-mergeable
>> or not mergeable or XML-mergeable or ...)
>
> [...]
>
>> Solution 0 (merge-mode)
>> ==========
>>
>> So we could add a property to each file which says whether the file is
>> to be considered line-wise mergeable by Subversion, and say that this
>> property will be the primary source of this information. What are the
>> pros and cons of this?
>
> Big con: Depends on auto-props, a mechanism which is not really good
> enough for the task. (Strictly speaking it doesn't, because users can
> set the property manually, but in a practical sense it does because in a
> production environment it is impractical for users to consistently set
> the property manually.)
I find this criticism misleading. It leaves the impression that the
OP's proposal depends exclusively on the merge-mode property to
determine merging behavior. This is not the case. Merge-mode is only
needed when the existing heuristics -- which remain unchanged by the
proposal -- prove insufficient to the task.
To wit:
On Tue, Nov 17, 2009 at 21:56, Mike Samuel <mi...@gmail.com> wrote:
>
> Subversion treats the following files as [[mergable]]:
>
> * Files with no svn:mime-type [[and no svn:merge-mode]]
> * Files with a svn:mime-type starting "text/"
> * Files with a svn:mime-type equal to "image/x-xbitmap"
> * Files with a svn:mime-type equal to "image/x-xpixmap"
> * [[Files with a svn:merge-mode that is equal to "simple"]]
>
Furthermore, the need for coordinating autoprops within a team exists
without this proposal, independent of issues of merge behavior. I fail
to see how the OP's proposal will in any way exacerbate that.
I've got about 100 lines of autoprops in the subversion config file I
maintain for my team. This is how we try to assure that most of our
files get reasonable mime-type, needs-lock, eol-style, from the
get-go. Periodically I fix things up with a bash script or two. The
need for this won't be affected by the merge-mode proposal.
// Ben
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419468
Re: Merge mode (Was Classifying files as binary or text)
Posted by Branko Cibej <br...@xbc.nu>.
Julian Foad wrote:
>> * there never was consensus for the "obvious" ones, for the simple
>> reason that they're not always obvious -- e.g., consider
>> application/xml, which sometimes can and sometimes can't be
>> properly merged.
>>
>
> We don't need to try to be too clever. Either we decide that Subversion
> shall do line-wise blame/diff/merge on these kinds of formats, or we
> decide that it won't. People can plug in their own merge tools if they
> need more advanced behaviour, so I think we should make these formats be
> classified as "text" and therefore use blame/diff/merge on them.
>
We'd be taking it upon ourselves to provide a definite answer to an
indeterminate question, and the only result would be that we'd piss off
the other half of our users.
[...]
>> But more about that in the "real" response.
>>
>
> Please first make sure you see my reply to B. Smith-Mannschott in this
> thread, in which my text begins "Ah, yes, it is illogical".
>
Yup, I will; have read it but now must rush. (It's a bit uncomfortable
to jump the gun with responses in this way ... please bear with me.)
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419470
Re: Merge mode (Was Classifying files as binary or text)
Posted by Julian Foad <ju...@btopenworld.com>.
On Wed, 2009-11-18 at 12:16 +0100, Branko Čibej wrote:
> Julian Foad wrote:
> > Julian Foad wrote:
> >
> >> I need to spend some time replying, late at night though it is.
> >>
>
> [...] (I'll respond to your original proposal later, but some quick
> notes here)
>
> > I wonder if I have missed some good reason not to update the built-in
> > list of MIME types that are considered "text". Just a constraint on the
> > available volunteer effort? I know it's not the 100% solution but it is
> > a 99% solution and UI am surprised we are not automatically starting
> > here.
> >
>
> IIRC there are two reasons why we haven't updated that list:
>
> * doing it in code is a never-ending battle to catch up with IANA;
Sure, but while it's in the code that's not a reason not to keep
updating it.
Meanwhile we should consider getting it out of the code, into some
configuration place. That is the direction in which my proposal heads.
> * there never was consensus for the "obvious" ones, for the simple
> reason that they're not always obvious -- e.g., consider
> application/xml, which sometimes can and sometimes can't be
> properly merged.
We don't need to try to be too clever. Either we decide that Subversion
shall do line-wise blame/diff/merge on these kinds of formats, or we
decide that it won't. People can plug in their own merge tools if they
need more advanced behaviour, so I think we should make these formats be
classified as "text" and therefore use blame/diff/merge on them.
> [...]
>
> > Big con: Depends on auto-props, a mechanism which is not really good
> > enough for the task.
>
> ... because it's strictly client-side, *and* because it relies only on
> file names; which, BTW, is a serious problem with your proposal.
None of my proposals relied only on file names.
> But more about that in the "real" response.
Please first make sure you see my reply to B. Smith-Mannschott in this
thread, in which my text begins "Ah, yes, it is illogical".
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419469
Re: Merge mode (Was Classifying files as binary or text)
Posted by Branko Cibej <br...@xbc.nu>.
Julian Foad wrote:
> Julian Foad wrote:
>
>> I need to spend some time replying, late at night though it is.
>>
[...] (I'll respond to your original proposal later, but some quick
notes here)
> I wonder if I have missed some good reason not to update the built-in
> list of MIME types that are considered "text". Just a constraint on the
> available volunteer effort? I know it's not the 100% solution but it is
> a 99% solution and UI am surprised we are not automatically starting
> here.
>
IIRC there are two reasons why we haven't updated that list:
* doing it in code is a never-ending battle to catch up with IANA;
* there never was consensus for the "obvious" ones, for the simple
reason that they're not always obvious -- e.g., consider
application/xml, which sometimes can and sometimes can't be
properly merged.
[...]
> Big con: Depends on auto-props, a mechanism which is not really good
> enough for the task.
... because it's strictly client-side, *and* because it relies only on
file names; which, BTW, is a serious problem with your proposal. But
more about that in the "real" response.
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419457
Re: Merge mode (Was Classifying files as binary or text)
Posted by Julian Foad <ju...@btopenworld.com>.
Julian Foad wrote:
> I need to spend some time replying, late at night though it is.
>
> Let me try to explain why I think a "how to merge" property should not
> be the primary indicator of how subversion should merge each file.
Some comments on my own posting...
> Principle
> =========
>
> I have read that, in the realm of data handling, there is a principle
> that it is a bad idea to tag data with annotations that say what kind of
> actions can or should be performed on it. That kind of coupling is
> unscalable. Instead, it is better to tag data with an indication of what
> meaning and/or what syntax the data has, and then let tools decide what
> to do, based on that information.
>
> We already have one data-type indicator: svn:mime-type. Now, MIME type
> is far from a complete data type specifier. It is insufficient for our
> needs, in theory. However, in practice, it is nearly sufficient. (See
> Problem 2 below for an exception.)
>
> We also have another data-type indicator: the file name. A file name is
> also an incomplete source of metadata, and some file names ("README" or
> "CHANGES") give no indication at all of the format, but it is useful in
> many cases ("*.py", "*.c").
We also potentially have a third indicator that can be useful: a scan of
the content to determine whether it is mostly plain text (implemented by
the mis-named svn_io_detect_mimetype()).
> Problem 1 (limited recognition of MIME types)
> =========
>
> Subversion mis-categorizes a lot of MIME types as "binary" (and
> therefore will not merge or diff or blame them) which really are
> line-based text formats.
>
> The list of such MIME types is continually evolving so it is not
> possible for Subversion to have a built-in complete list. However, it is
> easy for new releases of Subversion to have an updated list.
I wonder if I have missed some good reason not to update the built-in
list of MIME types that are considered "text". Just a constraint on the
available volunteer effort? I know it's not the 100% solution but it is
a 99% solution and UI am surprised we are not automatically starting
here.
> It is not much harder for Subversion to have a configuarable list of
> which MIME types (or MIME type patterns) should be considered mergeable.
> (The configuration could be extensible: it could say line-wise-mergeable
> or not mergeable or XML-mergeable or ...)
[...]
> Solution 0 (merge-mode)
> ==========
>
> So we could add a property to each file which says whether the file is
> to be considered line-wise mergeable by Subversion, and say that this
> property will be the primary source of this information. What are the
> pros and cons of this?
Big con: Depends on auto-props, a mechanism which is not really good
enough for the task. (Strictly speaking it doesn't, because users can
set the property manually, but in a practical sense it does because in a
production environment it is impractical for users to consistently set
the property manually.)
- Julian
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419449
Re: Merge mode (Was Classifying files as binary or text)
Posted by Julian Foad <ju...@btopenworld.com>.
I need to spend some time replying, late at night though it is.
Let me try to explain why I think a "how to merge" property should not
be the primary indicator of how subversion should merge each file.
Principle
=========
I have read that, in the realm of data handling, there is a principle
that it is a bad idea to tag data with annotations that say what kind of
actions can or should be performed on it. That kind of coupling is
unscalable. Instead, it is better to tag data with an indication of what
meaning and/or what syntax the data has, and then let tools decide what
to do, based on that information.
We already have one data-type indicator: svn:mime-type. Now, MIME type
is far from a complete data type specifier. It is insufficient for our
needs, in theory. However, in practice, it is nearly sufficient. (See
Problem 2 below for an exception.)
We also have another data-type indicator: the file name. A file name is
also an incomplete source of metadata, and some file names ("README" or
"CHANGES") give no indication at all of the format, but it is useful in
many cases ("*.py", "*.c").
Problem 1 (limited recognition of MIME types)
=========
Subversion mis-categorizes a lot of MIME types as "binary" (and
therefore will not merge or diff or blame them) which really are
line-based text formats.
The list of such MIME types is continually evolving so it is not
possible for Subversion to have a built-in complete list. However, it is
easy for new releases of Subversion to have an updated list.
It is not much harder for Subversion to have a configuarable list of
which MIME types (or MIME type patterns) should be considered mergeable.
(The configuration could be extensible: it could say line-wise-mergeable
or not mergeable or XML-mergeable or ...)
Problem 2 (mergeable and non-mergeable XML files)
=========
A user has some XML files on which a line-based merge is useful, and
some XML files on which that is not useful, and wishes to label both
kinds with svn:mime-type=text/xml. Let us suppose one format has each
XML tag on a separate line, and the other has them all run together and
line breaks inserted at arbitrary places in the sequence. It may be
possible to find a different MIME type for one of the file types, but
that may well not be possible, e.g. if they are both proprietary or
arbitrary XML formats.
This problem is not limited to XML files. Consider two "plain text"
files, MIME type text/plain, with different kinds of text in them. One
has line-based content, such as a shopping list, and changes usually
leave many lines unchanged. The other contains the text of a newspaper
article with line breaks at roughly every 70 characters in a stream of
words, so two similar versions of it may have very few whole lines in
common.
I believe this is a real but relatively uncommon requirement. It is a
genuine example of the MIME being insufficient to determine (line-wise)
mergeability. There are many file formats that can be regarded as being
(line-wise) mergeable or non-mergeable depending on some aspect of their
content that cannot be reflected in the MIME type. It is uncommon in the
sense that most Subversion users' needs can be satisfied by
distinguishing mergeability based on the MIME type, or better the MIME
type and file name taken together, of their files.
To solve this problem when it exists, there does indeed need to be
further metadata about the content type of the file. (Alternatively it
could be metadata that says how to merge the file, but see "Principle"
above.)
Solution 0 (merge-mode)
==========
So we could add a property to each file which says whether the file is
to be considered line-wise mergeable by Subversion, and say that this
property will be the primary source of this information. What are the
pros and cons of this?
Pro: The user can force a line-wise merge on one file and no merge
attempt on another file even when MIME type and file name are
insufficient distinguishers.
Pro: The user can forget about providing MIME type at all, and just set
this property to one of the pre-defined two types of merging (line-based
or none), if that is all the user cares about.
Con: This property associates the file with one simple kind of merging;
but the best merge tool available on the client may not be that simple
kind. If we want to use a better merge tool, say an XML-aware merge
tool, this property actually gets in the way: it tells us to use a
simple line-based merge on this file. It would have been better if the
property had said, "this file contains line-based XML, so you might want
to use an XML-aware merge rather than a simple line-based merge if you
can". In other words, we really want to tell the client what the content
type is, and let the client choose the best merge tool for that content
type.
Con: This property conveys redundant information. In almost all cases,
the MIME type and/or file name are sufficient information. It is wrong
to pretend that MIME type and file name are not good sources of
how-to-merge information, and to leave their currently weak and
deficient interpretation as just a deprecated backward-compatibility
fallback.
Con: Not extensible to diff, blame, etc. An indication that the file is
line-wise-mergeable is not really a good indication of whether the file
can be line-wise diffed or blamed.
Proposal
========
This is the full, long-term proposal. We can choose a subset of this to
do initially.
(1) Make svn merge/diff/blame take into account the file name as well as
the svn:mime-type in deciding whether to operate in a "line-wise" mode
or not operate at all.
(2) Update the built-in MIME type and filename patterns.
* Update the built-in selection based on svn:mime-type to recognise a
list of MIME types that is reasonably up-to-date right now (even though
it will be out of date by the time the released software is in use).
* Update the built-in selection based on file names to recognize a
reasonable list of file name patterns.
(3) Provide a client-side config for extending and overriding the rules
that map MIME type and file name to a merge/diff/blame mode. This mode
should be specifiable in the config, not just "line-wise" or "none" but
any other named mode. Provide config options for specifying the merge
tool, diff tool and blame tool per mode. Tools should be specifiable as
none, built-in or external.
(4) Add an optional property for selecting a particular merge mode (and
diff mode and blame mode) for the cases where (1) and (2) are
insufficient or inconvenient.
Regards,
- Julian
Mike Samuel wrote:
> Proposal:
> ========
> (1) Add documentation on the svn:merge-mode property that lists the
> allowed values as ("simple" and "none")
> (2) Add example autoprops rules to the documentation that sets
> svn:merge-mode to "simple" for the following file types
> application/ecmascript
> application/json
> application/xml
> image/svg+xml
> (3) Change the text quoted from the SVN manual under Background to
> read as below.
> (4) Update the implementation to agree.
>
> Subversion treats the following files as [[mergable]]:
>
> * Files with no svn:mime-type [[and no svn:merge-mode]]
> * Files with a svn:mime-type starting "text/"
> * Files with a svn:mime-type equal to "image/x-xbitmap"
> * Files with a svn:mime-type equal to "image/x-xpixmap"
> * [[Files with a svn:merge-mode that is equal to "simple"]]
>
> All other files are treated as [[unmergeable]], meaning that
> Subversion will:
>
> * Not attempt to automatically merge received changes with
> local changes during svn update or svn merge
> * Not show the differences as part of svn diff
> * Not show line-by-line attribution for svn blame
>
> In all other respects, Subversion treats [[mergable]] files the
> same as [[unmergeable]] files, e.g. if you set
> the svn:keywords or svn:eol-style properties, Subversion will
> perform keyword substitution
> or newline conversion on [[unmergeable]] files.
>
>
> Goal:
> ====
> To update the scheme by which svn {update,diff,merge,blame} to allow
> merging of files
> with svn:mime-type outside the hard-coded list currently used.
>
> This determination should be independent of the platform svn
> is running on, so independent of the set of supported character sets.
>
> This scheme should not complicate future extensions to the merge
> system which might wish to use a different merge policy, e.g. for XML
> than for source code files.
>
> This scheme should work with autoprops, and other mechanisms repository
> administrators use to manage files. Specifically, some kinds of XML can
> be meaningfully meged, and others cannot.
>
> This scheme should work within existing limitations, such as the inability
> to merge UTF-16 and UTF-32.
>
>
> Background:
> ==========
> The current behavior is described at
> http://subversion.tigris.org/faq.html#binary-files
>
> Subversion treats the following files as text:
>
> * Files with no svn:mime-type
> * Files with a svn:mime-type starting "text/"
> * Files with a svn:mime-type equal to "image/x-xbitmap"
> * Files with a svn:mime-type equal to "image/x-xpixmap"
>
> All other files are treated as binary, meaning that Subversion will:
>
> * Not attempt to automatically merge received changes with
> local changes during svn update or svn merge
> * Not show the differences as part of svn diff
> * Not show line-by-line attribution for svn blame
>
> In all other respects, Subversion treats binary files the same as
> text files, e.g. if you set
> the svn:keywords or svn:eol-style properties, Subversion will
> perform keyword substitution
> or newline conversion on binary files.
>
> Common source code mime-types are misclassified, and that problem is
> likely to grow because of current IANA policy.
> Mime-types are handed out by the IANA, which only assigns text/*
> mime-types for file-types that are meant to be human readable. Source
> code is explicitly not considered human readable. This is why many
> source code and data mime-types are in the application/* group or
> other non text/* groups: application/json, application/ecmascript,
> application/xml, image/svg+xml.
> RFC 4288 ( ftp://ftp.rfc-editor.org/in-notes/rfc4288.txt ) says this
> Expected uses for the "application" media type
> include but are not limited to file transfer, spreadsheets,
> presentations, scheduling data, and languages for "active"
> (computational) material.
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419155
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2419275