You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Daniel Näslund <da...@longitudo.com> on 2009/11/18 10:37:01 UTC
[PATCH] Do not display a binary prop diff in interactive conflict
resolver
Hi!
[[[
Do not display a binary property diff in the interactive conflict
resolver.
* subversion/include/svn_io.h
(svn_stream_detect_binary_mimetype): New. Declare.
* subversion/libsvn_subr/stream.c
(svn_stream_detect_binary_mimetype): Check if atleast 85 % of the
bytes fall within the ascii range of printable characters.
* subversion/libsvn_wc/props.c
(maybe_generate_propconflict): Call ..binary_mimetype() on the
mergestream. Set the conflict description according to the outcome.
* subversion/svn/conflict-callbacks.c
(svn_cl__conflict_handler): Print err msg if desc->is_binary when
invoking 'diff-full'.
Patch by: Daniel N�slund <daniel{_AT_}longitudo.com>
]]]
A recent thread started by Mike Samuel [1] discussed various strategies
for improving the detection of 'mergeability'. This patch only uses
existing solutions. I've taken a piece of code from
svn_io_detect_mimetype2() for detecting if a file buffer is binary. If
it is then the mime_type is set to "application/octet-stream". From the
mime-type it is determined if the property is binary or not.
In a follow-up patch I intend to extract the duplicated code for
detecting binary content into a libsvn_subr private function. Why not
get crazy and link in libmagic (used by the UNIX file command) to
determine what sort of content we have? Right now it says that anything
besides printable ascii is binary.
I'm attaching a script for testing this patch. Since I'm checking if the
content is binary on the mergestream I will end up with the diff beeing
displayed if it's below a certain treshold (when the number of
printable unidiff characters is more than 15 % of the total amount of
bytes.
Is it ok to have it like that or should I test for each of the original
streams instead? This was an easy solution. Perhaps too easy?
make check passed.
/Daniel
[
Re: [PATCH] Do not display a binary prop diff in interactive
conflict resolver
Posted by Daniel Näslund <da...@longitudo.com>.
> >> [[[
> >> Do not display a binary property diff in the interactive conflict
> >> resolver.
> >>
> >> * subversion/include/svn_io.h
> >> (svn_stream_detect_binary_mimetype): New. Declare.
> >>
> >> * subversion/libsvn_subr/stream.c
> >> (svn_stream_detect_binary_mimetype): Check if atleast 85 % of the
> >> bytes fall within the ascii range of printable characters.
> >>
> >> * subversion/libsvn_wc/props.c
> >> (maybe_generate_propconflict): Call ..binary_mimetype() on the
> >> mergestream. Set the conflict description according to the outcome.
> >>
> >> * subversion/svn/conflict-callbacks.c
> >> (svn_cl__conflict_handler): Print err msg if desc->is_binary when
> >> invoking 'diff-full'.
> >>
> >> Patch by: Daniel Näslund <daniel{_AT_}longitudo.com>
> >> ]]]
On Fri, Dec 04, 2009 at 11:42:06AM +1100, Gavin 'Beau' Baumanis wrote:
> Ping.
> This Patch submission has received no new comments.
I'm giving up on this one for now. I wanted to detect whether some
properties were binary or not using the mechanism in
svn_io_detect_binary_mimetype() but that one is flawed and only checks
if text is outside the printable ASCII range. Japanese text will be
treated as binary!
I'm hoping that something will come out of the discussion in [1]
regarding how to classify files as binary or text. My own humble
suggestion was to use libmagic. I think that somewhere within that
the aboved mentioned thread someone said that libmagic has flaws too but
I can't find it now.
In the end, how often do we use binary properties? Not often at all is
my guess?
Daniel
[1] http://svn.haxx.se/dev/archive-2009-11/index.shtml#536
Re: [PATCH] Do not display a binary prop diff in interactive
conflict resolver
Posted by Daniel Näslund <da...@longitudo.com>.
> >> [[[
> >> Do not display a binary property diff in the interactive conflict
> >> resolver.
> >>
> >> * subversion/include/svn_io.h
> >> (svn_stream_detect_binary_mimetype): New. Declare.
> >>
> >> * subversion/libsvn_subr/stream.c
> >> (svn_stream_detect_binary_mimetype): Check if atleast 85 % of the
> >> bytes fall within the ascii range of printable characters.
> >>
> >> * subversion/libsvn_wc/props.c
> >> (maybe_generate_propconflict): Call ..binary_mimetype() on the
> >> mergestream. Set the conflict description according to the outcome.
> >>
> >> * subversion/svn/conflict-callbacks.c
> >> (svn_cl__conflict_handler): Print err msg if desc->is_binary when
> >> invoking 'diff-full'.
> >>
> >> Patch by: Daniel Näslund <daniel{_AT_}longitudo.com>
> >> ]]]
On Fri, Dec 04, 2009 at 11:42:06AM +1100, Gavin 'Beau' Baumanis wrote:
> Ping.
> This Patch submission has received no new comments.
I'm giving up on this one for now. I wanted to detect whether some
properties were binary or not using the mechanism in
svn_io_detect_binary_mimetype() but that one is flawed and only checks
if text is outside the printable ASCII range. Japanese text will be
treated as binary!
I'm hoping that something will come out of the discussion in [1]
regarding how to classify files as binary or text. My own humble
suggestion was to use libmagic. I think that somewhere within that
the aboved mentioned thread someone said that libmagic has flaws too but
I can't find it now.
In the end, how often do we use binary properties? Not often at all is
my guess?
Daniel
[1] http://svn.haxx.se/dev/archive-2009-11/index.shtml#536
Re: [PATCH] Do not display a binary prop diff in interactive conflict resolver
Posted by Gavin 'Beau' Baumanis <ga...@thespidernet.com>.
Ping.
This Patch submission has received no new comments.
Beau.
On 20/11/2009, at 06:00 , Daniel Näslund wrote:
> On Wed, Nov 18, 2009 at 10:37:01AM +0100, Daniel Näslund wrote:
>> Hi!
>>
>> [[[
>> Do not display a binary property diff in the interactive conflict
>> resolver.
>>
>> * subversion/include/svn_io.h
>> (svn_stream_detect_binary_mimetype): New. Declare.
>>
>> * subversion/libsvn_subr/stream.c
>> (svn_stream_detect_binary_mimetype): Check if atleast 85 % of the
>> bytes fall within the ascii range of printable characters.
>>
>> * subversion/libsvn_wc/props.c
>> (maybe_generate_propconflict): Call ..binary_mimetype() on the
>> mergestream. Set the conflict description according to the outcome.
>>
>> * subversion/svn/conflict-callbacks.c
>> (svn_cl__conflict_handler): Print err msg if desc->is_binary when
>> invoking 'diff-full'.
>>
>> Patch by: Daniel Näslund <daniel{_AT_}longitudo.com>
>> ]]]
>
> svn_stream_detect_binary_mimetype() checked if more than 85 % of the
> content was "binary", then it set a the mimetype to
> "application/octetstream".
>
> This patch only checked if the mergestream was binary. Since the
> mergestream contains both diff headers and hunk symbols, small binary
> content was falsely said to be positive.
>
> Furthermore it used a detection algorithm that considered all characters
> outside the ascii range to be binary. When testing with chinese texts
> the resolver said that the content was binary.
>
> Furthermore, since it only checked the first kb of the mergestream it
> would not detect a situation where only the last part of the mergestream
> was binary.
>
> The two last bugs should also be a problem in svn_io_detect_mimetype2()
> from which I extracted the algorithm for detecting binary content. That
> function is used in the autoprop stuff.
>
> In the end: How often do we set a binary property? I'm starting to
> believe that I'm trying to fix something that doesn't need fixing. Since
> a property is generally small we could still get a useful diff in some
> cases where only one of the files is binary. If we escaped the control
> chars then there would be no harm.
>
> /Daniel
Re: [PATCH] Do not display a binary prop diff in interactive conflict resolver
Posted by Gavin 'Beau' Baumanis <ga...@thespidernet.com>.
Ping.
This Patch submission has received no new comments.
Beau.
On 20/11/2009, at 06:00 , Daniel Näslund wrote:
> On Wed, Nov 18, 2009 at 10:37:01AM +0100, Daniel Näslund wrote:
>> Hi!
>>
>> [[[
>> Do not display a binary property diff in the interactive conflict
>> resolver.
>>
>> * subversion/include/svn_io.h
>> (svn_stream_detect_binary_mimetype): New. Declare.
>>
>> * subversion/libsvn_subr/stream.c
>> (svn_stream_detect_binary_mimetype): Check if atleast 85 % of the
>> bytes fall within the ascii range of printable characters.
>>
>> * subversion/libsvn_wc/props.c
>> (maybe_generate_propconflict): Call ..binary_mimetype() on the
>> mergestream. Set the conflict description according to the outcome.
>>
>> * subversion/svn/conflict-callbacks.c
>> (svn_cl__conflict_handler): Print err msg if desc->is_binary when
>> invoking 'diff-full'.
>>
>> Patch by: Daniel Näslund <daniel{_AT_}longitudo.com>
>> ]]]
>
> svn_stream_detect_binary_mimetype() checked if more than 85 % of the
> content was "binary", then it set a the mimetype to
> "application/octetstream".
>
> This patch only checked if the mergestream was binary. Since the
> mergestream contains both diff headers and hunk symbols, small binary
> content was falsely said to be positive.
>
> Furthermore it used a detection algorithm that considered all characters
> outside the ascii range to be binary. When testing with chinese texts
> the resolver said that the content was binary.
>
> Furthermore, since it only checked the first kb of the mergestream it
> would not detect a situation where only the last part of the mergestream
> was binary.
>
> The two last bugs should also be a problem in svn_io_detect_mimetype2()
> from which I extracted the algorithm for detecting binary content. That
> function is used in the autoprop stuff.
>
> In the end: How often do we set a binary property? I'm starting to
> believe that I'm trying to fix something that doesn't need fixing. Since
> a property is generally small we could still get a useful diff in some
> cases where only one of the files is binary. If we escaped the control
> chars then there would be no harm.
>
> /Daniel
Re: [PATCH] Do not display a binary prop diff in interactive
conflict resolver
Posted by Daniel Näslund <da...@longitudo.com>.
On Wed, Nov 18, 2009 at 10:37:01AM +0100, Daniel Näslund wrote:
> Hi!
>
> [[[
> Do not display a binary property diff in the interactive conflict
> resolver.
>
> * subversion/include/svn_io.h
> (svn_stream_detect_binary_mimetype): New. Declare.
>
> * subversion/libsvn_subr/stream.c
> (svn_stream_detect_binary_mimetype): Check if atleast 85 % of the
> bytes fall within the ascii range of printable characters.
>
> * subversion/libsvn_wc/props.c
> (maybe_generate_propconflict): Call ..binary_mimetype() on the
> mergestream. Set the conflict description according to the outcome.
>
> * subversion/svn/conflict-callbacks.c
> (svn_cl__conflict_handler): Print err msg if desc->is_binary when
> invoking 'diff-full'.
>
> Patch by: Daniel Näslund <daniel{_AT_}longitudo.com>
> ]]]
svn_stream_detect_binary_mimetype() checked if more than 85 % of the
content was "binary", then it set a the mimetype to
"application/octetstream".
This patch only checked if the mergestream was binary. Since the
mergestream contains both diff headers and hunk symbols, small binary
content was falsely said to be positive.
Furthermore it used a detection algorithm that considered all characters
outside the ascii range to be binary. When testing with chinese texts
the resolver said that the content was binary.
Furthermore, since it only checked the first kb of the mergestream it
would not detect a situation where only the last part of the mergestream
was binary.
The two last bugs should also be a problem in svn_io_detect_mimetype2()
from which I extracted the algorithm for detecting binary content. That
function is used in the autoprop stuff.
In the end: How often do we set a binary property? I'm starting to
believe that I'm trying to fix something that doesn't need fixing. Since
a property is generally small we could still get a useful diff in some
cases where only one of the files is binary. If we escaped the control
chars then there would be no harm.
/Daniel
Re: [PATCH] Do not display a binary prop diff in interactive
conflict resolver
Posted by Julian Foad <ju...@btopenworld.com>.
Daniel Näslund wrote:
> [[[
> Do not display a binary property diff in the interactive conflict
> resolver.
The concept of a "binary property" is new in this patch, so a few
additional words would be useful here, to say what change you are
making:
Detect heuristically whether the property value is "text" or "binary"
and only allow an interactive diff and merge if it is "text".
> * subversion/include/svn_io.h
> (svn_stream_detect_binary_mimetype): New. Declare.
Please do not use "mimetype" in the name of this function, nor in its
result value. It has nothing to do with MIME types, it is just trying to
decide whether the content can be displayed as lines of text or not.
I would suggest something like "content_is_textual", with a Boolean
result type.
> * subversion/libsvn_subr/stream.c
> (svn_stream_detect_binary_mimetype): Check if atleast 85 % of the
> bytes fall within the ascii range of printable characters.
>
> * subversion/libsvn_wc/props.c
> (maybe_generate_propconflict): Call ..binary_mimetype() on the
> mergestream. Set the conflict description according to the
> outcome.
>
> * subversion/svn/conflict-callbacks.c
> (svn_cl__conflict_handler): Print err msg if desc->is_binary when
> invoking 'diff-full'.
>
> Patch by: Daniel Näslund <daniel{_AT_}longitudo.com>
> ]]]
> As I'm saying further down. I have only duplicated a part of
> svn_io_detect_mimetype2() and intend to refactor this part into a helper
> func in libsvn_subr.
Hi Daniel. Would you be able to factor out the code first, and then
build on it afterwards or at the same time? It makes me uncomfortable
when we add duplicated code, even when there is an intention to
un-duplicate it later.
- Julian
Re: [PATCH] Do not display a binary prop diff in interactive
conflict resolver
Posted by Stefan Sperling <st...@elego.de>.
On Wed, Nov 18, 2009 at 11:01:51AM +0100, Daniel Näslund wrote:
> On Wed, Nov 18, 2009 at 10:49:15AM +0100, Stefan Sperling wrote:
> > On Wed, Nov 18, 2009 at 10:37:01AM +0100, Daniel Näslund wrote:
> > > Index: subversion/libsvn_subr/stream.c
> > > ===================================================================
> > > --- subversion/libsvn_subr/stream.c (revision 881392)
> > > +++ subversion/libsvn_subr/stream.c (arbetskopia)
> > > @@ -1347,3 +1347,44 @@
> > >
> > > return SVN_NO_ERROR;
> > > }
> > > +
> > > +svn_error_t *
> > > +svn_stream_detect_binary_mimetype(const char **mimetype,
> > > + svn_stream_t *stream)
> > > +{
> > > + static const char * const generic_binary = "application/octet-stream";
> > > + char block[1024];
> > > + apr_size_t amt_read = sizeof(block);
> > > +
> > > + /* Default return value is NULL. */
> > > + *mimetype = NULL;
> > > +
> > > + SVN_ERR(svn_stream_read(stream, block, &amt_read));
> > > +
> > > + if (amt_read > 0)
> > > + {
> > > + apr_size_t i;
> > > + apr_size_t binary_count = 0;
> > > +
> > > + for (i = 0; i < amt_read; i++)
> > > + {
> > > + if (block[i] == 0)
> > > + {
> > > + binary_count = amt_read;
> > > + break;
> > > + }
> > > + if ((block[i] < 0x07)
> > > + || ((block[i] > 0x0D) && (block[i] < 0x20))
> > > + || (block[i] > 0x7F))
> > > + {
> > > + binary_count++;
> > > + }
> >
> > Unless I'm mistaken the "greater 0x7F" check will trigger on *any* UTF-8
> > continuation byte. See http://tools.ietf.org/html/rfc3629#section-3
>
> Yes, it will and this code is used for all the autoprops stuff! But
> strange results has been hidden by the fact that the detection code
> first checks for file endings. That's my guess atleast. A japanese text
> would be considered binary!
>
> As I'm saying further down. I have only duplicated a part of
> svn_io_detect_mimetype2() and intend to refactor this part into a helper
> func in libsvn_subr.
Using libmagic is possible, at least from a legal point of view.
It has a very simple 2-clause BSD-style license so we could link to it.
Stefan
Re: [PATCH] Do not display a binary prop diff in interactive
conflict resolver
Posted by Daniel Näslund <da...@longitudo.com>.
On Wed, Nov 18, 2009 at 10:49:15AM +0100, Stefan Sperling wrote:
> On Wed, Nov 18, 2009 at 10:37:01AM +0100, Daniel Näslund wrote:
> > Index: subversion/libsvn_subr/stream.c
> > ===================================================================
> > --- subversion/libsvn_subr/stream.c (revision 881392)
> > +++ subversion/libsvn_subr/stream.c (arbetskopia)
> > @@ -1347,3 +1347,44 @@
> >
> > return SVN_NO_ERROR;
> > }
> > +
> > +svn_error_t *
> > +svn_stream_detect_binary_mimetype(const char **mimetype,
> > + svn_stream_t *stream)
> > +{
> > + static const char * const generic_binary = "application/octet-stream";
> > + char block[1024];
> > + apr_size_t amt_read = sizeof(block);
> > +
> > + /* Default return value is NULL. */
> > + *mimetype = NULL;
> > +
> > + SVN_ERR(svn_stream_read(stream, block, &amt_read));
> > +
> > + if (amt_read > 0)
> > + {
> > + apr_size_t i;
> > + apr_size_t binary_count = 0;
> > +
> > + for (i = 0; i < amt_read; i++)
> > + {
> > + if (block[i] == 0)
> > + {
> > + binary_count = amt_read;
> > + break;
> > + }
> > + if ((block[i] < 0x07)
> > + || ((block[i] > 0x0D) && (block[i] < 0x20))
> > + || (block[i] > 0x7F))
> > + {
> > + binary_count++;
> > + }
>
> Unless I'm mistaken the "greater 0x7F" check will trigger on *any* UTF-8
> continuation byte. See http://tools.ietf.org/html/rfc3629#section-3
Yes, it will and this code is used for all the autoprops stuff! But
strange results has been hidden by the fact that the detection code
first checks for file endings. That's my guess atleast. A japanese text
would be considered binary!
As I'm saying further down. I have only duplicated a part of
svn_io_detect_mimetype2() and intend to refactor this part into a helper
func in libsvn_subr.
/Daniel
Re: [PATCH] Do not display a binary prop diff in interactive
conflict resolver
Posted by Stefan Sperling <st...@elego.de>.
On Wed, Nov 18, 2009 at 10:37:01AM +0100, Daniel Näslund wrote:
> Index: subversion/libsvn_subr/stream.c
> ===================================================================
> --- subversion/libsvn_subr/stream.c (revision 881392)
> +++ subversion/libsvn_subr/stream.c (arbetskopia)
> @@ -1347,3 +1347,44 @@
>
> return SVN_NO_ERROR;
> }
> +
> +svn_error_t *
> +svn_stream_detect_binary_mimetype(const char **mimetype,
> + svn_stream_t *stream)
> +{
> + static const char * const generic_binary = "application/octet-stream";
> + char block[1024];
> + apr_size_t amt_read = sizeof(block);
> +
> + /* Default return value is NULL. */
> + *mimetype = NULL;
> +
> + SVN_ERR(svn_stream_read(stream, block, &amt_read));
> +
> + if (amt_read > 0)
> + {
> + apr_size_t i;
> + apr_size_t binary_count = 0;
> +
> + for (i = 0; i < amt_read; i++)
> + {
> + if (block[i] == 0)
> + {
> + binary_count = amt_read;
> + break;
> + }
> + if ((block[i] < 0x07)
> + || ((block[i] > 0x0D) && (block[i] < 0x20))
> + || (block[i] > 0x7F))
> + {
> + binary_count++;
> + }
Unless I'm mistaken the "greater 0x7F" check will trigger on *any* UTF-8
continuation byte. See http://tools.ietf.org/html/rfc3629#section-3
Stefan
Re: [PATCH] Do not display a binary prop diff in interactive conflict
resolver
Posted by "C. Michael Pilato" <cm...@collab.net>.
Daniel Näslund wrote:
> * subversion/libsvn_subr/stream.c
> (svn_stream_detect_binary_mimetype): Check if atleast 85 % of the
> bytes fall within the ascii range of printable characters.
Was there no good way to factor out similar code from
svn_io_detect_mimetype2()? Looks like a bunch of overlap.
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Distributed Development On Demand