You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Edward d'Auvergne <tr...@gmail.com> on 2015/10/02 10:06:26 UTC
Bug report: The auto-props setting of svn:mime-type is impossible to avoid.
Hi all,
I was wondering if this should be considered a bug. At the FlightGear
project we have a 6 GB data svn repository for aircraft (
https://sourceforge.net/projects/flightgear/ ,
https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/ ). A large
quantity of the files, almost 30,000 in number, are XML text files.
The problem is that modern Subversion clients appear to all be using
libmagic to determine the file type, to then set the svn:mime-type
property. However this is rather problematic:
$ file -ib Aircraft/747-400/747-400-set.xml
application/xml; charset=us-ascii
Libmagic detects all these XML files as "application/xml". Hence they
are treated as binary files! There are many other text files with
extensions such as *.ac, *.nas, etc. present in the repository that in
the future might be detected by libmagic as "application/xyz".
We have looked at disabling [auto-props] both as the user in
~/.subversion/config and as root in /etc/subversion/config by setting:
enable-auto-props = no
And by commenting out the entirety of the [auto-props] section.
However this does not prevent svn:mime-type from being set on all
files.
As this bad behaviour can be so incredibly damaging for this
repository, I have written pre-commit repository hook script to
specifically to block any file that enters the data repository that
has svn:mime-type set (
http://thread.gmane.org/gmane.games.flightgear.devel/77606/focus=77796
). I also block svn:executable in another script. Below I'll give a
worked example that anyone can use and replicate this problem with.
The problem is that Subversion clients setting svn:mime-type
automatically, even when told not to, and the hook script catching
this causes a lot of developer confusion and prevents the use of 'svn
import'.
I was wondering if there was anything that has been missed here? Is
this a real bug? The svn:mime-type property is not needed and is not
desired for any file in this repository. Any help would be
appreciated.
Cheers,
Edward
P. S. Here is the detailed worked example:
Set up:
$ export BASE_DIR="~/flightgear/repo_testing/"
$ export BACKUP_DIR="~/flightgear/backup/"
Initialise a new svn repository:
$ cd $BASE_DIR
$ svnadmin create test1.svn
$ svn co file://$BASE_DIR/test1.svn/ test1
Checked out revision 0.
Copy scripts from FGAddon backup:
$ rsync -av svn.code.sf.net::p/flightgear/fgaddon/hooks $BACKUP_DIR
$ cd $BACKUP_DIR
$ cd hooks/
$ cp -vp pre-commit fgaddon-svn-* $BASE_DIR/test1/hooks/
Download the DaSH aircraft from
https://drive.google.com/file/d/0B-4Q5DonNb8yTi1Cd1J5Um1xMTg/view to
$BASE_DIR.
Testing 'svn import':
$ cd $BASE_DIR
$ tar xvf dash_v1.tar
dash/
dash/avlModel/
dash/dash-fdm.xml
dash/dash-set.xml
dash/Engines/
[snip]
$ svn import dash file://$BASE_DIR/test1.svn/DaSH_svn_import/ -m
"Initial import of the DaSH human powered aircraft.\n\nFor details see
the forum thread at
http://forum.flightgear.org/viewtopic.php?f=4&t=24495 ."
Adding (bin) dash/._thumbnail.jpg
Adding dash/Engines
Adding dash/Engines/dash-engine.xml
Adding dash/Engines/dash-prop.xml
Adding dash/Images
Adding (bin) dash/Images/._DashBoot_logo_Mod.png
Adding (bin) dash/Images/._dash-3view.png
Adding (bin) dash/Images/DashBoot_logo_Mod.png
Adding (bin) dash/Images/dash-3view.png
Adding dash/Models
Adding (bin) dash/Models/._SLE_21b.ac
Adding (bin) dash/Models/._dash-model.xml
Adding dash/Models/SLE_21b.ac
Adding dash/Models/dash-model.xml
Adding dash/avlModel
Adding (bin) dash/avlModel/._dae11.dat
Adding (bin) dash/avlModel/._dash.avl
Adding (bin) dash/avlModel/._plot.pdf
Adding (bin) dash/avlModel/._plot.ps
Adding dash/avlModel/dae11.dat
Adding dash/avlModel/dash.avl
Adding dash/avlModel/dash_stability.txt
Adding (bin) dash/avlModel/plot.pdf
Adding dash/avlModel/plot.ps
Adding dash/dash-fdm.xml
Adding dash/dash-set.xml
Adding (bin) dash/thumbnail.jpg
svn: E165001: Commit blocked by pre-commit hook (exit code 1) with output:
The svn:mime-type property is set on the files
['DaSH_svn_import/._thumbnail.jpg',
'DaSH_svn_import/Images/._DashBoot_logo_Mod.png', 'DaSH_svn_import/Images
/._dash-3view.png', 'DaSH_svn_import/Images/DashBoot_logo_Mod.png',
'DaSH_svn_import/Images/dash-3view.png', 'DaSH_svn_import/Models/._SLE_21b.ac',
'DaSH_svn_import/Models/._dash-model.xml',
'DaSH_svn_import/avlModel/._dae11.dat', 'DaSH_svn_import/avlModel/._dash.avl',
'DaSH_svn_import/avlModel/._plot.pdf', 'DaSH_svn_import/avlModel/._plot.ps',
'DaSH_svn_import/avlModel/plot.pdf', 'DaSH_svn_import/thumbnail.jpg'], aborting
the commit.
The current policy is that the svn:mime-type property is to be avoided in
FGAddon. Before committing, please remove this property by typing 'svn propdel
svn:mime-type file_name' for all affected files. Or to remove it recursively
from all files to be committed, in your aircraft directory type 'svn propdel
svn:mime-type -R'.
To avoid the svn:mime-type property being set by your subversion client, the
subversion configuration should be edited and all [auto-props] entries commented
out. In addition the setting "enable-auto-props = yes" should be deleted from
all subversion config files (in certain svn clients, this will use libmagic to
determine the mime-type and set this for all new files). This includes the user
configuration file at $HOME/.subversion/config or, if this is not present, the
global configuration file at /etc/subversion/config in GNU/Linux and Mac OS X or
%appdata%\subversion\config and %appdata%\roaming\subversion\config in MS
Windows.
Check what was imported:
$ cd $BASE_DIR/test1
$ svn up
$ svn log -v
Nothing went into the repository. Try another way:
$ cd $BASE_DIR/test1
$ svn mkdir DaSH_svn_import
$ svn ci -m "Created a directory for testing a new aircraft addition
using 'svn add'."
$ cd DaSH_svn_import
$ cp -urvp
$ cp -urvp $BASE_DIR/dash/* .
‘/flightgear/repo_testing/mimetype//dash/avlModel’ -> ‘./avlModel’
‘/flightgear/repo_testing/mimetype//dash/avlModel/._dae11.dat’ ->
‘./avlModel/._dae11.dat’
‘/flightgear/repo_testing/mimetype//dash/avlModel/dae11.dat’ ->
‘./avlModel/dae11.dat’
‘/flightgear/repo_testing/mimetype//dash/avlModel/._dash.avl’ ->
‘./avlModel/._dash.avl’
‘/flightgear/repo_testing/mimetype//dash/avlModel/dash.avl’ ->
‘./avlModel/dash.avl’
‘/flightgear/repo_testing/mimetype//dash/avlModel/dash_stability.txt’
-> ‘./avlModel/dash_stability.txt’
[snip]
$ rm -f */._*
$ svn add *
$ svn pl -R
Properties on 'Images/DashBoot_logo_Mod.png':
svn:mime-type
Properties on 'Images/dash-3view.png':
svn:mime-type
Properties on 'avlModel/plot.pdf':
svn:mime-type
Properties on 'thumbnail.jpg':
svn:mime-type
These properties then have to be removed prior to committing with:
$ svn pdel -R svn:mime-type
$ svn ci -m "Import of the DaSH aircraft using 'svn add'."
Re: Bug report: The auto-props setting of svn:mime-type is impossible
to avoid.
Posted by Edward d'Auvergne <tr...@gmail.com>.
On 2 October 2015 at 12:30, Philip Martin <ph...@wandisco.com> wrote:
> "Edward d'Auvergne" <tr...@gmail.com> writes:
>
>> I was wondering if there was anything that has been missed here? Is
>> this a real bug? The svn:mime-type property is not needed and is not
>> desired for any file in this repository. Any help would be
>> appreciated.
>
> You can disable libmagic by setting the environment variable MAGIC
>
> $ svnadmin create repo
> $ svn co file://`pwd`/repo wc
> $ echo "<?xml version='1.0' encoding='UTF-8' ?>" > wc/a.xml
> $ echo "<?xml version='1.0' encoding='UTF-8' ?>" > wc/b.xml
> $ svn add wc/a.xml
> A (bin) wc/a.xml
> $ MAGIC= svn add wc/b.xml
> A wc/b.xml
> $ svn pl -vR wc
> Properties on 'wc/a.xml':
> svn:mime-type
> application/xml
Hi Philip,
Thank you for the response. I have tried out your suggestion with the
MAGIC env variable, however unfortunately on my Mageia Linux 5 system,
this had no effect. The full details, to allow for 100%
reproducibility, are given below.
Cheers,
Edward
P. S. Here are 3 different tests:
Set up:
$ export BASE_DIR="/flightgear/repo_testing/mimetype/"
$ export BACKUP_DIR="/flightgear/backup/"
Initialise a new svn repository:
$ cd $BASE_DIR
$ svnadmin create test2.svn
$ svn co file://$BASE_DIR/test2.svn/ test2
Checked out revision 0.
Copy scripts from FGAddon backup:
$ rsync -av svn.code.sf.net::p/flightgear/fgaddon/hooks $BACKUP_DIR
$ cd $BACKUP_DIR
$ cd hooks/
$ cp -vp pre-commit fgaddon-svn-* $BASE_DIR/test2.svn/hooks/
Download the DaSH aircraft from
https://drive.google.com/file/d/0B-4Q5DonNb8yTi1Cd1J5Um1xMTg/view to
$BASE_DIR.
Testing 'svn import':
$ cd $BASE_DIR
$ tar xvf dash_v1.tar
dash/
dash/avlModel/
dash/dash-fdm.xml
dash/dash-set.xml
dash/Engines/
[snip]
$ MAGIC= svn import dash file://$BASE_DIR/test2.svn/DaSH_svn_import/
-m "Initial import of the DaSH human powered aircraft.\n\nFor details
see the forum thread at
http://forum.flightgear.org/viewtopic.php?f=4&t=24495 ."
Adding (bin) dash/._thumbnail.jpg
Adding dash/Engines
Adding dash/Engines/dash-engine.xml
Adding dash/Engines/dash-prop.xml
Adding dash/Images
Adding (bin) dash/Images/._DashBoot_logo_Mod.png
Adding (bin) dash/Images/._dash-3view.png
Adding (bin) dash/Images/DashBoot_logo_Mod.png
Adding (bin) dash/Images/dash-3view.png
Adding dash/Models
Adding (bin) dash/Models/._SLE_21b.ac
Adding (bin) dash/Models/._dash-model.xml
Adding dash/Models/SLE_21b.ac
Adding dash/Models/dash-model.xml
Adding dash/avlModel
Adding (bin) dash/avlModel/._dae11.dat
Adding (bin) dash/avlModel/._dash.avl
Adding (bin) dash/avlModel/._plot.pdf
Adding (bin) dash/avlModel/._plot.ps
Adding dash/avlModel/dae11.dat
Adding dash/avlModel/dash.avl
Adding dash/avlModel/dash_stability.txt
Adding (bin) dash/avlModel/plot.pdf
Adding dash/avlModel/plot.ps
Adding dash/dash-fdm.xml
Adding dash/dash-set.xml
Adding (bin) dash/thumbnail.jpg
svn: E165001: Commit blocked by pre-commit hook (exit code 1) with output:
The svn:mime-type property is set on the files
['DaSH_svn_import/._thumbnail.jpg',
'DaSH_svn_import/Images/._DashBoot_logo_Mod.png', 'DaSH_svn_import/Images
/._dash-3view.png', 'DaSH_svn_import/Images/DashBoot_logo_Mod.png',
'DaSH_svn_import/Images/dash-3view.png', 'DaSH_svn_import/Models/._SLE_21b.ac',
'DaSH_svn_import/Models/._dash-model.xml',
'DaSH_svn_import/avlModel/._dae11.dat', 'DaSH_svn_import/avlModel/._dash.avl',
'DaSH_svn_import/avlModel/._plot.pdf', 'DaSH_svn_import/avlModel/._plot.ps',
'DaSH_svn_import/avlModel/plot.pdf', 'DaSH_svn_import/thumbnail.jpg'], aborting
the commit.
The current policy is that the svn:mime-type property is to be avoided in
FGAddon. Before committing, please remove this property by typing 'svn propdel
svn:mime-type file_name' for all affected files. Or to remove it recursively
from all files to be committed, in your aircraft directory type 'svn propdel
svn:mime-type -R'.
To avoid the svn:mime-type property being set by your subversion client, the
subversion configuration should be edited and all [auto-props] entries commented
out. In addition the setting "enable-auto-props = yes" should be deleted from
all subversion config files (in certain svn clients, this will use libmagic to
determine the mime-type and set this for all new files). This includes the user
configuration file at $HOME/.subversion/config or, if this is not present, the
global configuration file at /etc/subversion/config in GNU/Linux and Mac OS X or
%appdata%\subversion\config and %appdata%\roaming\subversion\config in MS
Windows.
Test 2:
$ export MAGIC=
$ svn import dash file://$BASE_DIR/test2.svn/DaSH_svn_import/ -m
"Initial import of the DaSH human powered aircraft.\n\nFor details see
the forum thread at
http://forum.flightgear.org/viewtopic.php?f=4&t=24495 ."
Failed!
Test 3:
$ unset MAGIC
$ svn import dash file://$BASE_DIR/test2.svn/DaSH_svn_import/ -m
"Initial import of the DaSH human powered aircraft.\n\nFor details see
the forum thread at
http://forum.flightgear.org/viewtopic.php?f=4&t=24495 ."
Failed!
Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.
Posted by Philip Martin <ph...@wandisco.com>.
"Edward d'Auvergne" <tr...@gmail.com> writes:
> I was wondering if there was anything that has been missed here? Is
> this a real bug? The svn:mime-type property is not needed and is not
> desired for any file in this repository. Any help would be
> appreciated.
You can disable libmagic by setting the environment variable MAGIC
$ svnadmin create repo
$ svn co file://`pwd`/repo wc
$ echo "<?xml version='1.0' encoding='UTF-8' ?>" > wc/a.xml
$ echo "<?xml version='1.0' encoding='UTF-8' ?>" > wc/b.xml
$ svn add wc/a.xml
A (bin) wc/a.xml
$ MAGIC= svn add wc/b.xml
A wc/b.xml
$ svn pl -vR wc
Properties on 'wc/a.xml':
svn:mime-type
application/xml
--
Philip Martin
WANdisco
RE: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: edward.dauvergne@gmail.com
> [mailto:edward.dauvergne@gmail.com] On Behalf Of Edward d'Auvergne
> Sent: dinsdag 6 oktober 2015 12:21
> To: Edward d'Auvergne <tr...@gmail.com>; Greg Stein
> <gs...@gmail.com>; users@subversion.apache.org
> Subject: Re: Bug report: The auto-props setting of svn:mime-type is
> impossible to avoid.
>
> On 4 October 2015 at 13:32, Stefan Sperling <st...@elego.de> wrote:
> > On Sun, Oct 04, 2015 at 12:52:33PM +0200, Edward d'Auvergne wrote:
> >> I would maybe suggest introducing an option for disabling the entire
> >> automatic property subsystem, i.e. it combines both of these, and
> >> overrides them. This is an interesting thought experiment - devise
> >> any name for such a top level automatic property override - and this
> >> should convincingly demonstrate the confusion that the current setting
> >> names induces.
> >
> > I think that would just make the situation worse by adding to
> > the current mess.
> >
> > I agree the documentation needs to be improved. Would you have time
> > to update the documentation and send patches for it?
>
> Ok, I can look into it. Hpwever the instructions for setting the
> MAGIC environmental variable needs some discussion (see below).
>
>
> >> The problem with the svn:mime-type block is that for svn clients <
> >> 1.9, 'svn import' can never work. Specifically because the libmagic
> >> layer overwrites any user decision to disable automatic properties.
> >
> > There's another trick to disable libmagic which works with all
> > client versions. I just tested it with 1.8.
> >
> > Set the MAGIC environment variable to a file which does not exist.
> >
> > env MAGIC=/nonexistent svn import ...
> >
> > This effectively disabled libmagic support.
>
> I've tried all combinations and nothing works (on my Linux system with
> svn 1.8.14). For example:
>
> $ env MAGIC=/nonexistent svn import ...
> $ MAGIC=/nonexistent svn import ...
> [block]
>
> $ env MAGIC=False svn import ...
> $ MAGIC=False svn import ...
> [block]
>
> $ export MAGIC=/nonexistent
> $ svn import ...
> [block]
>
> $ export MAGIC=False
> $ svn import ...
> [block]
>
> $ touch empty_magic
> $ MAGIC=$BASE_DIR/empty_magic svn import ...
> [block]
>
> $ env MAGIC=$BASE_DIR/empty_magic svn import ...
> [block]
>
> $ head -n 1 /usr/share/mime/magic > empty_magic
> $ MAGIC=$BASE_DIR/empty_magic svn import ...
> [block]
>
> Not a single one of these combinations work - svn:mime-type is always
> being set by 'svn import'. Therefore blocking svn:mime-type in a
> pre-commit hook completely kills the option of using 'svn import', and
> there is no way to avoid this. Looking more carefully at what is
> happening:
>
> $ cd $BASE_DIR
> $ cp dash/Images/DashBoot_logo_Mod.png test4
> $ cp dash/dash-set.xml test4
> $ cd test4
> $ svn add DashBoot_logo_Mod.png
> $ svn add dash-set.xml
> $ svn plist -R
> Properties on 'DashBoot_logo_Mod.png':
> svn:mime-type
> $ svn pget svn:mime-type DashBoot_logo_Mod.png
> application/octet-stream
>
> So setting MAGIC is having an effect, but Subversion is falling back,
> probably via a non-magic internal code path, to the default
> svn:mime-type of "application/octet-stream" for PNG and some other
When a file is clearly binary we set application/octet stream, but that is not what we were discussing here. (We check the first 4 Kbyte or something and perform some statistics on that. See the source of svn_io_detect_mimetype2() for details)
If we don't do that we would try to perform text translations and text merges on files that contain '\0' bytes.... That is simply not going to work. That trigger shouldn't apply to xml files though, unless it starts with a huge amount of non ASCII characters.
In some old versions of Subversion the test would say that a file is binary if it only contained a UTF-8 byte order mark, but we filter that one now as some common tools (read: Microsoft Visual Studio) often creates new files with just that content.
There is also an option to set your own mimetype map before we call into magic.
But looking at this... the way to disable the option with 1.9+ is
$ svn add --config-option config:miscellany:enable-magic-file=no file.xml
That doesn't depend on libmagic implementation details such as the environment variable.
With 1.7 and 1.8 you can use the mimetype file with your overrides, as this is processed before libmagic
$ svn add --config-option config:miscellany: mime-types-file=mimemap file.xml
(the mime-types-file option was added to the config in 1.5, but we didn't have --config-option there)
The file 'mimemap' can then contain something like
[[
text/plain .xml
]]
(First token on the line is the mimetype. All further tokens are extensions. # lines allowed)
Bert
Re: Bug report: The auto-props setting of svn:mime-type is
impossible to avoid.
Posted by Stefan Sperling <st...@elego.de>.
On Tue, Oct 06, 2015 at 12:20:47PM +0200, Edward d'Auvergne wrote:
> So setting MAGIC is having an effect, but Subversion is falling back,
> probably via a non-magic internal code path, to the default
> svn:mime-type of "application/octet-stream" for PNG and some other
> files.
If you don't allow svn:mime-type for binary files, you're gonna have
a hard time. Subversion has always been setting this property on binary
files. Please read our documentation about this (at svnbook.org).
If you don't set this property on binary files, your users will
complain that output of 'svn diff' contains binary garbage,
and 'svn merge' will suddenly try to merge PNG files as text.
Make your pre-commit hook smarter. Limit it to *.xml files.
Re: Bug report: The auto-props setting of svn:mime-type is impossible
to avoid.
Posted by Edward d'Auvergne <tr...@gmail.com>.
On 4 October 2015 at 13:32, Stefan Sperling <st...@elego.de> wrote:
> On Sun, Oct 04, 2015 at 12:52:33PM +0200, Edward d'Auvergne wrote:
>> I would maybe suggest introducing an option for disabling the entire
>> automatic property subsystem, i.e. it combines both of these, and
>> overrides them. This is an interesting thought experiment - devise
>> any name for such a top level automatic property override - and this
>> should convincingly demonstrate the confusion that the current setting
>> names induces.
>
> I think that would just make the situation worse by adding to
> the current mess.
>
> I agree the documentation needs to be improved. Would you have time
> to update the documentation and send patches for it?
Ok, I can look into it. Hpwever the instructions for setting the
MAGIC environmental variable needs some discussion (see below).
>> The problem with the svn:mime-type block is that for svn clients <
>> 1.9, 'svn import' can never work. Specifically because the libmagic
>> layer overwrites any user decision to disable automatic properties.
>
> There's another trick to disable libmagic which works with all
> client versions. I just tested it with 1.8.
>
> Set the MAGIC environment variable to a file which does not exist.
>
> env MAGIC=/nonexistent svn import ...
>
> This effectively disabled libmagic support.
I've tried all combinations and nothing works (on my Linux system with
svn 1.8.14). For example:
$ env MAGIC=/nonexistent svn import ...
$ MAGIC=/nonexistent svn import ...
[block]
$ env MAGIC=False svn import ...
$ MAGIC=False svn import ...
[block]
$ export MAGIC=/nonexistent
$ svn import ...
[block]
$ export MAGIC=False
$ svn import ...
[block]
$ touch empty_magic
$ MAGIC=$BASE_DIR/empty_magic svn import ...
[block]
$ env MAGIC=$BASE_DIR/empty_magic svn import ...
[block]
$ head -n 1 /usr/share/mime/magic > empty_magic
$ MAGIC=$BASE_DIR/empty_magic svn import ...
[block]
Not a single one of these combinations work - svn:mime-type is always
being set by 'svn import'. Therefore blocking svn:mime-type in a
pre-commit hook completely kills the option of using 'svn import', and
there is no way to avoid this. Looking more carefully at what is
happening:
$ cd $BASE_DIR
$ cp dash/Images/DashBoot_logo_Mod.png test4
$ cp dash/dash-set.xml test4
$ cd test4
$ svn add DashBoot_logo_Mod.png
$ svn add dash-set.xml
$ svn plist -R
Properties on 'DashBoot_logo_Mod.png':
svn:mime-type
$ svn pget svn:mime-type DashBoot_logo_Mod.png
application/octet-stream
So setting MAGIC is having an effect, but Subversion is falling back,
probably via a non-magic internal code path, to the default
svn:mime-type of "application/octet-stream" for PNG and some other
files. The MAGIC env variable prevents the XML files from not having
svn:mime-type set, but other files are still being set. Hence 'svn
import' is blocked, and there is zero way to avoid the setting of
svn:mime-type. Because of this behaviour, I'm willing to bet that svn
1.9 will also suffer from the inability to disable the automatic
property setting of svn:mime-type (for some reason I can't do this on
a self compiled but not installed svn trunk version, as it is using
the wrong svnlook version and setting SVNLOOK doesn't seem to help).
Regards,
Edward
P. S. Again, as always, here are some detailed instructions to allow
for a 100% reproducibility of all these tests by anyone interested:
Set up:
$ export BASE_DIR="/flightgear/repo_testing/mimetype/"
$ export BACKUP_DIR="/flightgear/backup/"
Initialise a new svn repository:
$ cd $BASE_DIR
$ svnadmin create test4.svn
$ svn co file://$BASE_DIR/test4.svn/ test4
Checked out revision 0.
Copy scripts from FGAddon backup:
$ rsync -av svn.code.sf.net::p/flightgear/fgaddon/hooks $BACKUP_DIR
$ cd $BACKUP_DIR
$ cd fgaddon/hooks/
$ cp -vp pre-commit fgaddon-svn-* $BASE_DIR/test4.svn/hooks/
Download the DaSH aircraft from
https://drive.google.com/file/d/0B-4Q5DonNb8yTi1Cd1J5Um1xMTg/view to
$BASE_DIR.
Testing 'svn import':
$ cd $BASE_DIR
$ tar xvf dash_v1.tar
dash/
dash/avlModel/
dash/dash-fdm.xml
dash/dash-set.xml
dash/Engines/
[snip]
$ head -n 1 /usr/share/mime/magic > empty_magic
$ export MAGIC=$BASE_DIR/empty_magic
$ MAGIC=$BASE_DIR/empty_magic svn import dash
file://$BASE_DIR/test4.svn/DaSH_svn_import/ -m "Initial import of the
DaSH human powered aircraft.\n\nFor details see the forum thread at
http://forum.flightgear.org/viewtopic.php?f=4&t=24495 ."
Adding (bin) dash/._thumbnail.jpg
Adding dash/Engines
Adding dash/Engines/dash-engine.xml
Adding dash/Engines/dash-prop.xml
Adding dash/Images
Adding (bin) dash/Images/._DashBoot_logo_Mod.png
Adding (bin) dash/Images/._dash-3view.png
Adding (bin) dash/Images/DashBoot_logo_Mod.png
Adding (bin) dash/Images/dash-3view.png
Adding dash/Models
Adding (bin) dash/Models/._SLE_21b.ac
Adding (bin) dash/Models/._dash-model.xml
Adding dash/Models/SLE_21b.ac
Adding dash/Models/dash-model.xml
Adding dash/avlModel
Adding (bin) dash/avlModel/._dae11.dat
Adding (bin) dash/avlModel/._dash.avl
Adding (bin) dash/avlModel/._plot.pdf
Adding (bin) dash/avlModel/._plot.ps
Adding dash/avlModel/dae11.dat
Adding dash/avlModel/dash.avl
Adding dash/avlModel/dash_stability.txt
Adding (bin) dash/avlModel/plot.pdf
Adding dash/avlModel/plot.ps
Adding dash/dash-fdm.xml
Adding dash/dash-set.xml
Adding (bin) dash/thumbnail.jpg
svn: E165001: Commit blocked by pre-commit hook (exit code 1) with output:
The svn:mime-type property is set on the files
['DaSH_svn_import/._thumbnail.jpg',
'DaSH_svn_import/Images/._DashBoot_logo_Mod.png', 'DaSH_svn_import/Images
/._dash-3view.png', 'DaSH_svn_import/Images/DashBoot_logo_Mod.png',
'DaSH_svn_import/Images/dash-3view.png', 'DaSH_svn_import/Models/._SLE_21b.ac',
'DaSH_svn_import/Models/._dash-model.xml',
'DaSH_svn_import/avlModel/._dae11.dat', 'DaSH_svn_import/avlModel/._dash.avl',
'DaSH_svn_import/avlModel/._plot.pdf', 'DaSH_svn_import/avlModel/._plot.ps',
'DaSH_svn_import/avlModel/plot.pdf', 'DaSH_svn_import/thumbnail.jpg'], aborting
the commit.
The current policy is that the svn:mime-type property is to be avoided in
FGAddon. Before committing, please remove this property by typing 'svn propdel
svn:mime-type file_name' for all affected files. Or to remove it recursively
from all files to be committed, in your aircraft directory type 'svn propdel
svn:mime-type -R'.
To avoid the svn:mime-type property being set by your subversion client, the
subversion configuration should be edited and all [auto-props] entries commented
out. In addition the setting "enable-auto-props = yes" should be deleted from
all subversion config files (in certain svn clients, this will use libmagic to
determine the mime-type and set this for all new files). This includes the user
configuration file at $HOME/.subversion/config or, if this is not present, the
global configuration file at /etc/subversion/config in GNU/Linux and Mac OS X or
%appdata%\subversion\config and %appdata%\roaming\subversion\config in MS
Windows.
Test 2:
$ cd $BASE_DIR
$ cp dash/dash-set.xml test4
$ cd test4
$ svn add dash-set.xml
$ svn plist -R
Properties on 'dash-set.xml':
svn:mime-type
$ svn pget svn:mime-type dash-set.xml
application/octet-stream
$
RE: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Stefan Sperling [mailto:stsp@elego.de]
> Sent: zondag 4 oktober 2015 13:32
> To: Edward d'Auvergne <tr...@gmail.com>
> Cc: Greg Stein <gs...@gmail.com>; users@subversion.apache.org
> Subject: Re: Bug report: The auto-props setting of svn:mime-type is
> impossible to avoid.
>
> On Sun, Oct 04, 2015 at 12:52:33PM +0200, Edward d'Auvergne wrote:
> > I would maybe suggest introducing an option for disabling the entire
> > automatic property subsystem, i.e. it combines both of these, and
> > overrides them. This is an interesting thought experiment - devise
> > any name for such a top level automatic property override - and this
> > should convincingly demonstrate the confusion that the current setting
> > names induces.
>
> I think that would just make the situation worse by adding to
> the current mess.
>
> I agree the documentation needs to be improved. Would you have time
> to update the documentation and send patches for it?
>
> > The problem with the svn:mime-type block is that for svn clients <
> > 1.9, 'svn import' can never work. Specifically because the libmagic
> > layer overwrites any user decision to disable automatic properties.
>
> There's another trick to disable libmagic which works with all
> client versions. I just tested it with 1.8.
>
> Set the MAGIC environment variable to a file which does not exist.
>
> env MAGIC=/nonexistent svn import ...
>
> This effectively disabled libmagic support.
For Windows I don't think you need any disabling, as our build doesn't even
support building with libmagic there. (Perhaps somebody patched something in
their own build... but I don't think that is really an interesting
scenario).
Note that the Cygwin version is an exception to most of this... It is a
Windows build but tries to do everything the unix way.
Bert
Re: Bug report: The auto-props setting of svn:mime-type is
impossible to avoid.
Posted by Stefan Sperling <st...@elego.de>.
On Sun, Oct 04, 2015 at 12:52:33PM +0200, Edward d'Auvergne wrote:
> I would maybe suggest introducing an option for disabling the entire
> automatic property subsystem, i.e. it combines both of these, and
> overrides them. This is an interesting thought experiment - devise
> any name for such a top level automatic property override - and this
> should convincingly demonstrate the confusion that the current setting
> names induces.
I think that would just make the situation worse by adding to
the current mess.
I agree the documentation needs to be improved. Would you have time
to update the documentation and send patches for it?
> The problem with the svn:mime-type block is that for svn clients <
> 1.9, 'svn import' can never work. Specifically because the libmagic
> layer overwrites any user decision to disable automatic properties.
There's another trick to disable libmagic which works with all
client versions. I just tested it with 1.8.
Set the MAGIC environment variable to a file which does not exist.
env MAGIC=/nonexistent svn import ...
This effectively disabled libmagic support.
Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 4, 2015, at 5:52 AM, Edward d'Auvergne wrote:
> So the following setting disables [auto-props]:
>
> enable-auto-props = no
>
> If a svn:mime-type was defined in the config file, this is now
> disabled. However in this case, [magic-auto-props] then decides to
> tag everything with svn:mime-type. So the desired user action of
> disabling automatic properties is replaced by the second layer of
> magic that automatically adds a property, contrary to the direct
> English translation of "enable-auto-props = no" -> "do not enable
> automatic properties". Note how this design is incredibly confusing
> from the user's perspective, when the desired action is to avoid
> having svn:mime-type set.
>
> This is clearly overcome with the new option:
>
> enable-magic-file = no
>
> However the svn documentation really needs to clarify this
> double-layered automatic property setting and, to override it, that
> you need to set both:
>
> enable-auto-props = no
> enable-magic-file = no
My introduction to Subversion as many years ago so I didn't know it worked that way. Thanks for pointing it out.
You didn't mention whether you tried using the Repository Dictated Configuration feature of Subversion 1.8 and up to push this configuration to your clients. Does that work, and satisfy your needs?
> The problem with the svn:mime-type block is that for svn clients <
> 1.9, 'svn import' can never work. Specifically because the libmagic
> layer overwrites any user decision to disable automatic properties.
> The only solution I can see is to give these instructions in the
> FlightGear documentation:
>
> - To use the 'svn import' command, you must have version 1.9 or
> higher of Subversion installed, and in your ~/.subversion/config file
> set both the "enable-auto-props = no" and "enable-magic-file = no"
> options.
>
> - To avoid the automatic setting of svn properties with 'svn add'
> and the blocking of commits due to the svn:mime-type property being
> set, you must use version 1.9 or higher of Subversion, and in your
> ~/.subversion/config file set both the "enable-auto-props = no" and
> "enable-magic-file = no" options.
Sorry, I must've missed why Subversion 1.9 is required?
Re: Bug report: The auto-props setting of svn:mime-type is impossible
to avoid.
Posted by Edward d'Auvergne <tr...@gmail.com>.
]On 3 October 2015 at 13:19, Stefan Sperling <st...@elego.de> wrote:
> On Sat, Oct 03, 2015 at 11:13:08AM +0200, Edward d'Auvergne wrote:
>> is it really not a bug when:
>>
>> enable-auto-props = yes
>>
>> and:
>>
>> enable-auto-props = no
>>
>> both enable auto-props?
>>
>> Cheers,
>>
>> Edward
>
> I think your best way forward is what Ryan suggested: Ensure svn:mime-type
> for *.xml is set to 'text/plain' by setting svn:autoprops at the root of
> your repository. Then, over time, as your users upgrade to 1.8 clients
> and beyond, this problem will fix itself.
>
> Note that setting mime-type to 'text/plain' will have the same effect
> as if a file which is a "text file" (as guessed by SVN based on
> implementation-defined heuristics) does not have a mime-type set.
>
> When I added the libmagic support (yes, I'm responsible) I had to ensure
> the existing auto-props mechanism kept working. So mime-type detection
> with libmagic happens *after* autoprops and only if autoprops don't set
> a mime-type. That's where this design decision came from.
>
> I'm sorry this implementation doesn't work out of the box for you,
> and if I had known about your situation I might have made different
> choices while implementing this feature years ago. Alas, it's water
> under the bridge now, and you're the first person to complain about
> this ever, and there are mulitple ways you can work around the problem.
>
> So please stop calling behaviour you don't agree with a "bug".
> Call it what it is, a particular behaviour, which might not fit
> every single use case out there.
>
> I hope this works out fine for you in the end, one way or another.
> I'm very happy to see SVN supporting an awesome project like yours!
Hi,
Maybe I should better explain the confusion from the perspective of
the Subversion user, and the current inability to use 'svn import'. I
have been poking around the code and test suite to work out what the
design is, as this is very unclear in the docs. I see two parts:
[auto-props] - User defined svn properties to be automatically
applied with 'svn add' or 'svn import', defined in the user or global
'config' file.
[magic-auto-props] - A fallback for svn:mime-type when
[auto-props] returns nothing. This uses libmagic to set a specific
svn property automatically. I use the [magic-auto-props] label as
this is clearly an automatic property.
Adding to the confusion is that that the libmagic automatic property
layer is described in the "Automatic Property Setting" section of the
svn manual:
http://svnbook.red-bean.com/en/1.8/svn.advanced.props.html
So the following setting disables [auto-props]:
enable-auto-props = no
If a svn:mime-type was defined in the config file, this is now
disabled. However in this case, [magic-auto-props] then decides to
tag everything with svn:mime-type. So the desired user action of
disabling automatic properties is replaced by the second layer of
magic that automatically adds a property, contrary to the direct
English translation of "enable-auto-props = no" -> "do not enable
automatic properties". Note how this design is incredibly confusing
from the user's perspective, when the desired action is to avoid
having svn:mime-type set.
This is clearly overcome with the new option:
enable-magic-file = no
However the svn documentation really needs to clarify this
double-layered automatic property setting and, to override it, that
you need to set both:
enable-auto-props = no
enable-magic-file = no
I would maybe suggest introducing an option for disabling the entire
automatic property subsystem, i.e. it combines both of these, and
overrides them. This is an interesting thought experiment - devise
any name for such a top level automatic property override - and this
should convincingly demonstrate the confusion that the current setting
names induces.
As for the situation for the FlightGear aircraft repository, the file
diversity is so insanely huge that we just cannot ever trust the 3rd
party - libmagic - in the future to not cause other text files to be
treated as binary in the svn repository. The inbuild svn client and
server detection of binary vs. text without svn properties is all that
is desired. This is rock stable and works perfectly 100% of the time.
I have not encountered a situation where a text file should be treated
as binary (i.e. large text files consisting of only 1 or 2 or a few
lines). I.e. only situation were the svn:mime-type property override
of the binary vs. text detection is useful (from the repository
backend and binary vs. text perspective). In most cases, the aircraft
developers will not register the missing diff as a svn property
problem and just commit with everything treated as binary. This was
what has happened historically in a number cases, and is why I have
written a pre-commit hook script to block all svn:mime-type
properties.
The problem with the svn:mime-type block is that for svn clients <
1.9, 'svn import' can never work. Specifically because the libmagic
layer overwrites any user decision to disable automatic properties.
The only solution I can see is to give these instructions in the
FlightGear documentation:
- To use the 'svn import' command, you must have version 1.9 or
higher of Subversion installed, and in your ~/.subversion/config file
set both the "enable-auto-props = no" and "enable-magic-file = no"
options.
- To avoid the automatic setting of svn properties with 'svn add'
and the blocking of commits due to the svn:mime-type property being
set, you must use version 1.9 or higher of Subversion, and in your
~/.subversion/config file set both the "enable-auto-props = no" and
"enable-magic-file = no" options.
I just hope that Linux distributions quickly transition to svn 1.9 clients.
Cheers,
Edward
Re: Bug report: The auto-props setting of svn:mime-type is
impossible to avoid.
Posted by Stefan Sperling <st...@elego.de>.
On Sat, Oct 03, 2015 at 11:13:08AM +0200, Edward d'Auvergne wrote:
> is it really not a bug when:
>
> enable-auto-props = yes
>
> and:
>
> enable-auto-props = no
>
> both enable auto-props?
>
> Cheers,
>
> Edward
I think your best way forward is what Ryan suggested: Ensure svn:mime-type
for *.xml is set to 'text/plain' by setting svn:autoprops at the root of
your repository. Then, over time, as your users upgrade to 1.8 clients
and beyond, this problem will fix itself.
Note that setting mime-type to 'text/plain' will have the same effect
as if a file which is a "text file" (as guessed by SVN based on
implementation-defined heuristics) does not have a mime-type set.
When I added the libmagic support (yes, I'm responsible) I had to ensure
the existing auto-props mechanism kept working. So mime-type detection
with libmagic happens *after* autoprops and only if autoprops don't set
a mime-type. That's where this design decision came from.
I'm sorry this implementation doesn't work out of the box for you,
and if I had known about your situation I might have made different
choices while implementing this feature years ago. Alas, it's water
under the bridge now, and you're the first person to complain about
this ever, and there are mulitple ways you can work around the problem.
So please stop calling behaviour you don't agree with a "bug".
Call it what it is, a particular behaviour, which might not fit
every single use case out there.
I hope this works out fine for you in the end, one way or another.
I'm very happy to see SVN supporting an awesome project like yours!
Re: Bug report: The auto-props setting of svn:mime-type is impossible
to avoid.
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Sat, Oct 3, 2015 at 6:53 AM, Ryan Schmidt
<su...@ryandesign.com> wrote:
>
> On Oct 3, 2015, at 4:13 AM, Edward d'Auvergne wrote:
>
>> is it really not a bug when:
>>
>> enable-auto-props = yes
>>
>> and:
>>
>> enable-auto-props = no
>>
>> both enable auto-props?
>
> Obviously, "enable-auto-props = yes" should enable auto-props and "enable-auto-props = no" should disable them. "no" is the default according to the documentation:
>
> http://svnbook.red-bean.com/en/1.7/svn.advanced.confarea.html#svn.advanced.confarea.opts.config
Whether or not it works on the client settings reliably, it's breaking
these developer's workflow for even accidentally enabled use of
auto-props to happen. I'm afraid their only real option is the
pre-commit hook to prevent poisoning the data, because they don't
necessarily have control of the client configurations. The risk of
that pre-commit hook is that it forbids binary xml's at all, which
does not seem such a big loss in this case.
Mind you, this is the sort of thing that a README at the top of the
"trunk" is useful for, to let users know of workflow problems in
their.
Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 3, 2015, at 4:13 AM, Edward d'Auvergne wrote:
> is it really not a bug when:
>
> enable-auto-props = yes
>
> and:
>
> enable-auto-props = no
>
> both enable auto-props?
Obviously, "enable-auto-props = yes" should enable auto-props and "enable-auto-props = no" should disable them. "no" is the default according to the documentation:
http://svnbook.red-bean.com/en/1.7/svn.advanced.confarea.html#svn.advanced.confarea.opts.config
Re: Bug report: The auto-props setting of svn:mime-type is impossible
to avoid.
Posted by Edward d'Auvergne <tr...@gmail.com>.
On 3 October 2015 at 11:01, Greg Stein <gs...@gmail.com> wrote:
> I do understand this, but this is not a "bug". It is how Subversion is
> designed to work. For strange edge cases, the xml people want content to be
> labeled application/xml rather than a text subtype. ... I *do* understand
> that this negatively impacts your development workflow.
>
> Note that you can always remove the property after the "svn add", but before
> you "svn commit". Others have commented on ways to avoid the automatic
> placement of the property, though that will be increasingly difficult in
> heterogeneous environments with different svn clients. Subversion just
> "helpfully" provides a property, but you *don't* have to commit it that way.
> And committing it that way, and removing it later, has no effect on your
> actual content.
>
> Your hook is a good way to prevent *new* files with that svn:mime-type on
> them. Your repository has tens of thousands now, sure, but honestly... the
> concern is about new files, and their rate of addition. It seems once
> somebody sees a "cannot merge a binary file" error, then they can just go
> delete the property and continue their work.
>
> If one of the knobs that Stefan suggest is not working properly, then we can
> track that down and fix it.
Hi,
The "svn add", "svn propdel svn:mime-type -R", "svn ci" workflow is
what we currently must use. Though it would be good if we could cut
out the middle "svn propdel" step, as the propdel command can be
difficult to find or missing in certain svn GUI clients (I've heard
this about TortoiseSVN). The big problem is with the "svn import"
command, which essentially combines "svn add *" and "svn ci" into one
without having a local checked out copy of the repository. In this
case, inserting "svn propdel" in the middle is not possible. Anyway,
is it really not a bug when:
enable-auto-props = yes
and:
enable-auto-props = no
both enable auto-props?
Cheers,
Edward
Re: Bug report: The auto-props setting of svn:mime-type is impossible
to avoid.
Posted by Greg Stein <gs...@gmail.com>.
I do understand this, but this is not a "bug". It is how Subversion is
designed to work. For strange edge cases, the xml people want content to be
labeled application/xml rather than a text subtype. ... I *do* understand
that this negatively impacts your development workflow.
Note that you can always remove the property after the "svn add", but
before you "svn commit". Others have commented on ways to avoid the
automatic placement of the property, though that will be increasingly
difficult in heterogeneous environments with different svn clients.
Subversion just "helpfully" provides a property, but you *don't* have to
commit it that way. And committing it that way, and removing it later, has
no effect on your actual content.
Your hook is a good way to prevent *new* files with that svn:mime-type on
them. Your repository has tens of thousands now, sure, but honestly... the
concern is about new files, and their rate of addition. It seems once
somebody sees a "cannot merge a binary file" error, then they can just go
delete the property and continue their work.
If one of the knobs that Stefan suggest is not working properly, then we
can track that down and fix it.
Cheers,
-g
On Sat, Oct 3, 2015 at 3:48 AM, Edward d'Auvergne <tr...@gmail.com>
wrote:
> On 3 October 2015 at 01:05, Greg Stein <gs...@gmail.com> wrote:
> > On Fri, Oct 02, 2015 at 10:06:26AM +0200, Edward d'Auvergne wrote:
> >>...
> >> As this bad behaviour can be so incredibly damaging for this
> >> repository,
> >
> > Note: the files themselves are not "damaged" -- Subversion will never
> > alter the contents of a file when it is first imported/added. It may
> > make a file look different in the working copy, according to
> > svn:eol-style. But the svn:mime-type is just a property associated
> > with the file. Its only real use is when the file is served via HTTP
> > to a web browser.
>
> Hi Greg,
>
> The damage is to the repository backend and to the development process
> itself. Treating text files as binary has a number of significant
> consequences for development. For example:
>
> - 'svn up' can no longer perform merges (G).
> - 'svn merge' or the svnmerge.py script can no longer function to
> merge changes to the same file.
> - A one character change to a massive text file results in a diff and
> db/revs backend file the size of the whole file, and not just a few
> lines for a proper diff.
> - Changes can no longer be reviewed (
> https://sourceforge.net/p/flightgear/mailman/flightgear-fgaddon-commitlogs/
> ).
>
> This is a severe bug. Here is an interesting sign as well:
>
> $ svn propset svn:mime-type "application/xml" dash-set.xml
> svn: warning: 'application/xml' is a binary mime-type but file
> '/flightgear/repo_testing/mimetype/test1/DaSH_svn_import/dash-set.xml'
> looks like text; diff, merge, blame, and other operations will stop
> working on this file
> property 'svn:mime-type' set on 'dash-set.xml'
> $
>
> Unfortunately 'svn add' and 'svn import' set this binary mime-type
> without warning, and when explicitly told not to.
>
>
> > If a file is committed to the repository with an svn:mime-type, why
> > don't you just delete/correct the property in a following commit?
>
> Does this answer your question ;)
>
> [edward@localhost flightgear-fgaddon]$ svn info
> Path: .
> Working Copy Root Path: /flightgear/flightgear-fgaddon
> URL: svn+ssh://edauvergne@svn.code.sf.net/p/flightgear/fgaddon/trunk
> Relative URL: ^/trunk
> Repository Root: svn+ssh://edauvergne@svn.code.sf.net/p/flightgear/fgaddon
> Repository UUID: 9eb9bc84-db9d-4e05-893b-26dd41ac94a6
> [snip]
> [edward@localhost flightgear-fgaddon]$ find -type f | grep -v .svn | wc -l
> 69458
> [edward@localhost flightgear-fgaddon]$ find -name "*.xml" | wc -l
> 27262
> [edward@localhost flightgear-fgaddon]$ find -name "*.ac" | wc -l
> 12502
> [edward@localhost flightgear-fgaddon]$ find -name "*.nas" | wc -l
> 2840
>
> Manually monitoring this huge beast is out of the question.
>
> Cheers,
>
> Edward
>
Re: Bug report: The auto-props setting of svn:mime-type is impossible
to avoid.
Posted by Edward d'Auvergne <tr...@gmail.com>.
On 3 October 2015 at 01:05, Greg Stein <gs...@gmail.com> wrote:
> On Fri, Oct 02, 2015 at 10:06:26AM +0200, Edward d'Auvergne wrote:
>>...
>> As this bad behaviour can be so incredibly damaging for this
>> repository,
>
> Note: the files themselves are not "damaged" -- Subversion will never
> alter the contents of a file when it is first imported/added. It may
> make a file look different in the working copy, according to
> svn:eol-style. But the svn:mime-type is just a property associated
> with the file. Its only real use is when the file is served via HTTP
> to a web browser.
Hi Greg,
The damage is to the repository backend and to the development process
itself. Treating text files as binary has a number of significant
consequences for development. For example:
- 'svn up' can no longer perform merges (G).
- 'svn merge' or the svnmerge.py script can no longer function to
merge changes to the same file.
- A one character change to a massive text file results in a diff and
db/revs backend file the size of the whole file, and not just a few
lines for a proper diff.
- Changes can no longer be reviewed (
https://sourceforge.net/p/flightgear/mailman/flightgear-fgaddon-commitlogs/
).
This is a severe bug. Here is an interesting sign as well:
$ svn propset svn:mime-type "application/xml" dash-set.xml
svn: warning: 'application/xml' is a binary mime-type but file
'/flightgear/repo_testing/mimetype/test1/DaSH_svn_import/dash-set.xml'
looks like text; diff, merge, blame, and other operations will stop
working on this file
property 'svn:mime-type' set on 'dash-set.xml'
$
Unfortunately 'svn add' and 'svn import' set this binary mime-type
without warning, and when explicitly told not to.
> If a file is committed to the repository with an svn:mime-type, why
> don't you just delete/correct the property in a following commit?
Does this answer your question ;)
[edward@localhost flightgear-fgaddon]$ svn info
Path: .
Working Copy Root Path: /flightgear/flightgear-fgaddon
URL: svn+ssh://edauvergne@svn.code.sf.net/p/flightgear/fgaddon/trunk
Relative URL: ^/trunk
Repository Root: svn+ssh://edauvergne@svn.code.sf.net/p/flightgear/fgaddon
Repository UUID: 9eb9bc84-db9d-4e05-893b-26dd41ac94a6
[snip]
[edward@localhost flightgear-fgaddon]$ find -type f | grep -v .svn | wc -l
69458
[edward@localhost flightgear-fgaddon]$ find -name "*.xml" | wc -l
27262
[edward@localhost flightgear-fgaddon]$ find -name "*.ac" | wc -l
12502
[edward@localhost flightgear-fgaddon]$ find -name "*.nas" | wc -l
2840
Manually monitoring this huge beast is out of the question.
Cheers,
Edward
Re: Bug report: The auto-props setting of svn:mime-type is
impossible to avoid.
Posted by Greg Stein <gs...@gmail.com>.
On Fri, Oct 02, 2015 at 10:06:26AM +0200, Edward d'Auvergne wrote:
>...
> As this bad behaviour can be so incredibly damaging for this
> repository,
Note: the files themselves are not "damaged" -- Subversion will never
alter the contents of a file when it is first imported/added. It may
make a file look different in the working copy, according to
svn:eol-style. But the svn:mime-type is just a property associated
with the file. Its only real use is when the file is served via HTTP
to a web browser.
If a file is committed to the repository with an svn:mime-type, why
don't you just delete/correct the property in a following commit?
Cheers,
-g
Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Oct 3, 2015, at 3:25 AM, Edward d'Auvergne wrote:
> On 2 October 2015 at 11:33, Stefan Sperling wrote:
>
>> - configure autoprops for *.xml in ~/.subversion/config to set
>> the svn:mime-type property to 'text/plain'.
>> Autoprops always override automatic detection with libmagic.
>
> From the perspective of Subversion users, libmagic is a 3rd party. We
> cannot trust them to not change the magic file again to something that
> treats text files in svn as binary files. This FG aircraft repository
> is one of the biggest version controlled repositories in the open
> source world, by far. It has an incredible diversity of text files
> with different extensions, and we would like these to forever be
> treated as text. Due to the bad behaviour of libmagic, banning
> svn:mime-type from the FG repository is the only practical option.
So configure autoprops correctly for those various extensions.
You can even use repository-dictated configuration to make those autoprops definitions automatic for all your diverse users.
http://blogs.collab.net/subversion/the-road-to-repository-dictated-configuration-day-2-autoprops#.Vg-ZjbRobfA
On Oct 3, 2015, at 3:48 AM, Edward d'Auvergne wrote:
> The damage is to the repository backend and to the development process
> itself. Treating text files as binary has a number of significant
> consequences for development. For example:
>
> - 'svn up' can no longer perform merges (G).
> - 'svn merge' or the svnmerge.py script can no longer function to
> merge changes to the same file.
> - A one character change to a massive text file results in a diff and
> db/revs backend file the size of the whole file, and not just a few
> lines for a proper diff.
> - Changes can no longer be reviewed (
> https://sourceforge.net/p/flightgear/mailman/flightgear-fgaddon-commitlogs/
> ).
>
> This is a severe bug.
Not a bug: intended behavior for binary files. The "bug" is that Subversion, via libmagic, has decided a file is binary, when you consider it to be text.
The XML specification defines a number of correct MIME types for XML files, including some beginning with "text/xml" (intended for human-readable XML files) and "application/xml" (intended for XML files not meant for humans to read). libmagic can identify whether a file is XML, but it cannot know whether the file is meant for human consumption, so it picks on of these. The person committing the file can correct this prior to committing if it is inaccurate. Or if you don't want the property set at all, use autoprops.
> Here is an interesting sign as well:
>
> $ svn propset svn:mime-type "application/xml" dash-set.xml
> svn: warning: 'application/xml' is a binary mime-type but file
> '/flightgear/repo_testing/mimetype/test1/DaSH_svn_import/dash-set.xml'
> looks like text; diff, merge, blame, and other operations will stop
> working on this file
> property 'svn:mime-type' set on 'dash-set.xml'
> $
>
> Unfortunately 'svn add' and 'svn import' set this binary mime-type
> without warning, and when explicitly told not to.
Subversion doesn't do things without being told to do so. There are defaults for how MIME types are set on files, which you can override.
> Manually monitoring this huge beast is out of the question.
Of course you wouldn't monitor it manually; you would monitor it via a pre-commit hook script that prevents commits that don't conform to your requirements.
Re: Bug report: The auto-props setting of svn:mime-type is impossible
to avoid.
Posted by Edward d'Auvergne <tr...@gmail.com>.
On 2 October 2015 at 11:33, Stefan Sperling <st...@elego.de> wrote:
> On Fri, Oct 02, 2015 at 10:06:26AM +0200, Edward d'Auvergne wrote:
>> Hi all,
>>
>> I was wondering if this should be considered a bug. At the FlightGear
>> project we have a 6 GB data svn repository for aircraft (
>> https://sourceforge.net/projects/flightgear/ ,
>> https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/ ). A large
>> quantity of the files, almost 30,000 in number, are XML text files.
>>
>> The problem is that modern Subversion clients appear to all be using
>> libmagic to determine the file type, to then set the svn:mime-type
>> property. However this is rather problematic:
>>
>> $ file -ib Aircraft/747-400/747-400-set.xml
>> application/xml; charset=us-ascii
>>
>> Libmagic detects all these XML files as "application/xml". Hence they
>> are treated as binary files! There are many other text files with
>> extensions such as *.ac, *.nas, etc. present in the repository that in
>> the future might be detected by libmagic as "application/xyz".
>>
>> We have looked at disabling [auto-props] both as the user in
>> ~/.subversion/config and as root in /etc/subversion/config by setting:
>>
>> enable-auto-props = no
>>
>> And by commenting out the entirety of the [auto-props] section.
>> However this does not prevent svn:mime-type from being set on all
>> files.
>>
>> As this bad behaviour can be so incredibly damaging for this
>> repository, I have written pre-commit repository hook script to
>> specifically to block any file that enters the data repository that
>> has svn:mime-type set (
>> http://thread.gmane.org/gmane.games.flightgear.devel/77606/focus=77796
>> ). I also block svn:executable in another script. Below I'll give a
>> worked example that anyone can use and replicate this problem with.
>> The problem is that Subversion clients setting svn:mime-type
>> automatically, even when told not to, and the hook script catching
>> this causes a lot of developer confusion and prevents the use of 'svn
>> import'.
>>
>> I was wondering if there was anything that has been missed here? Is
>> this a real bug? The svn:mime-type property is not needed and is not
>> desired for any file in this repository. Any help would be
>> appreciated.
>>
>> Cheers,
>>
>> Edward
>
> Hi Edward,
>
> the pre-commit hook script you wrote blocks commits using an
> undesired client configuration, so that's a good first step.
Hi Stefan,
Thanks for the detailed response!
> For the second step, there are several ways of tweaking the client-side
> behaviour. I hope at least one of these alternatives will help you.
>
> - compile SVN without libmagic support
Unfortunately this is difficult. The FlightGear (FG) aircraft
collection contains about 500 aircraft, mostly by different and
multiple authors. So we have GNU/Linux, Mac OS X, and MS Windows
users/developers. And we have users of the command line svn clients
and GUI tools. Standard svn, KDESvn, SlikSVN, TortoiseSVN, the list
goes on... The huge diversity limits the practical options.
> - write you own magic file and use the MAGIC environment variable
> to point svn at it, setting the desired mime-type for XML files
This might work. I'll soon try out Philip's suggestion and see how
that works. I'm not sure how this will work with the GUI clients.
> - configure autoprops for *.xml in ~/.subversion/config to set
> the svn:mime-type property to 'text/plain'.
> Autoprops always override automatic detection with libmagic.
>From the perspective of Subversion users, libmagic is a 3rd party. We
cannot trust them to not change the magic file again to something that
treats text files in svn as binary files. This FG aircraft repository
is one of the biggest version controlled repositories in the open
source world, by far. It has an incredible diversity of text files
with different extensions, and we would like these to forever be
treated as text. Due to the bad behaviour of libmagic, banning
svn:mime-type from the FG repository is the only practical option.
> - set 'enable-magic-file = no' in ~/.subversion/config
> This option exists only as of Subversion 1.9, I believe.
> Consider the following trick to view a default configuration in the
> file /tmp/svncfg/config, which lists all available options in your
> current version of svn, with comments explaining their purpose:
> svn help --config-dir /tmp/svncfg
> less /tmp/svncfg/config
This sounds like an option, but not many of the FG aircraft developers
will have access to this. For example I'm using Mageia Linux 5, which
was only released a few months ago, that this ships with svn, version
1.8.14 (r1692801). However the option:
enable-auto-props = no
Should be equivalent of:
disable-auto-props = yes
I do not know why when telling the svn client to not use auto-props,
the svn client ignores you and uses auto-props anyway. To me this
looks like a bug (I hope the Subversion developers read this list).
Cheers,
Edward
Re: Bug report: The auto-props setting of svn:mime-type is
impossible to avoid.
Posted by Stefan Sperling <st...@elego.de>.
On Fri, Oct 02, 2015 at 10:06:26AM +0200, Edward d'Auvergne wrote:
> Hi all,
>
> I was wondering if this should be considered a bug. At the FlightGear
> project we have a 6 GB data svn repository for aircraft (
> https://sourceforge.net/projects/flightgear/ ,
> https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/ ). A large
> quantity of the files, almost 30,000 in number, are XML text files.
>
> The problem is that modern Subversion clients appear to all be using
> libmagic to determine the file type, to then set the svn:mime-type
> property. However this is rather problematic:
>
> $ file -ib Aircraft/747-400/747-400-set.xml
> application/xml; charset=us-ascii
>
> Libmagic detects all these XML files as "application/xml". Hence they
> are treated as binary files! There are many other text files with
> extensions such as *.ac, *.nas, etc. present in the repository that in
> the future might be detected by libmagic as "application/xyz".
>
> We have looked at disabling [auto-props] both as the user in
> ~/.subversion/config and as root in /etc/subversion/config by setting:
>
> enable-auto-props = no
>
> And by commenting out the entirety of the [auto-props] section.
> However this does not prevent svn:mime-type from being set on all
> files.
>
> As this bad behaviour can be so incredibly damaging for this
> repository, I have written pre-commit repository hook script to
> specifically to block any file that enters the data repository that
> has svn:mime-type set (
> http://thread.gmane.org/gmane.games.flightgear.devel/77606/focus=77796
> ). I also block svn:executable in another script. Below I'll give a
> worked example that anyone can use and replicate this problem with.
> The problem is that Subversion clients setting svn:mime-type
> automatically, even when told not to, and the hook script catching
> this causes a lot of developer confusion and prevents the use of 'svn
> import'.
>
> I was wondering if there was anything that has been missed here? Is
> this a real bug? The svn:mime-type property is not needed and is not
> desired for any file in this repository. Any help would be
> appreciated.
>
> Cheers,
>
> Edward
Hi Edward,
the pre-commit hook script you wrote blocks commits using an
undesired client configuration, so that's a good first step.
For the second step, there are several ways of tweaking the client-side
behaviour. I hope at least one of these alternatives will help you.
- compile SVN without libmagic support
- write you own magic file and use the MAGIC environment variable
to point svn at it, setting the desired mime-type for XML files
- configure autoprops for *.xml in ~/.subversion/config to set
the svn:mime-type property to 'text/plain'.
Autoprops always override automatic detection with libmagic.
- set 'enable-magic-file = no' in ~/.subversion/config
This option exists only as of Subversion 1.9, I believe.
Consider the following trick to view a default configuration in the
file /tmp/svncfg/config, which lists all available options in your
current version of svn, with comments explaining their purpose:
svn help --config-dir /tmp/svncfg
less /tmp/svncfg/config
RE: Bug report: The auto-props setting of svn:mime-type is
impossibleto avoid.
Posted by be...@qqmail.nl.
The problem with xml in general is that is used in many scenarios. In some usages it can be handled as text, while in other usages it is really binary data where every whitespace change could be a breaking change. Xml is just a container format that makes many usages possible.
We can't make that decision for every xml user in Subversion by just flipping the behavior. (We discussed this many times before)
The best fix would be that magic really detects your xml form from the other xml forms, like it already detects many other forms. In many cases it can do this based on a xmlns definition or other hint.
If your format is really text like application/* is a mismatch for you, but when you are storing signed information -with a signature at the end- in xml you really don't want to see automatic merges. I think the original decision to register an application/* type by the xml inventers was a safe decision.
Your files are just not 'generic xml', and should have a more specific type.
Bert
From: Ivan Zhakov
Sent: zondag 4 oktober 2015 11:35
To: Edward d'Auvergne
Cc: users@subversion.apache.org
Subject: Re: Bug report: The auto-props setting of svn:mime-type is impossibleto avoid.
On 2 October 2015 at 11:06, Edward d'Auvergne <tr...@gmail.com> wrote:
> Hi all,
>
> I was wondering if this should be considered a bug. At the FlightGear
> project we have a 6 GB data svn repository for aircraft (
> https://sourceforge.net/projects/flightgear/ ,
> https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/ ). A large
> quantity of the files, almost 30,000 in number, are XML text files.
>
> The problem is that modern Subversion clients appear to all be using
> libmagic to determine the file type, to then set the svn:mime-type
> property. However this is rather problematic:
>
> $ file -ib Aircraft/747-400/747-400-set.xml
> application/xml; charset=us-ascii
>
> Libmagic detects all these XML files as "application/xml". Hence they
> are treated as binary files! There are many other text files with
> extensions such as *.ac, *.nas, etc. present in the repository that in
> the future might be detected by libmagic as "application/xyz".
>
Easy fix would be change svn_mime_type_is_binary() to consider
'application/xml' as text file. We can also extend for other specific
application/* mimetypes that are text files. We already have special
handling for 'image/x-xbitmap' to consider it as text file.
Does it make sense?
--
Ivan Zhakov
RE: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Stefan Sperling [mailto:stsp@elego.de]
> Subject: Re: Bug report: The auto-props setting of svn:mime-type is
> impossible to avoid.
>
> > This whole discussion -in its many iterations- is one of the reasons why
I
> > never looked at enabling this feature on Windows.
>
> And nobody has been asking for it to be enabled?
I don't remember anybody asking...
I doubt many users explicitly enable it on other platforms either, but we do
an autodetect there, which usually just works.
Bert
Re: Bug report: The auto-props setting of svn:mime-type is
impossible to avoid.
Posted by Stefan Sperling <st...@elego.de>.
On Mon, Oct 05, 2015 at 12:06:59AM +0200, Bert Huijben wrote:
> I'm not sure if I would call it a security problem when a user adds a file of
> their choosing to Subversion though :-)
Yes, typical SVN use cases are of no concern.
One case I could imagine where this might matter is some automated
service which records user-provided files in SVN.
Another case is where people use SVN to version home directories
which include a Downloads folder with files such as images from
the internet.
That's all a bit far fetched, though I know of one person using SVN
to keep /home under version control.
> This whole discussion -in its many iterations- is one of the reasons why I
> never looked at enabling this feature on Windows.
And nobody has been asking for it to be enabled?
RE: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Stefan Sperling [mailto:stsp@elego.de]
> Sent: zondag 4 oktober 2015 22:01
> To: Branko Čibej <br...@apache.org>
> Cc: users@subversion.apache.org
> Subject: Re: Bug report: The auto-props setting of svn:mime-type is
> impossible to avoid.
>
> On Sun, Oct 04, 2015 at 09:16:04PM +0200, Branko Čibej wrote:
> > On the other hand, I am surprised that the logic that uses libmagic
> > isn't turned off with 'enable-auto-props=no'. After all, using libmagic
> > is just a convenient extension to the definitions in the [auto-props]
> > section.
>
> Recall that the idea was to make setting svn:mime-type convenient.
>
> Before we added this feature, people had to fiddle with their client
> config, know what MIME-types are about, what useful values to set
> them to, and keep their config consistent across all systems they
> used since libmagic support pre-dates the svn:autoprops feature.
> In a sane universe, almost nobody would ever bother setting it up that way.
>
> I am not opposed to the idea, though. But not because of the fairly
> minor configurability issues raised in this thread. Rather, because
> the more I learn about how libmagic actually works, the more potentially
> dangerous, from the security point of view, it seems to be.
> The libmagic parser could potentially cause security problems whenever
> people are adding untrusted files to SVN. Which is a good reason to make
> this feature opt-in.
I would support changing Subversion to -by default- disable this feature, unless it is enabled in the configuration (or some flag passed to 'svn add' in 1.10+).
I'm not sure if I would call it a security problem when a user adds a file of their choosing to Subversion though :-)
This whole discussion -in its many iterations- is one of the reasons why I never looked at enabling this feature on Windows.
Bert
Re: Bug report: The auto-props setting of svn:mime-type is
impossible to avoid.
Posted by Stefan Sperling <st...@elego.de>.
On Sun, Oct 04, 2015 at 09:16:04PM +0200, Branko Čibej wrote:
> On the other hand, I am surprised that the logic that uses libmagic
> isn't turned off with 'enable-auto-props=no'. After all, using libmagic
> is just a convenient extension to the definitions in the [auto-props]
> section.
Recall that the idea was to make setting svn:mime-type convenient.
Before we added this feature, people had to fiddle with their client
config, know what MIME-types are about, what useful values to set
them to, and keep their config consistent across all systems they
used since libmagic support pre-dates the svn:autoprops feature.
In a sane universe, almost nobody would ever bother setting it up that way.
I am not opposed to the idea, though. But not because of the fairly
minor configurability issues raised in this thread. Rather, because
the more I learn about how libmagic actually works, the more potentially
dangerous, from the security point of view, it seems to be.
The libmagic parser could potentially cause security problems whenever
poeple are adding untrusted files to SVN. Which is a good reason to make
this feature opt-in.
Re: Bug report: The auto-props setting of svn:mime-type is impossible
to avoid.
Posted by Branko Čibej <br...@apache.org>.
On 04.10.2015 11:35, Ivan Zhakov wrote:
> On 2 October 2015 at 11:06, Edward d'Auvergne <tr...@gmail.com> wrote:
>> Hi all,
>>
>> I was wondering if this should be considered a bug. At the FlightGear
>> project we have a 6 GB data svn repository for aircraft (
>> https://sourceforge.net/projects/flightgear/ ,
>> https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/ ). A large
>> quantity of the files, almost 30,000 in number, are XML text files.
>>
>> The problem is that modern Subversion clients appear to all be using
>> libmagic to determine the file type, to then set the svn:mime-type
>> property. However this is rather problematic:
>>
>> $ file -ib Aircraft/747-400/747-400-set.xml
>> application/xml; charset=us-ascii
>>
>> Libmagic detects all these XML files as "application/xml". Hence they
>> are treated as binary files! There are many other text files with
>> extensions such as *.ac, *.nas, etc. present in the repository that in
>> the future might be detected by libmagic as "application/xyz".
>>
> Easy fix would be change svn_mime_type_is_binary() to consider
> 'application/xml' as text file. We can also extend for other specific
> application/* mimetypes that are text files. We already have special
> handling for 'image/x-xbitmap' to consider it as text file.
>
> Does it make sense?
Please let's not have that discussion again. A quick search of the
archives will show that we've talked this idea to death already.
To summarize: To Subversion, "text" implies that you can do textual
diff/merge with the contents. With XML, that may sometimes work, but
usually won't. And the problem doesn't stop with 'application/xml'
because there are literally hundreds of media types for documents that
use XML.
On the other hand, I am surprised that the logic that uses libmagic
isn't turned off with 'enable-auto-props=no'. After all, using libmagic
is just a convenient extension to the definitions in the [auto-props]
section.
-- Brane
Re: Bug report: The auto-props setting of svn:mime-type is impossible
to avoid.
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 2 October 2015 at 11:06, Edward d'Auvergne <tr...@gmail.com> wrote:
> Hi all,
>
> I was wondering if this should be considered a bug. At the FlightGear
> project we have a 6 GB data svn repository for aircraft (
> https://sourceforge.net/projects/flightgear/ ,
> https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/ ). A large
> quantity of the files, almost 30,000 in number, are XML text files.
>
> The problem is that modern Subversion clients appear to all be using
> libmagic to determine the file type, to then set the svn:mime-type
> property. However this is rather problematic:
>
> $ file -ib Aircraft/747-400/747-400-set.xml
> application/xml; charset=us-ascii
>
> Libmagic detects all these XML files as "application/xml". Hence they
> are treated as binary files! There are many other text files with
> extensions such as *.ac, *.nas, etc. present in the repository that in
> the future might be detected by libmagic as "application/xyz".
>
Easy fix would be change svn_mime_type_is_binary() to consider
'application/xml' as text file. We can also extend for other specific
application/* mimetypes that are text files. We already have special
handling for 'image/x-xbitmap' to consider it as text file.
Does it make sense?
--
Ivan Zhakov