You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Stephen White <sw...@corefiling.com> on 2015/03/27 14:02:40 UTC
Subversion checkout issue with certain short names on Windows
If you pass a Windows short-name as the destination to "svn co" then it
will normally work correctly, unless the short name is actually longer
than the "long name" in which case the string handling seems to go awry.
First a working example:
> mkdir ALongName.impl
> dir /x
...
27/03/2015 12:50 <DIR> ALONGN~1.IMP ALongName.impl
...
>svn co --depth=files http://svn.apache.org/repos/asf/subversion/trunk
ALONGN~1.IMP
A ALongName.impl\NOTICE
A ALongName.impl\get-deps.sh
A ALongName.impl\Makefile.in
A ALongName.impl\LICENSE
A ALongName.impl\build.conf
A ALongName.impl\win-tests.py
A ALongName.impl\COMMITTERS
A ALongName.impl\README
A ALongName.impl\BUGS
A ALongName.impl\configure.ac
A ALongName.impl\TODO
A ALongName.impl\.ycm_extra_conf.py
A ALongName.impl\INSTALL
A ALongName.impl\CHANGES
A ALongName.impl\autogen.sh
A ALongName.impl\gen-make.py
A ALongName.impl\aclocal.m4
U ALongName.impl
Checked out revision 1669571.
That's behaving as expected. SVN checks out into the ALongName.impl
directory, as both the long name and the short name should be
synonymous. It outputs the long name, not the short name given on the
command-line, which was slightly surprising but not a problem.
Presumably the result of some canonicalisation code somewhere.
Now lets try the same sequence of operations with a different directory
name:
> mkdir api.impl
> dir /x
...
27/03/2015 12:52 <DIR> API~1.IMP api.impl
...
>svn co --depth=files http://svn.apache.org/repos/asf/subversion/trunk
API~1.IMP
A api.implP\NOTICE
A api.implP\get-deps.sh
A api.implP\Makefile.in
A api.implP\LICENSE
A api.implP\build.conf
A api.implP\win-tests.py
A api.implP\COMMITTERS
A api.implP\README
A api.implP\BUGS
A api.implP\configure.ac
A api.implP\TODO
A api.implP\.ycm_extra_conf.py
A api.implP\INSTALL
A api.implP\CHANGES
A api.implP\autogen.sh
A api.implP\gen-make.py
A api.implP\aclocal.m4
U api.implP
Checked out revision 1669571.
Here the behaviour is incorrect, it's created a new folder "api.implP"
and checkout out to that. The spurious "P" character is the last
character of the short name, so it looks like a buffer is being
overwritten incorrectly in situations where the long name turns out to
be shorter than the short name. That happens in this case because,
although "api.impl" is short, the "file extension" is more than 3
characters so this isn't a valid Windows 8.3 (short) name.
The use of the --depth=files option isn't signficant here, it just makes
the reproduction steps quicker to run.
I think this is a pretty clear example of a bug, does anyone else agree?
Thanks,
--
Stephen White, CoreFiling Limited
http://www.corefiling.com
Phone: +44-1865-203192
Re: Subversion checkout issue with certain short names on Windows
Posted by Branko Čibej <br...@wandisco.com>.
On 15.06.2015 14:16, Bert Huijben wrote:
> For the mailinglist archive:
>
>> -----Original Message-----
>> From: Bert Huijben [mailto:bert@qqmail.nl]
>> Sent: zondag 29 maart 2015 17:50
>> To: 'Stephen White'; dev@subversion.apache.org
>> Subject: RE: Subversion checkout issue with certain short names on Windows
>>
>> It looks like a bug, but not one that we can fix in Subversion. We use the apr
>> library for all our path calculations. I'm guessing that this problem occurs in the
>> APR truename implementation, as after that the code to convert back from
>> absolute paths back to shortnames would be affected
>>
>> What do you see if you just type 'svn info API~1.IMP' for that directory. (I don't
>> think this is in any way specific to checkout)
>>
>>
>> The obvious workaround is of course to just use the full name instead of the
>> short name.
>>
>>
>> Interesting followup questions would be what version of Subversion and what
>> version of apr you are using. Perhaps this issue was already fixed.
> I've identified the bug in apr and posted a patch to the apr development list to fix the problem. See the ' [Patch] apr_file_path_merge returns incorrect truepath on certain 'short paths' on Windows' thread.
>
> I hope they'll fix the problem in the next release... but I've posted multiple patches over the last year (affecting Subversion) and some of them haven't been applied yet.
Some of that is my fail, too. I know I've promised both you and Ivan to
get some patches into APR, and ... well, it hasn't happened. It would
help if we had a list of pending patches.
-- Brane
RE: Subversion checkout issue with certain short names on Windows
Posted by Bert Huijben <be...@qqmail.nl>.
For the mailinglist archive:
> -----Original Message-----
> From: Bert Huijben [mailto:bert@qqmail.nl]
> Sent: zondag 29 maart 2015 17:50
> To: 'Stephen White'; dev@subversion.apache.org
> Subject: RE: Subversion checkout issue with certain short names on Windows
>
> It looks like a bug, but not one that we can fix in Subversion. We use the apr
> library for all our path calculations. I'm guessing that this problem occurs in the
> APR truename implementation, as after that the code to convert back from
> absolute paths back to shortnames would be affected
>
> What do you see if you just type 'svn info API~1.IMP' for that directory. (I don't
> think this is in any way specific to checkout)
>
>
> The obvious workaround is of course to just use the full name instead of the
> short name.
>
>
> Interesting followup questions would be what version of Subversion and what
> version of apr you are using. Perhaps this issue was already fixed.
I've identified the bug in apr and posted a patch to the apr development list to fix the problem. See the ' [Patch] apr_file_path_merge returns incorrect truepath on certain 'short paths' on Windows' thread.
I hope they'll fix the problem in the next release... but I've posted multiple patches over the last year (affecting Subversion) and some of them haven't been applied yet.
Bert
Re: Subversion checkout issue with certain short names on Windows
Posted by Julian Foad <ju...@btopenworld.com>.
Ping! I just noticed this thread had gone quiet. Anybody want to add
anything to the discussion, or file an issue or write a test for it?
- Julian
On 30 March 2015 at 19:32, Stefan Kueng <to...@gmail.com> wrote:
> On 29.03.2015 17:50, Bert Huijben wrote:
>>> -----Original Message-----
>>> From: Stephen White [mailto:swhite@corefiling.com]
>>> Sent: vrijdag 27 maart 2015 14:03
>>> To: dev@subversion.apache.org
>>> Subject: Subversion checkout issue with certain short names on Windows
>>>
>>> If you pass a Windows short-name as the destination to "svn co" then it
>>> will normally work correctly, unless the short name is actually longer
>>> than the "long name" in which case the string handling seems to go awry.
>>>
>>> First a working example:
>>>
>>> > mkdir ALongName.impl
>>> > dir /x
>>> ...
>>> 27/03/2015 12:50 <DIR> ALONGN~1.IMP ALongName.impl
>>> ...
>>> >svn co --depth=files http://svn.apache.org/repos/asf/subversion/trunk
>>> ALONGN~1.IMP
>>> A ALongName.impl\NOTICE
>>> A ALongName.impl\get-deps.sh
>>> A ALongName.impl\Makefile.in
>>> A ALongName.impl\LICENSE
>>> A ALongName.impl\build.conf
>>> A ALongName.impl\win-tests.py
>>> A ALongName.impl\COMMITTERS
>>> A ALongName.impl\README
>>> A ALongName.impl\BUGS
>>> A ALongName.impl\configure.ac
>>> A ALongName.impl\TODO
>>> A ALongName.impl\.ycm_extra_conf.py
>>> A ALongName.impl\INSTALL
>>> A ALongName.impl\CHANGES
>>> A ALongName.impl\autogen.sh
>>> A ALongName.impl\gen-make.py
>>> A ALongName.impl\aclocal.m4
>>> U ALongName.impl
>>> Checked out revision 1669571.
>>>
>>>
>>> That's behaving as expected. SVN checks out into the ALongName.impl
>>> directory, as both the long name and the short name should be
>>> synonymous. It outputs the long name, not the short name given on the
>>> command-line, which was slightly surprising but not a problem.
>>> Presumably the result of some canonicalisation code somewhere.
>>>
>>> Now lets try the same sequence of operations with a different directory
>>> name:
>>>
>>> > mkdir api.impl
>>> > dir /x
>>> ...
>>> 27/03/2015 12:52 <DIR> API~1.IMP api.impl
>>> ...
>>> >svn co --depth=files http://svn.apache.org/repos/asf/subversion/trunk
>>> API~1.IMP
>>> A api.implP\NOTICE
>>> A api.implP\get-deps.sh
>>> A api.implP\Makefile.in
>>> A api.implP\LICENSE
>>> A api.implP\build.conf
>>> A api.implP\win-tests.py
>>> A api.implP\COMMITTERS
>>> A api.implP\README
>>> A api.implP\BUGS
>>> A api.implP\configure.ac
>>> A api.implP\TODO
>>> A api.implP\.ycm_extra_conf.py
>>> A api.implP\INSTALL
>>> A api.implP\CHANGES
>>> A api.implP\autogen.sh
>>> A api.implP\gen-make.py
>>> A api.implP\aclocal.m4
>>> U api.implP
>>> Checked out revision 1669571.
>>>
>>>
>>> Here the behaviour is incorrect, it's created a new folder "api.implP"
>>> and checkout out to that. The spurious "P" character is the last
>>> character of the short name, so it looks like a buffer is being
>>> overwritten incorrectly in situations where the long name turns out to
>>> be shorter than the short name. That happens in this case because,
>>> although "api.impl" is short, the "file extension" is more than 3
>>> characters so this isn't a valid Windows 8.3 (short) name.
>>>
>>> The use of the --depth=files option isn't signficant here, it just makes
>>> the reproduction steps quicker to run.
>>>
>>> I think this is a pretty clear example of a bug, does anyone else agree?
>>
>>
>> It looks like a bug, but not one that we can fix in Subversion. We use the
>> apr library for all our path calculations. I'm guessing that this problem
>> occurs in the APR truename implementation, as after that the code to convert
>> back from absolute paths back to shortnames would be affected
>>
>> What do you see if you just type 'svn info API~1.IMP' for that directory.
>> (I don't think this is in any way specific to checkout)
>>
>>
>> The obvious workaround is of course to just use the full name instead of
>> the short name.
>>
>>
>> Interesting followup questions would be what version of Subversion and
>> what version of apr you are using. Perhaps this issue was already fixed.
>
>
> From what I can see just by looking at the code, the problem is that the
> paths passed on the command line are used almost as-is, with only
> canonicalization done on them. Those paths are used then in some parts of
> the code, but once those paths are passed to the apr file-io functions,
> those paths are (inside apr) translated to the long form. Callbacks then use
> those long paths, but the rest of the code still uses the short paths, and
> in this case they're not identical and that leads to the described problem.
> I'm pretty sure that there are other problems because of this.
>
> I had to fix this in TSVN by making sure that all paths passed to it (either
> on the command line or by entering them anywhere in the UI) are first
> converted to the long format before I pass them on to the svn APIs.
>
> A possible fix would be to do that in the svn command line client as well,
> maybe in svn_opt__arg_canonicalize_path(). But this would require ifdef
> statements and windows specific code, which I'm aware is not something that
> the svn devs really like.
> The calculated paths in svn_opt__arg_canonicalize_path would have to be
> passed through the windows API GetLongPathName():
> https://msdn.microsoft.com/en-us/library/windows/desktop/aa364980%28v=vs.85%29.aspx
>
> Stefan
Re: Subversion checkout issue with certain short names on Windows
Posted by Stefan Kueng <to...@gmail.com>.
On 29.03.2015 17:50, Bert Huijben wrote:
>
>
>> -----Original Message-----
>> From: Stephen White [mailto:swhite@corefiling.com]
>> Sent: vrijdag 27 maart 2015 14:03
>> To: dev@subversion.apache.org
>> Subject: Subversion checkout issue with certain short names on Windows
>>
>> If you pass a Windows short-name as the destination to "svn co" then it
>> will normally work correctly, unless the short name is actually longer
>> than the "long name" in which case the string handling seems to go awry.
>>
>> First a working example:
>>
>> > mkdir ALongName.impl
>> > dir /x
>> ...
>> 27/03/2015 12:50 <DIR> ALONGN~1.IMP ALongName.impl
>> ...
>> >svn co --depth=files http://svn.apache.org/repos/asf/subversion/trunk
>> ALONGN~1.IMP
>> A ALongName.impl\NOTICE
>> A ALongName.impl\get-deps.sh
>> A ALongName.impl\Makefile.in
>> A ALongName.impl\LICENSE
>> A ALongName.impl\build.conf
>> A ALongName.impl\win-tests.py
>> A ALongName.impl\COMMITTERS
>> A ALongName.impl\README
>> A ALongName.impl\BUGS
>> A ALongName.impl\configure.ac
>> A ALongName.impl\TODO
>> A ALongName.impl\.ycm_extra_conf.py
>> A ALongName.impl\INSTALL
>> A ALongName.impl\CHANGES
>> A ALongName.impl\autogen.sh
>> A ALongName.impl\gen-make.py
>> A ALongName.impl\aclocal.m4
>> U ALongName.impl
>> Checked out revision 1669571.
>>
>>
>> That's behaving as expected. SVN checks out into the ALongName.impl
>> directory, as both the long name and the short name should be
>> synonymous. It outputs the long name, not the short name given on the
>> command-line, which was slightly surprising but not a problem.
>> Presumably the result of some canonicalisation code somewhere.
>>
>> Now lets try the same sequence of operations with a different directory
>> name:
>>
>> > mkdir api.impl
>> > dir /x
>> ...
>> 27/03/2015 12:52 <DIR> API~1.IMP api.impl
>> ...
>> >svn co --depth=files http://svn.apache.org/repos/asf/subversion/trunk
>> API~1.IMP
>> A api.implP\NOTICE
>> A api.implP\get-deps.sh
>> A api.implP\Makefile.in
>> A api.implP\LICENSE
>> A api.implP\build.conf
>> A api.implP\win-tests.py
>> A api.implP\COMMITTERS
>> A api.implP\README
>> A api.implP\BUGS
>> A api.implP\configure.ac
>> A api.implP\TODO
>> A api.implP\.ycm_extra_conf.py
>> A api.implP\INSTALL
>> A api.implP\CHANGES
>> A api.implP\autogen.sh
>> A api.implP\gen-make.py
>> A api.implP\aclocal.m4
>> U api.implP
>> Checked out revision 1669571.
>>
>>
>> Here the behaviour is incorrect, it's created a new folder "api.implP"
>> and checkout out to that. The spurious "P" character is the last
>> character of the short name, so it looks like a buffer is being
>> overwritten incorrectly in situations where the long name turns out to
>> be shorter than the short name. That happens in this case because,
>> although "api.impl" is short, the "file extension" is more than 3
>> characters so this isn't a valid Windows 8.3 (short) name.
>>
>> The use of the --depth=files option isn't signficant here, it just makes
>> the reproduction steps quicker to run.
>>
>> I think this is a pretty clear example of a bug, does anyone else agree?
>
> It looks like a bug, but not one that we can fix in Subversion. We use the apr library for all our path calculations. I'm guessing that this problem occurs in the APR truename implementation, as after that the code to convert back from absolute paths back to shortnames would be affected
>
> What do you see if you just type 'svn info API~1.IMP' for that directory. (I don't think this is in any way specific to checkout)
>
>
> The obvious workaround is of course to just use the full name instead of the short name.
>
>
> Interesting followup questions would be what version of Subversion and what version of apr you are using. Perhaps this issue was already fixed.
From what I can see just by looking at the code, the problem is that
the paths passed on the command line are used almost as-is, with only
canonicalization done on them. Those paths are used then in some parts
of the code, but once those paths are passed to the apr file-io
functions, those paths are (inside apr) translated to the long form.
Callbacks then use those long paths, but the rest of the code still uses
the short paths, and in this case they're not identical and that leads
to the described problem.
I'm pretty sure that there are other problems because of this.
I had to fix this in TSVN by making sure that all paths passed to it
(either on the command line or by entering them anywhere in the UI) are
first converted to the long format before I pass them on to the svn APIs.
A possible fix would be to do that in the svn command line client as
well, maybe in svn_opt__arg_canonicalize_path(). But this would require
ifdef statements and windows specific code, which I'm aware is not
something that the svn devs really like.
The calculated paths in svn_opt__arg_canonicalize_path would have to be
passed through the windows API GetLongPathName():
https://msdn.microsoft.com/en-us/library/windows/desktop/aa364980%28v=vs.85%29.aspx
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest interface to (Sub)version control
/_/ \_\ http://tortoisesvn.net
RE: Subversion checkout issue with certain short names on Windows
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Stephen White [mailto:swhite@corefiling.com]
> Sent: vrijdag 27 maart 2015 14:03
> To: dev@subversion.apache.org
> Subject: Subversion checkout issue with certain short names on Windows
>
> If you pass a Windows short-name as the destination to "svn co" then it
> will normally work correctly, unless the short name is actually longer
> than the "long name" in which case the string handling seems to go awry.
>
> First a working example:
>
> > mkdir ALongName.impl
> > dir /x
> ...
> 27/03/2015 12:50 <DIR> ALONGN~1.IMP ALongName.impl
> ...
> >svn co --depth=files http://svn.apache.org/repos/asf/subversion/trunk
> ALONGN~1.IMP
> A ALongName.impl\NOTICE
> A ALongName.impl\get-deps.sh
> A ALongName.impl\Makefile.in
> A ALongName.impl\LICENSE
> A ALongName.impl\build.conf
> A ALongName.impl\win-tests.py
> A ALongName.impl\COMMITTERS
> A ALongName.impl\README
> A ALongName.impl\BUGS
> A ALongName.impl\configure.ac
> A ALongName.impl\TODO
> A ALongName.impl\.ycm_extra_conf.py
> A ALongName.impl\INSTALL
> A ALongName.impl\CHANGES
> A ALongName.impl\autogen.sh
> A ALongName.impl\gen-make.py
> A ALongName.impl\aclocal.m4
> U ALongName.impl
> Checked out revision 1669571.
>
>
> That's behaving as expected. SVN checks out into the ALongName.impl
> directory, as both the long name and the short name should be
> synonymous. It outputs the long name, not the short name given on the
> command-line, which was slightly surprising but not a problem.
> Presumably the result of some canonicalisation code somewhere.
>
> Now lets try the same sequence of operations with a different directory
> name:
>
> > mkdir api.impl
> > dir /x
> ...
> 27/03/2015 12:52 <DIR> API~1.IMP api.impl
> ...
> >svn co --depth=files http://svn.apache.org/repos/asf/subversion/trunk
> API~1.IMP
> A api.implP\NOTICE
> A api.implP\get-deps.sh
> A api.implP\Makefile.in
> A api.implP\LICENSE
> A api.implP\build.conf
> A api.implP\win-tests.py
> A api.implP\COMMITTERS
> A api.implP\README
> A api.implP\BUGS
> A api.implP\configure.ac
> A api.implP\TODO
> A api.implP\.ycm_extra_conf.py
> A api.implP\INSTALL
> A api.implP\CHANGES
> A api.implP\autogen.sh
> A api.implP\gen-make.py
> A api.implP\aclocal.m4
> U api.implP
> Checked out revision 1669571.
>
>
> Here the behaviour is incorrect, it's created a new folder "api.implP"
> and checkout out to that. The spurious "P" character is the last
> character of the short name, so it looks like a buffer is being
> overwritten incorrectly in situations where the long name turns out to
> be shorter than the short name. That happens in this case because,
> although "api.impl" is short, the "file extension" is more than 3
> characters so this isn't a valid Windows 8.3 (short) name.
>
> The use of the --depth=files option isn't signficant here, it just makes
> the reproduction steps quicker to run.
>
> I think this is a pretty clear example of a bug, does anyone else agree?
It looks like a bug, but not one that we can fix in Subversion. We use the apr library for all our path calculations. I'm guessing that this problem occurs in the APR truename implementation, as after that the code to convert back from absolute paths back to shortnames would be affected
What do you see if you just type 'svn info API~1.IMP' for that directory. (I don't think this is in any way specific to checkout)
The obvious workaround is of course to just use the full name instead of the short name.
Interesting followup questions would be what version of Subversion and what version of apr you are using. Perhaps this issue was already fixed.
Bert