You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Markus Schaber <m....@codesys.com> on 2013/08/22 14:26:41 UTC
Long Pathes on Windows and SQLite
Hi,
Some days ago, I stumbled across a problem, and could analyze it a little bit those days.
Windows has a general limit on the path length of 260 characters on some APIs, while others allow a length of about 32k characters (with some restrictions).
Most (if not all) of the SVN codebase seems to work fine with those long pathes. However, it seems that SQLite is not (yet) long path clean:
--- snip ---
mkdir repo
svnadmin create repo
svn checkout file:///%CD%/repo workingcopy
svn mkdir workingcopy\foo
svn commit -mm workingcopy
SET fifty=01234567890123456789012345678901234567890123456789
SET twofifty=%fifty%%fifty%%fifty%%fifty%%fifty%
rem svn mkdir does not work with the long path, but the following
rem workaround is fine:
pushd workingcopy
svn update
svn mkdir newdir
svn mv newdir %twofifty%
svn propset svn:externals "^/foo foo" %twofifty%
svn update
popd
--- snap ---
The last "svn update" command fails with the following error message:
--- snip ---
Updating '.':
Fetching external item into '0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789\foo':
svn: warning: W200030: sqlite: unable to open database file: 'D:/test/longpathtest/workingcopy/0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789/foo/.svn/wc.db'
Revision 1.
svn: E205011: Failure occurred processing one or more externals definitions
--- snap ---
I already mentioned this problem on IRC, where Bert meant that it is an issue in SQLite itself which has its own windows abstracton layer.
Now, some questions arise:
1) Is this issue worth to file a bug report in the SVN database? (Up to now, none of our customers has been bitten by that issue, but it is very likely that one will show up sooner or later...)
And for the folks more experienced with SQLite:
2) Is the SQLite project open and responsive for bug reports?
3) Is it possible to feed SQLite a "long" path using the \\?\ prefix?
4) Is it possible to work around this issue by providing our own SQLite VFS (which may delegate everything except file opening to the default one)?
Best regards
Markus Schaber
CODESYS(r) a trademark of 3S-Smart Software Solutions GmbH
Inspiring Automation Solutions
3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50
E-Mail: m.schaber@codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com
Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
AW: AW: Long Pathes on Windows and SQLite
Posted by Markus Schaber <m....@codesys.com>.
Hi, Bert,
Von: Bert Huijben [mailto:bert@qqmail.nl]
> From: Branko Čibej [mailto:brane@wandisco.com]
> > On 29.08.2013 08:10, Markus Schaber wrote:
> [Fixing long path issues for externals]
>
> Given that the Windows Explorer and many other programs don't handle these
> long paths anyway, I'm not really interested in this specific fix. A nicer
> fix would be to finally start storing external information in their parent
> working copy.
> (Which is on my backlog combined with 'svn stash' preparations)
That just reminds me that there is still no official way to rename/move an external wc without losing local changes within that subtree.
Best regards
Markus Schaber
CODESYS® a trademark of 3S-Smart Software Solutions GmbH
Inspiring Automation Solutions
3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50
E-Mail: m.schaber@codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com
Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
RE: AW: Long Pathes on Windows and SQLite
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Branko Čibej [mailto:brane@wandisco.com]
> Sent: donderdag 29 augustus 2013 11:01
> To: dev@subversion.apache.org
> Subject: Re: AW: Long Pathes on Windows and SQLite
>
> On 29.08.2013 08:10, Markus Schaber wrote:
> > Hi, Bert,
> >
> > Von: Bert Huijben [mailto:bert@qqmail.nl]
> >> From: Markus Schaber [mailto:m.schaber@codesys.com]
> >>> Von: Branko Čibej [mailto:brane@wandisco.com]
> >>>> On 22.08.2013 14:26, Markus Schaber wrote:
> >>>>> Some days ago, I stumbled across a problem, and could analyze it a
> >>>>> little bit those days.
> >>>>>
> >>>>> Windows has a general limit on the path length of 260 characters
> >>>>> on some APIs, while others allow a length of about 32k characters
> >>>>> (with some restrictions).
> >>>>>
> >>>>> Most (if not all) of the SVN codebase seems to work fine with
> >>>>> those long pathes.
> >>>> Yes, APR takes care to use the Unicode APIs and use \\?.
> >>>>
> >>>>> However, it seems that SQLite is not (yet) long path clean:
> >>> [....]
> >>> I did also contact the SQLite developers about this issue, and in
> >>> http://www.mail-archive.com/sqlite-users@sqlite.org/msg78853.html,
> >>> Richard Hipp saids:
> >>>
> >>> | SQLite version 3.8.0 allows for longer windows pathnames up to
> >>> | 3*MAX_PATH bytes, which is 3x more space that was allowed before.
> >>> | This
> >>> is
> >>> | still not 32K but might be sufficient for your needs. The limit can
> >>> | be extended at compile-time.
> >>>
> >>> This could help us to solve the problem.
> >> Looking at the patch this is not going to solve your issue.
> >> [http://www.sqlite.org/src/info/bb06e15790]
> >>
> >> This makes it possible to use paths that need more bytes to encode a
> path,
> >> but it doesn't lift the MAX_PATH characters in Unicode limitation in the
> way
> >> APR does via absolute paths and using the \\?\ override to certain API
> >> functions.
> > There's good news on the SQLite side:
> >
> > http://www.mail-archive.com/sqlite-users@sqlite.org/msg78853.html
> says:
> >
> > | The latest check-in on trunk adds a new windows VFS module called
> > | "win32-longpath" which accepts the full-length 32KB pathnames. Add the
> > | string "win32-longpath" as the 4th argument to sqlite3_open_v2() (on
> > | windows only, of course) and everything should just work after that.
> >
> > The source change is at http://www.sqlite.org/cgi/src/info/12d0a8859d
> >
> > It seems this new VFS supports long path names if they start with the \\?\
> prefix.
> >
> > So as soon as the next SQLite version is released, we can include support
> for it into SVN, maybe with a compile-time check on what SQLite version is
> available.
>
> It will have to be a runtime check, but that's OK. On the other hand,
> we'd have to hoist APR's code for creating those prefixed paths in the
> first place. It's currently private in APR, and I'm not too happy with
> the idea of duplicating it ...
I'm starting to think that we might want to raise our requirements to 3.8.0 for Subversion 1.9.0 anyway for other reasons (Partial indexes will improve performance for quite some cases), so maybe we should go to 3.8.1.
Given that the Windows Explorer and many other programs don't handle these long paths anyway, I'm not really interested in this specific fix. A nicer fix would be to finally start storing external information in their parent working copy.
(Which is on my backlog combined with 'svn stash' preparations)
Bert
Re: AW: Long Pathes on Windows and SQLite
Posted by Branko Čibej <br...@wandisco.com>.
On 29.08.2013 08:10, Markus Schaber wrote:
> Hi, Bert,
>
> Von: Bert Huijben [mailto:bert@qqmail.nl]
>> From: Markus Schaber [mailto:m.schaber@codesys.com]
>>> Von: Branko Čibej [mailto:brane@wandisco.com]
>>>> On 22.08.2013 14:26, Markus Schaber wrote:
>>>>> Some days ago, I stumbled across a problem, and could analyze it a
>>>>> little bit those days.
>>>>>
>>>>> Windows has a general limit on the path length of 260 characters
>>>>> on some APIs, while others allow a length of about 32k characters
>>>>> (with some restrictions).
>>>>>
>>>>> Most (if not all) of the SVN codebase seems to work fine with
>>>>> those long pathes.
>>>> Yes, APR takes care to use the Unicode APIs and use \\?.
>>>>
>>>>> However, it seems that SQLite is not (yet) long path clean:
>>> [....]
>>> I did also contact the SQLite developers about this issue, and in
>>> http://www.mail-archive.com/sqlite-users@sqlite.org/msg78853.html,
>>> Richard Hipp saids:
>>>
>>> | SQLite version 3.8.0 allows for longer windows pathnames up to
>>> | 3*MAX_PATH bytes, which is 3x more space that was allowed before.
>>> | This
>>> is
>>> | still not 32K but might be sufficient for your needs. The limit can
>>> | be extended at compile-time.
>>>
>>> This could help us to solve the problem.
>> Looking at the patch this is not going to solve your issue.
>> [http://www.sqlite.org/src/info/bb06e15790]
>>
>> This makes it possible to use paths that need more bytes to encode a path,
>> but it doesn't lift the MAX_PATH characters in Unicode limitation in the way
>> APR does via absolute paths and using the \\?\ override to certain API
>> functions.
> There's good news on the SQLite side:
>
> http://www.mail-archive.com/sqlite-users@sqlite.org/msg78853.html says:
>
> | The latest check-in on trunk adds a new windows VFS module called
> | "win32-longpath" which accepts the full-length 32KB pathnames. Add the
> | string "win32-longpath" as the 4th argument to sqlite3_open_v2() (on
> | windows only, of course) and everything should just work after that.
>
> The source change is at http://www.sqlite.org/cgi/src/info/12d0a8859d
>
> It seems this new VFS supports long path names if they start with the \\?\ prefix.
>
> So as soon as the next SQLite version is released, we can include support for it into SVN, maybe with a compile-time check on what SQLite version is available.
It will have to be a runtime check, but that's OK. On the other hand,
we'd have to hoist APR's code for creating those prefixed paths in the
first place. It's currently private in APR, and I'm not too happy with
the idea of duplicating it ...
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com
AW: Long Pathes on Windows and SQLite
Posted by Markus Schaber <m....@codesys.com>.
Hi, Bert,
Von: Bert Huijben [mailto:bert@qqmail.nl]
> From: Markus Schaber [mailto:m.schaber@codesys.com]
> > Von: Branko Čibej [mailto:brane@wandisco.com]
> > > On 22.08.2013 14:26, Markus Schaber wrote:
> > > > Some days ago, I stumbled across a problem, and could analyze it a
> > > > little bit those days.
> > > >
> > > > Windows has a general limit on the path length of 260 characters
> > > > on some APIs, while others allow a length of about 32k characters
> > > > (with some restrictions).
> > > >
> > > > Most (if not all) of the SVN codebase seems to work fine with
> > > > those long pathes.
> > >
> > > Yes, APR takes care to use the Unicode APIs and use \\?.
> > >
> > > > However, it seems that SQLite is not (yet) long path clean:
> > [....]
> > I did also contact the SQLite developers about this issue, and in
> > http://www.mail-archive.com/sqlite-users@sqlite.org/msg78853.html,
> > Richard Hipp saids:
> >
> > | SQLite version 3.8.0 allows for longer windows pathnames up to
> > | 3*MAX_PATH bytes, which is 3x more space that was allowed before.
> > | This
> > is
> > | still not 32K but might be sufficient for your needs. The limit can
> > | be extended at compile-time.
> >
> > This could help us to solve the problem.
>
> Looking at the patch this is not going to solve your issue.
> [http://www.sqlite.org/src/info/bb06e15790]
>
> This makes it possible to use paths that need more bytes to encode a path,
> but it doesn't lift the MAX_PATH characters in Unicode limitation in the way
> APR does via absolute paths and using the \\?\ override to certain API
> functions.
There's good news on the SQLite side:
http://www.mail-archive.com/sqlite-users@sqlite.org/msg78853.html says:
| The latest check-in on trunk adds a new windows VFS module called
| "win32-longpath" which accepts the full-length 32KB pathnames. Add the
| string "win32-longpath" as the 4th argument to sqlite3_open_v2() (on
| windows only, of course) and everything should just work after that.
The source change is at http://www.sqlite.org/cgi/src/info/12d0a8859d
It seems this new VFS supports long path names if they start with the \\?\ prefix.
So as soon as the next SQLite version is released, we can include support for it into SVN, maybe with a compile-time check on what SQLite version is available.
Best regards
Markus Schaber
CODESYS® a trademark of 3S-Smart Software Solutions GmbH
Inspiring Automation Solutions
3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50
E-Mail: m.schaber@codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com
Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
RE: Long Pathes on Windows and SQLite
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Markus Schaber [mailto:m.schaber@codesys.com]
> Sent: vrijdag 23 augustus 2013 16:25
> To: Branko Čibej; dev@subversion.apache.org
> Subject: AW: Long Pathes on Windows and SQLite
>
> Hi, All,
>
> Von: Branko Čibej [mailto:brane@wandisco.com]
> > On 22.08.2013 14:26, Markus Schaber wrote:
> > > Some days ago, I stumbled across a problem, and could analyze it a little
> > > bit those days.
> > >
> > > Windows has a general limit on the path length of 260 characters on some
> > > APIs, while others allow a length of about 32k characters (with some
> > > restrictions).
> > >
> > > Most (if not all) of the SVN codebase seems to work fine with those long
> > > pathes.
> >
> > Yes, APR takes care to use the Unicode APIs and use \\?.
> >
> > > However, it seems that SQLite is not (yet) long path clean:
> >
> > Sigh...
> > [...snip...]
> >
> > > I already mentioned this problem on IRC, where Bert meant that it is an
> > > issue in SQLite itself which has its own windows abstracton layer.
> > >
> > > Now, some questions arise:
> > > 1) Is this issue worth to file a bug report in the SVN database? (Up
> > > to now, none of our customers has been bitten by that issue, but it is
> > > very likely that one will show up sooner or later...)
> >
> > IMO it's a good idea to track this, but I propose you also file the report to
> > SQLite (and put a link to that report in our report).
> > [snip]
>
> I did file http://subversion.tigris.org/issues/show_bug.cgi?id=4417.
>
> I did also contact the SQLite developers about this issue, and in
> http://www.mail-archive.com/sqlite-users@sqlite.org/msg78853.html,
> Richard Hipp saids:
>
> | SQLite version 3.8.0 allows for longer windows pathnames up to
> | 3*MAX_PATH bytes, which is 3x more space that was allowed before. This
> is
> | still not 32K but might be sufficient for your needs. The limit can be
> | extended at compile-time.
>
> This could help us to solve the problem.
Looking at the patch this is not going to solve your issue.
[http://www.sqlite.org/src/info/bb06e15790]
This makes it possible to use paths that need more bytes to encode a path, but it doesn't lift the MAX_PATH characters in Unicode limitation in the way APR does via absolute paths and using the \\?\ override to certain API functions.
Bert
AW: Long Pathes on Windows and SQLite
Posted by Markus Schaber <m....@codesys.com>.
Hi, All,
Von: Branko Čibej [mailto:brane@wandisco.com]
> On 22.08.2013 14:26, Markus Schaber wrote:
> > Some days ago, I stumbled across a problem, and could analyze it a little
> > bit those days.
> >
> > Windows has a general limit on the path length of 260 characters on some
> > APIs, while others allow a length of about 32k characters (with some
> > restrictions).
> >
> > Most (if not all) of the SVN codebase seems to work fine with those long
> > pathes.
>
> Yes, APR takes care to use the Unicode APIs and use \\?.
>
> > However, it seems that SQLite is not (yet) long path clean:
>
> Sigh...
> [...snip...]
>
> > I already mentioned this problem on IRC, where Bert meant that it is an
> > issue in SQLite itself which has its own windows abstracton layer.
> >
> > Now, some questions arise:
> > 1) Is this issue worth to file a bug report in the SVN database? (Up
> > to now, none of our customers has been bitten by that issue, but it is
> > very likely that one will show up sooner or later...)
>
> IMO it's a good idea to track this, but I propose you also file the report to
> SQLite (and put a link to that report in our report).
> [snip]
I did file http://subversion.tigris.org/issues/show_bug.cgi?id=4417.
I did also contact the SQLite developers about this issue, and in http://www.mail-archive.com/sqlite-users@sqlite.org/msg78853.html, Richard Hipp saids:
| SQLite version 3.8.0 allows for longer windows pathnames up to
| 3*MAX_PATH bytes, which is 3x more space that was allowed before. This is
| still not 32K but might be sufficient for your needs. The limit can be
| extended at compile-time.
This could help us to solve the problem.
Best regards
Markus Schaber
CODESYS® a trademark of 3S-Smart Software Solutions GmbH
Inspiring Automation Solutions
3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50
E-Mail: m.schaber@codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com
Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
Re: Long Pathes on Windows and SQLite
Posted by Branko Čibej <br...@wandisco.com>.
On 22.08.2013 14:26, Markus Schaber wrote:
> Hi,
>
> Some days ago, I stumbled across a problem, and could analyze it a little bit those days.
>
> Windows has a general limit on the path length of 260 characters on some APIs, while others allow a length of about 32k characters (with some restrictions).
>
> Most (if not all) of the SVN codebase seems to work fine with those long pathes.
Yes, APR takes care to use the Unicode APIs and use \\?.
> However, it seems that SQLite is not (yet) long path clean:
Sigh...
[...snip...]
> I already mentioned this problem on IRC, where Bert meant that it is an issue in SQLite itself which has its own windows abstracton layer.
>
> Now, some questions arise:
> 1) Is this issue worth to file a bug report in the SVN database? (Up to now, none of our customers has been bitten by that issue, but it is very likely that one will show up sooner or later...)
IMO it's a good idea to track this, but I propose you also file the
report to SQLite (and put a link to that report in our report).
> And for the folks more experienced with SQLite:
> 2) Is the SQLite project open and responsive for bug reports?
> 3) Is it possible to feed SQLite a "long" path using the \\?\ prefix?
No idea, on either count.
> 4) Is it possible to work around this issue by providing our own SQLite VFS (which may delegate everything except file opening to the default one)?
I haven't looked closely at SQLite's VFS API, but a quick look at the
VFS object indicates that this would probably work -- as a stogap.
It would be a lot better to have a fix in SQLite; it shouldn't be too
hard to create a patch, never mind that the SQLite code is a mess. :)
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com