You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Josh Rowe <Jo...@microsoft.com> on 2012/10/19 04:41:18 UTC
Bug: Windows Server 2012 NTFS Deduplication, Special Status has
Changed
I didn't see this reported in the bug database, nor did a casual web search turn this up:
When de-duplication is enabled on a Windows Server 2012 NTFS volume, the de-duplication engine replaces files that contain duplicate data with reparse points. Subversion then reports that those files have unexpectedly changed special status. For example:
PS I:\temp\packages-2> svn commit
svn: E145001: Commit failed (details follow):
svn: E145001: Entry 'I:\temp\packages-2\Roslyn.Compilers.VisualBasic.1.2.20906.2\lib\net45\Roslyn.Compilers.VisualBasic.dll' has unexpectedly changed special status
Once the files have been deduplicated, the workaround is to disable deduplication on a folder, then copy the versioned controlled directory to the new location.
Repro steps:
Enable deduplication on a Win2012 NTFS volume.
Check out two copies of a repository onto the deduplicated volume.
Wait for deduplication to occur, or force it to happen.
Make a change to one copy of the repository.
Attempt to commit with "svn commit".
Jmr
RE: Bug: Windows Server 2012 NTFS Deduplication, Special Status has Changed
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Duncan Booth [mailto:duncan.booth@suttoncourtenay.org.uk]
> Sent: vrijdag 19 oktober 2012 15:44
> To: dev@subversion.apache.org
> Subject: Re: Bug: Windows Server 2012 NTFS Deduplication, Special Status
> has Changed
>
> =?UTF-8?B?QnJhbmtvIMSMaWJlag==?= <br...@wandisco.com> wrote:
>
> > On 19.10.2012 04:41, Josh Rowe wrote:
> >> I didn't see this reported in the bug database, nor did a casual web
> >> search turn this up:
> >>
> >> When de-duplication is enabled on a Windows Server 2012 NTFS volume,
> >> the de-duplication engine replaces files that contain duplicate data
> >> with reparse points. Subversion then reports that those files have
> >> unexpectedly changed special status. For example:
> >>
> >> PS I:\temp\packages-2> svn commit
> >> svn: E145001: Commit failed (details follow):
> >> svn: E145001: Entry
> >> 'I:\temp\packages-2\Roslyn.Compilers.VisualBasic.1.2.20906.2\lib
> \net45
> >> \Roslyn.Compilers.VisualBasic.dll' has unexpectedly changed special
> >> status
> >>
> >> Once the files have been deduplicated, the workaround is to disable
> >> deduplication on a folder, then copy the versioned controlled
> >> directory to the new location.
> >>
> >> Repro steps:
> >>
> >> Enable deduplication on a Win2012 NTFS volume.
> >> Check out two copies of a repository onto the deduplicated volume.
> >> Wait for deduplication to occur, or force it to happen.
> >> Make a change to one copy of the repository.
> >> Attempt to commit with "svn commit".
> >
> >
> > This is not a Subversion bug. If the user tells NTFS to replace files
> > with application-visible symbolic links, which apparently is what NTFS
> > deduplication does, then there's hardly a sane way for Subversion to
> > know that the user didn't do this intentionally.
> >
> Symbolic links in Windows are reparse points, but not all reparse points
> are symbolic links. The de-duplication appears to use a specific data
> deduplication type of reparse point, so Subversion could make the
> distinction in a perfectly sane manner.
>
> Every reparse point has an identifier tag identifying the type of
> reparse point which includes the following possible values:
>
> IO_REPARSE_TAG_CSV
> IO_REPARSE_TAG_DEDUP
> IO_REPARSE_TAG_DFS
> IO_REPARSE_TAG_DFSR
> IO_REPARSE_TAG_HSM
> IO_REPARSE_TAG_HSM2
> IO_REPARSE_TAG_MOUNT_POINT
> IO_REPARSE_TAG_NFS
> IO_REPARSE_TAG_RESERVED_ONE
> IO_REPARSE_TAG_RESERVED_RANGE
> IO_REPARSE_TAG_RESERVED_ZERO
> IO_REPARSE_TAG_SIS
> IO_REPARSE_TAG_SYMLINK
> IO_REPARSE_TAG_WIM
> (other values are also possible as Microsoft provide a mechanism for
> defining new tags)
>
> If Subversion treated the IO_REPARSE_TAG_SYMLINK as a symlink and any
> other type as a normal file that might be a sane solution.
>
>
> It is unfair to blame the user: the user may not have control over the
> settings an administrator makes for the server file system. There is a
> workaround (have Subversion checkouts excluded from deduplication by
> policy) but that assumes the administrator of the server is willing to
> make appropriate policy changes.
The APR library that Subversion uses to detect whether a filesystem is a
file, a directory or some link doesn't report this information to us.
Retrieving this information ourself per reparse point would be extremely
expensive on network fileystems as it would be an additional operation per
node.
Also looking at the number of previous reports (0), it appears that this is
an extreme edge case, mostly applicable to fileservers while we recommend
using working copies on local harddisks.
"Working copies are cheap"..., etc.
I think the proper fix belongs in apr, which will then automatically fix
Subversion. The FindFirstFile() api reports the necessary flag, so I don't
think it will be very hard to fix it there.
Bert
Re: Bug: Windows Server 2012 NTFS Deduplication, Special Status has Changed
Posted by Duncan Booth <du...@suttoncourtenay.org.uk>.
=?UTF-8?B?QnJhbmtvIMSMaWJlag==?= <br...@wandisco.com> wrote:
> On 19.10.2012 04:41, Josh Rowe wrote:
>> I didn't see this reported in the bug database, nor did a casual web
>> search turn this up:
>>
>> When de-duplication is enabled on a Windows Server 2012 NTFS volume,
>> the de-duplication engine replaces files that contain duplicate data
>> with reparse points. Subversion then reports that those files have
>> unexpectedly changed special status. For example:
>>
>> PS I:\temp\packages-2> svn commit
>> svn: E145001: Commit failed (details follow):
>> svn: E145001: Entry
>> 'I:\temp\packages-2\Roslyn.Compilers.VisualBasic.1.2.20906.2\lib
\net45
>> \Roslyn.Compilers.VisualBasic.dll' has unexpectedly changed special
>> status
>>
>> Once the files have been deduplicated, the workaround is to disable
>> deduplication on a folder, then copy the versioned controlled
>> directory to the new location.
>>
>> Repro steps:
>>
>> Enable deduplication on a Win2012 NTFS volume.
>> Check out two copies of a repository onto the deduplicated volume.
>> Wait for deduplication to occur, or force it to happen.
>> Make a change to one copy of the repository.
>> Attempt to commit with "svn commit".
>
>
> This is not a Subversion bug. If the user tells NTFS to replace files
> with application-visible symbolic links, which apparently is what NTFS
> deduplication does, then there's hardly a sane way for Subversion to
> know that the user didn't do this intentionally.
>
Symbolic links in Windows are reparse points, but not all reparse points
are symbolic links. The de-duplication appears to use a specific data
deduplication type of reparse point, so Subversion could make the
distinction in a perfectly sane manner.
Every reparse point has an identifier tag identifying the type of
reparse point which includes the following possible values:
IO_REPARSE_TAG_CSV
IO_REPARSE_TAG_DEDUP
IO_REPARSE_TAG_DFS
IO_REPARSE_TAG_DFSR
IO_REPARSE_TAG_HSM
IO_REPARSE_TAG_HSM2
IO_REPARSE_TAG_MOUNT_POINT
IO_REPARSE_TAG_NFS
IO_REPARSE_TAG_RESERVED_ONE
IO_REPARSE_TAG_RESERVED_RANGE
IO_REPARSE_TAG_RESERVED_ZERO
IO_REPARSE_TAG_SIS
IO_REPARSE_TAG_SYMLINK
IO_REPARSE_TAG_WIM
(other values are also possible as Microsoft provide a mechanism for
defining new tags)
If Subversion treated the IO_REPARSE_TAG_SYMLINK as a symlink and any
other type as a normal file that might be a sane solution.
It is unfair to blame the user: the user may not have control over the
settings an administrator makes for the server file system. There is a
workaround (have Subversion checkouts excluded from deduplication by
policy) but that assumes the administrator of the server is willing to
make appropriate policy changes.
Re: Bug: Windows Server 2012 NTFS Deduplication, Special Status has
Changed
Posted by Branko Čibej <br...@wandisco.com>.
On 19.10.2012 04:41, Josh Rowe wrote:
> I didn't see this reported in the bug database, nor did a casual web search turn this up:
>
> When de-duplication is enabled on a Windows Server 2012 NTFS volume, the de-duplication engine replaces files that contain duplicate data with reparse points. Subversion then reports that those files have unexpectedly changed special status. For example:
>
> PS I:\temp\packages-2> svn commit
> svn: E145001: Commit failed (details follow):
> svn: E145001: Entry 'I:\temp\packages-2\Roslyn.Compilers.VisualBasic.1.2.20906.2\lib\net45\Roslyn.Compilers.VisualBasic.dll' has unexpectedly changed special status
>
> Once the files have been deduplicated, the workaround is to disable deduplication on a folder, then copy the versioned controlled directory to the new location.
>
> Repro steps:
>
> Enable deduplication on a Win2012 NTFS volume.
> Check out two copies of a repository onto the deduplicated volume.
> Wait for deduplication to occur, or force it to happen.
> Make a change to one copy of the repository.
> Attempt to commit with "svn commit".
This is not a Subversion bug. If the user tells NTFS to replace files
with application-visible symbolic links, which apparently is what NTFS
deduplication does, then there's hardly a sane way for Subversion to
know that the user didn't do this intentionally.
-- Brane
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download