You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Ivan Zhakov <iv...@visualsvn.com> on 2014/03/12 14:58:28 UTC
Re: Subversion checked-out files not indexed in Windows search
On 10 March 2014 18:25, Gareth McCaughan
<Ga...@lightblueoptics.com> wrote:
> It appears that files checked out using Subversion (1.7 onwards)
> are marked as not indexable by Windows search. The way in which
> this happens appears to be as follows:
>
> When the WC is created:
>
> 1. .svn directory is created, and marked as hidden and not indexable.
> 2. .svn/tmp directory is created, and inherits that not-indexable
> property. (Or maybe it's given it explicitly; I haven't checked.)
>
> On checkout:
>
> 1. Temporary file is created under .svn, and accordingly gets its
> not-indexable flag set.
> 2. Temporary file is moved to its proper location; this doesn't
> cause the not-indexable flag to be cleared.
>
> So the point is that the NOT_CONTENT_INDEXABLE attribute on a
> directory doesn't exactly mean "don't search inside this directory"
> but "make files created in this directory searchable". Then when
> they get moved elsewhere they retain that unsearchability.
>
> (If skeptical, you can see this "by hand" on a Windows machine,
> as follows: Create a directory A and subdirectory A\B. Right-click
> on B, select "Properties", select "Advanced", uncheck the
> allow-indexing box. Create two files A\B\C.txt and A\B\D.txt
> and put the word "wombat" in each. Now right-click D.txt,
> select "Properties", select "Advanced", and *check* the
> allow-indexing box; and drag C.txt out of A\B and into A.
> Now, in A, search for "wombat" using Windows Search. It will
> find D.txt even though it's in the unsearchable folder A\B,
> and it will not find C.txt even though it's in the searchable
> folder A, because what actually matters is the attributes on
> the files.)
>
> *
>
> Windows Search is of course generally a poor tool for searching
> source code, but this still seems fairly clearly (1) unintended
> and (2) undesirable.
>
> Should I file an issue? If not, will someone else?
>
> *
>
> I am not familiar with the code responsible for this, but
> a little browsing suggests that the rename is being done
> in svn_stream__install_stream (called from run_file_install
> in libsvn_wc/workqueue.c) by a call to SetFileInformationByHandle,
> and that this function is capable of changing the file attributes
> at the same time as doing the rename. I make no claim that
> this would actually be a good solution.
>
> *
>
> (Note 1. This seems to have been reported informally a couple
> of times before, but I don't see that it's ever received an
> official answer, had an issue filed in the Subversion issue
> tracker, or been fixed. Apologies if it has and I've missed it.)
>
> (Note 2. I have observed the behaviour described above on
> Windows 7 using svn 1.7.5, and on Windows 8 using svn 1.8.3.
> There is nothing in the CHANGES file to suggest it's likely
> to have been fixed since then, but I can check with the latest
> release if that's necessary.)
>
> (Note 3. I am not subscribed to users@ but will check for
> replies in the online archives; best to Cc: me, though, if
> you have questions or comments you want to be sure I'll see.)
>
It looks like serious issue. Could you please file issue in Subversion
issue tracker:
http://subversion.tigris.org/issues
--
Ivan Zhakov
CTO | VisualSVN | http://www.visualsvn.com
Re: Subversion checked-out files not indexed in Windows search
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 13 March 2014 21:52, Stefan Kueng <to...@gmail.com> wrote:
> On 13.03.2014 14:14, Bert Huijben wrote:
>>
>>
>>
>>> -----Original Message-----
>>> From: Ivan Zhakov [mailto:ivan@visualsvn.com]
>>> Sent: donderdag 13 maart 2014 13:56
>>> To: Branko Čibej
>>> Cc: Subversion Development
>>> Subject: Re: Subversion checked-out files not indexed in Windows search
>>>
>>> On 13 March 2014 16:46, Branko Čibej <br...@wandisco.com> wrote:
>>>>
>>>> On 13.03.2014 13:37, Ivan Zhakov wrote:
>>>>
>>>> On 12 March 2014 18:17, Gareth McCaughan
>>>> <Ga...@lightblueoptics.com> wrote:
>>>>
>>>> Ivan Zhakov wrote:
>>>>
>>>> It looks like serious issue. Could you please file issue in Subversion
>>>> issue tracker: http://subversion.tigris.org/issues
>>>>
>>>> Done. Issue #4478.
>>>>
>>>> Gareth, thanks a lot!
>>>>
>>>> It seems we have second reason to create temporary files in target WC
>>>> directory, instead of .svn/tmp. Another problem we had before is
>>>> inherited ACL on Windows discussed year ago:
>>>> http://mail-archives.apache.org/mod_mbox/subversion-
>>>
>>> dev/201309.mbox/%3C20130928110059.d1bb8d007dfe7b26cbcb4f719cb77fd6
>>> .be88925bf6.wbe%40email16.secureserver.net%3E
>>>>
>>>>
>>>> I've prepared stupid patch to create temporary files in WC, instead of
>>>> .svn/tmp and it seems working fine.
>>>>
>>>>
>>>> -1. This leaves us with no reliable way to clean up the working copy in
>>>> case
>>>> of an aborted operation.
>>>>
>>>> There should be a better way for managing ACLs on windows that does not
>>>> require us to mess up the working copy. I'm perfectly happy with just
>>>> documenting that we don't support different ACLs for .svn and the rest
>>>> of
>>>> the WC, at least for now.
>>>>
>>> Please note that problem reported is not about inherited ACL: now
>>> users got NOT_INDEXED attribute on all WC files, because .svn marked
>>> to by not indexed by Windows Search and tools like that.
>>
>>
>> I'm working on a better patch for this, that doesn't have this problem and
>> will improve performance on the pristine store operations as well.
>>
>> But since when is this a huge problem? We applied this setting since 1.6
>> if I remember correctly... I'm more surprised that you didn't know about
>> this?
>
>
> Instead of setting the attributes after moving the files, how about creating
> a new folder "wctemp" in the .svn folder and set the attribute on that
> folder. Then use that folder to create the temp files which are intended for
> moving to the working copy? That way temp files don't have to be created
> inside the working copy.
>
Because user may change "not indexed" attribute of WC folder and
children should inherit it.
--
Ivan Zhakov
Re: Subversion checked-out files not indexed in Windows search
Posted by Stefan Kueng <to...@gmail.com>.
On 13.03.2014 14:14, Bert Huijben wrote:
>
>
>> -----Original Message-----
>> From: Ivan Zhakov [mailto:ivan@visualsvn.com]
>> Sent: donderdag 13 maart 2014 13:56
>> To: Branko Čibej
>> Cc: Subversion Development
>> Subject: Re: Subversion checked-out files not indexed in Windows search
>>
>> On 13 March 2014 16:46, Branko Čibej <br...@wandisco.com> wrote:
>>> On 13.03.2014 13:37, Ivan Zhakov wrote:
>>>
>>> On 12 March 2014 18:17, Gareth McCaughan
>>> <Ga...@lightblueoptics.com> wrote:
>>>
>>> Ivan Zhakov wrote:
>>>
>>> It looks like serious issue. Could you please file issue in Subversion
>>> issue tracker: http://subversion.tigris.org/issues
>>>
>>> Done. Issue #4478.
>>>
>>> Gareth, thanks a lot!
>>>
>>> It seems we have second reason to create temporary files in target WC
>>> directory, instead of .svn/tmp. Another problem we had before is
>>> inherited ACL on Windows discussed year ago:
>>> http://mail-archives.apache.org/mod_mbox/subversion-
>> dev/201309.mbox/%3C20130928110059.d1bb8d007dfe7b26cbcb4f719cb77fd6
>> .be88925bf6.wbe%40email16.secureserver.net%3E
>>>
>>> I've prepared stupid patch to create temporary files in WC, instead of
>>> .svn/tmp and it seems working fine.
>>>
>>>
>>> -1. This leaves us with no reliable way to clean up the working copy in case
>>> of an aborted operation.
>>>
>>> There should be a better way for managing ACLs on windows that does not
>>> require us to mess up the working copy. I'm perfectly happy with just
>>> documenting that we don't support different ACLs for .svn and the rest of
>>> the WC, at least for now.
>>>
>> Please note that problem reported is not about inherited ACL: now
>> users got NOT_INDEXED attribute on all WC files, because .svn marked
>> to by not indexed by Windows Search and tools like that.
>
> I'm working on a better patch for this, that doesn't have this problem and will improve performance on the pristine store operations as well.
>
> But since when is this a huge problem? We applied this setting since 1.6 if I remember correctly... I'm more surprised that you didn't know about this?
Instead of setting the attributes after moving the files, how about
creating a new folder "wctemp" in the .svn folder and set the attribute
on that folder. Then use that folder to create the temp files which are
intended for moving to the working copy? That way temp files don't have
to be created inside the working copy.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest interface to (Sub)version control
/_/ \_\ http://tortoisesvn.net
RE: Subversion checked-out files not indexed in Windows search
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Branko Čibej [mailto:brane@wandisco.com]
> Sent: dinsdag 18 maart 2014 11:37
> To: 'Subversion Development'
> Subject: Re: Subversion checked-out files not indexed in Windows search
>
> On 18.03.2014 11:20, Bert Huijben wrote:
> > But the reason we set this flag is mostly gone anyway... I think we should
> just remove our code to touch that attribute and use the apr attribute
> function like we did before. The additional indexing is not really our problem
> (and the indexer won't care about a few more MB), and it is very easy to
> filter on *.svn-base if somebody want that.
>
> Dunno ... I think any indexing in the .svn directory is a mistake and
> likely to cause confusion. Not because of performance degradation, but
> because users really shouldn't care about the contents of that
> directory, including not seeing them in search results.
>
> Of course, tweaking that flag when the file is renamed has its own set
> of "interesting" problems ... for example, we'd have to check the flag
> and inheritance settings on the parent directory.
We are not the one indexing :)
The attribute is just a hint supported by some of the possible indexers (many have positive and negative overrides). On some filesystems the flag isn't even supported.
Introducing a second tmp directory to create pristine files in (perhaps below .svn/pristine) would allow setting the flag just for the pristine files.
I'm not sure if that really requires a format bump... We could just clean it like .svn/tmp when there are no locks and no working queue items.
Bert
Re: Subversion checked-out files not indexed in Windows search
Posted by Branko Čibej <br...@wandisco.com>.
On 18.03.2014 11:20, Bert Huijben wrote:
> But the reason we set this flag is mostly gone anyway... I think we should just remove our code to touch that attribute and use the apr attribute function like we did before. The additional indexing is not really our problem (and the indexer won't care about a few more MB), and it is very easy to filter on *.svn-base if somebody want that.
Dunno ... I think any indexing in the .svn directory is a mistake and
likely to cause confusion. Not because of performance degradation, but
because users really shouldn't care about the contents of that
directory, including not seeing them in search results.
Of course, tweaking that flag when the file is renamed has its own set
of "interesting" problems ... for example, we'd have to check the flag
and inheritance settings on the parent directory.
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com
RE: Subversion checked-out files not indexed in Windows search
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Ivan Zhakov [mailto:ivan@visualsvn.com]
> Sent: dinsdag 18 maart 2014 11:07
> To: Bert Huijben
> Cc: Branko Čibej; Subversion Development
> Subject: Re: Subversion checked-out files not indexed in Windows search
>
> On 13 March 2014 17:14, Bert Huijben <be...@qqmail.nl> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Ivan Zhakov [mailto:ivan@visualsvn.com]
> >> Sent: donderdag 13 maart 2014 13:56
> >> To: Branko Čibej
> >> Cc: Subversion Development
> >> Subject: Re: Subversion checked-out files not indexed in Windows search
> >>
> >> On 13 March 2014 16:46, Branko Čibej <br...@wandisco.com> wrote:
> >> > On 13.03.2014 13:37, Ivan Zhakov wrote:
> >> >
> >> > On 12 March 2014 18:17, Gareth McCaughan
> >> > <Ga...@lightblueoptics.com> wrote:
> >> >
> >> > Ivan Zhakov wrote:
> >> >
> >> > It looks like serious issue. Could you please file issue in Subversion
> >> > issue tracker: http://subversion.tigris.org/issues
> >> >
> >> > Done. Issue #4478.
> >> >
> >> > Gareth, thanks a lot!
> >> >
> >> > It seems we have second reason to create temporary files in target WC
> >> > directory, instead of .svn/tmp. Another problem we had before is
> >> > inherited ACL on Windows discussed year ago:
> >> > http://mail-archives.apache.org/mod_mbox/subversion-
> >>
> dev/201309.mbox/%3C20130928110059.d1bb8d007dfe7b26cbcb4f719cb77fd6
> >> .be88925bf6.wbe%40email16.secureserver.net%3E
> >> >
> >> > I've prepared stupid patch to create temporary files in WC, instead of
> >> > .svn/tmp and it seems working fine.
> >> >
> >> >
> >> > -1. This leaves us with no reliable way to clean up the working copy in
> case
> >> > of an aborted operation.
> >> >
> >> > There should be a better way for managing ACLs on windows that does
> not
> >> > require us to mess up the working copy. I'm perfectly happy with just
> >> > documenting that we don't support different ACLs for .svn and the rest
> of
> >> > the WC, at least for now.
> >> >
> >> Please note that problem reported is not about inherited ACL: now
> >> users got NOT_INDEXED attribute on all WC files, because .svn marked
> >> to by not indexed by Windows Search and tools like that.
> >
> > I'm working on a better patch for this, that doesn't have this problem and
> > will improve performance on the pristine store operations as well.
> That's good news. Could you please share idea of your patch?
>
> I'm thinking of something like this:
> 1. Create temp file in target directory.
> 2. Set DELETE_ON_CLOSE=TRUE using SetFileInformationByHandle()
> 3. Write contents to temporary file
> 4. Set DELETE_ON_CLOSE=FALSE using SetFileInformationByHandle()
> 5. Rename to final location using SetFileInformationByHandle()
> 6. Delete if failed.
> 7. CloseFile()
The idea I was working on didn't involve creating files in locations where they could work against us in a case of a crash. I don't think delete on close will be handled on system failures, just on application failures.
The original reason to set the NOT_INDEXED flag on the .svn subdirectory (patches by stefanking and you), was that this caused quite a huge slowdown as the file indexer on Windows 7 and XP continuously monitors for changes and started indexing the file even before we moved it in place. (Even triggering an NTFS bug in Windows 7 pre SP 1)
The new install code keeps the file under a full lock that allows us to move the file in place without outside handling of a file. (A virusscanner can still read it while we have it open, but as we already have the move and delete privileges we don't have to wait to obtain move rights).
I think we should stick to the original (far pre Subversion 1.0) model of creating files in the .svn wc directories, for the same reason Brane alreadt vetoed your idea of creating the files in the directory. The reasoning behind that is very solid.
That some users want to see permissions different than in all previous versions is not a good reason to slow things down (another open, ACL check, ACL update, close) and/or to make the working copy less stable (possibility of files in the working copy that hides real changes and are easily to accidentally commit).
We never touched ACLs, on any platforms... I don't think we should start with doing that, especially at a performance cost.
I hoped that for the non-indexing bit we could just pass the attribute to CreateFileW(), but this function explicitly ignores this attribute. It only applies it from the parent directory. Using a different tempdir for creating new pristine files would work.
But the reason we set this flag is mostly gone anyway... I think we should just remove our code to touch that attribute and use the apr attribute function like we did before. The additional indexing is not really our problem (and the indexer won't care about a few more MB), and it is very easy to filter on *.svn-base if somebody want that.
Bert
>
> This will be Windows Vista only code since
> SetFileInformationByHandle() is not available for older platforms. But
> newer Windows SDKs has compatibility library FileExtd.lib for Windows
> XP.
>
>
> >
> > But since when is this a huge problem? We applied this setting since 1.6 if I
> remember correctly...
> > I'm more surprised that you didn't know about this?
> >
> I think setting NOT_INDEXED attribute is big problem, while ACL is
> nice to have but not so critical. I was not aware about this problem
> before.
>
>
> --
> Ivan Zhakov
> CTO | VisualSVN | http://www.visualsvn.com
Re: Subversion checked-out files not indexed in Windows search
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 13 March 2014 17:14, Bert Huijben <be...@qqmail.nl> wrote:
>
>
>> -----Original Message-----
>> From: Ivan Zhakov [mailto:ivan@visualsvn.com]
>> Sent: donderdag 13 maart 2014 13:56
>> To: Branko Čibej
>> Cc: Subversion Development
>> Subject: Re: Subversion checked-out files not indexed in Windows search
>>
>> On 13 March 2014 16:46, Branko Čibej <br...@wandisco.com> wrote:
>> > On 13.03.2014 13:37, Ivan Zhakov wrote:
>> >
>> > On 12 March 2014 18:17, Gareth McCaughan
>> > <Ga...@lightblueoptics.com> wrote:
>> >
>> > Ivan Zhakov wrote:
>> >
>> > It looks like serious issue. Could you please file issue in Subversion
>> > issue tracker: http://subversion.tigris.org/issues
>> >
>> > Done. Issue #4478.
>> >
>> > Gareth, thanks a lot!
>> >
>> > It seems we have second reason to create temporary files in target WC
>> > directory, instead of .svn/tmp. Another problem we had before is
>> > inherited ACL on Windows discussed year ago:
>> > http://mail-archives.apache.org/mod_mbox/subversion-
>> dev/201309.mbox/%3C20130928110059.d1bb8d007dfe7b26cbcb4f719cb77fd6
>> .be88925bf6.wbe%40email16.secureserver.net%3E
>> >
>> > I've prepared stupid patch to create temporary files in WC, instead of
>> > .svn/tmp and it seems working fine.
>> >
>> >
>> > -1. This leaves us with no reliable way to clean up the working copy in case
>> > of an aborted operation.
>> >
>> > There should be a better way for managing ACLs on windows that does not
>> > require us to mess up the working copy. I'm perfectly happy with just
>> > documenting that we don't support different ACLs for .svn and the rest of
>> > the WC, at least for now.
>> >
>> Please note that problem reported is not about inherited ACL: now
>> users got NOT_INDEXED attribute on all WC files, because .svn marked
>> to by not indexed by Windows Search and tools like that.
>
> I'm working on a better patch for this, that doesn't have this problem and
> will improve performance on the pristine store operations as well.
That's good news. Could you please share idea of your patch?
I'm thinking of something like this:
1. Create temp file in target directory.
2. Set DELETE_ON_CLOSE=TRUE using SetFileInformationByHandle()
3. Write contents to temporary file
4. Set DELETE_ON_CLOSE=FALSE using SetFileInformationByHandle()
5. Rename to final location using SetFileInformationByHandle()
6. Delete if failed.
7. CloseFile()
This will be Windows Vista only code since
SetFileInformationByHandle() is not available for older platforms. But
newer Windows SDKs has compatibility library FileExtd.lib for Windows
XP.
>
> But since when is this a huge problem? We applied this setting since 1.6 if I remember correctly...
> I'm more surprised that you didn't know about this?
>
I think setting NOT_INDEXED attribute is big problem, while ACL is
nice to have but not so critical. I was not aware about this problem
before.
--
Ivan Zhakov
CTO | VisualSVN | http://www.visualsvn.com
RE: Subversion checked-out files not indexed in Windows search
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Ivan Zhakov [mailto:ivan@visualsvn.com]
> Sent: donderdag 13 maart 2014 13:56
> To: Branko Čibej
> Cc: Subversion Development
> Subject: Re: Subversion checked-out files not indexed in Windows search
>
> On 13 March 2014 16:46, Branko Čibej <br...@wandisco.com> wrote:
> > On 13.03.2014 13:37, Ivan Zhakov wrote:
> >
> > On 12 March 2014 18:17, Gareth McCaughan
> > <Ga...@lightblueoptics.com> wrote:
> >
> > Ivan Zhakov wrote:
> >
> > It looks like serious issue. Could you please file issue in Subversion
> > issue tracker: http://subversion.tigris.org/issues
> >
> > Done. Issue #4478.
> >
> > Gareth, thanks a lot!
> >
> > It seems we have second reason to create temporary files in target WC
> > directory, instead of .svn/tmp. Another problem we had before is
> > inherited ACL on Windows discussed year ago:
> > http://mail-archives.apache.org/mod_mbox/subversion-
> dev/201309.mbox/%3C20130928110059.d1bb8d007dfe7b26cbcb4f719cb77fd6
> .be88925bf6.wbe%40email16.secureserver.net%3E
> >
> > I've prepared stupid patch to create temporary files in WC, instead of
> > .svn/tmp and it seems working fine.
> >
> >
> > -1. This leaves us with no reliable way to clean up the working copy in case
> > of an aborted operation.
> >
> > There should be a better way for managing ACLs on windows that does not
> > require us to mess up the working copy. I'm perfectly happy with just
> > documenting that we don't support different ACLs for .svn and the rest of
> > the WC, at least for now.
> >
> Please note that problem reported is not about inherited ACL: now
> users got NOT_INDEXED attribute on all WC files, because .svn marked
> to by not indexed by Windows Search and tools like that.
I'm working on a better patch for this, that doesn't have this problem and will improve performance on the pristine store operations as well.
But since when is this a huge problem? We applied this setting since 1.6 if I remember correctly... I'm more surprised that you didn't know about this?
Bert
AW: Subversion checked-out files not indexed in Windows search
Posted by Markus Schaber <m....@codesys.com>.
Hi, Bert,
Von: Bert Huijben [mailto:bert@qqmail.nl]
> From: Markus Schaber [mailto:m.schaber@codesys.com]
> > Hi,
> >
> > Von: Jason Kresowaty [mailto:jason@binarycoder.net]
> > > On 3/13/2014 9:08 AM, Branko Čibej wrote:
> > > > I understand that; it simply means that when we install a file
> > > > from .svn/tmp into the working copy proper, we have to change that
> > > > attribute; either before or after the file is moved into place.
> > >
> > > When this came up on ACLs, there seemed to be a lot of resistance to
> > changing
> > > the file after it was in place. Kind of switching the subject, but
> > > if you're okay with changing files after they are in place, consider
> > > also propagating the ACL at that time too.
> >
> > What about first setting the ACL and attributes while the file still
> > is in the temp folder, and then moving the file to the final place?
>
> To what would you want to set the ACL?
To the desired values which resemble the ones for a file created in
the final place.
Best regards
Markus Schaber
CODESYS® a trademark of 3S-Smart Software Solutions GmbH
Inspiring Automation Solutions
3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50
E-Mail: m.schaber@codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com
Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
Re: Subversion checked-out files not indexed in Windows search
Posted by Jason Kresowaty <ja...@binarycoder.net>.
On 3/17/2014 5:40 AM, Bert Huijben wrote:
>
> And updating these attributes one at a time during checkout would be huge (not a small) performance killer during checkout, as both operations would be updating the MFT.
>
> We should try to perform as much operations as possible while having the files open anyway, as each file open (which at the kernel level includes operations as setting properties) will involve virus scanners, etc.
>
>
> With the current trunk code the best solution might be to just remove the setting of the 'don't index' property, as the current code already moves the files in place while still open and 100% locked from indexers.
> (Those indexers really slowed us down on Windows XP and 7 around the time that we introduced this flag as the indexer opened the file between us creating and moving the file)
>
> Bert
>
>
>
As far as the ACLs, I think Windows Explorer uses this technique, so if
it causes problems with scanners, these problems happen just working
with files in Explorer too.
It is more complexity, but you could check if the 'attributes' are
incorrect and only make the API call if there is something to change.
Thus, only those users affected by the problem incur any decrease in
performance.
(This is assuming the API's in question are not already optimized for
this case; if I recall correctly, I don't think these kinds of APIs edit
the modified date in any case, for example.)
Re: Subversion checked-out files not indexed in Windows search
Posted by Thomas Åkesson <th...@akesson.cc>.
On 17 mar 2014, at 10:40, Bert Huijben <be...@qqmail.nl> wrote:
>
>
>> -----Original Message-----
>> From: Markus Schaber [mailto:m.schaber@codesys.com]
>> Sent: maandag 17 maart 2014 10:07
>> To: Jason Kresowaty; dev@subversion.apache.org
>> Subject: AW: Subversion checked-out files not indexed in Windows search
>>
>> Hi,
>>
>> Von: Jason Kresowaty [mailto:jason@binarycoder.net]
>>> On 3/13/2014 9:08 AM, Branko Čibej wrote:
>>>> I understand that; it simply means that when we install a file from
>>>> .svn/tmp into the working copy proper, we have to change that
>>>> attribute; either before or after the file is moved into place.
>>>
>>> When this came up on ACLs, there seemed to be a lot of resistance to
>> changing
>>> the file after it was in place. Kind of switching the subject, but if you're
>>> okay with changing files after they are in place, consider also propagating
>>> the ACL at that time too.
>>
>> What about first setting the ACL and attributes while the file still is in the
>> temp folder, and then moving the file to the final place?
>
> To what would you want to set the ACL?
>
> The code snippet posted makes the file inherit the ACL of its parent directory, which in this case would be the temp directory.
>
> Which is exactly the current behavior.
>
> And updating these attributes one at a time during checkout would be huge (not a small) performance killer during checkout, as both operations would be updating the MFT.
>
> We should try to perform as much operations as possible while having the files open anyway, as each file open (which at the kernel level includes operations as setting properties) will involve virus scanners, etc.
>
>
> With the current trunk code the best solution might be to just remove the setting of the 'don't index' property, as the current code already moves the files in place while still open and 100% locked from indexers.
> (Those indexers really slowed us down on Windows XP and 7 around the time that we introduced this flag as the indexer opened the file between us creating and moving the file)
Taking that line of thought one step further; The "don't index' property should probably be set on 'pristine' and 'wc.db' (for completeness also on 'entries' and 'format') rather than on '.svn'. Notably avoiding to set/inherit that property on tmp. I think setting the property is quite important, particularly for 'pristine'.
Should solve the indexing issue but not the ACL issue.
/Thomas Å.
RE: Subversion checked-out files not indexed in Windows search
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Markus Schaber [mailto:m.schaber@codesys.com]
> Sent: maandag 17 maart 2014 10:07
> To: Jason Kresowaty; dev@subversion.apache.org
> Subject: AW: Subversion checked-out files not indexed in Windows search
>
> Hi,
>
> Von: Jason Kresowaty [mailto:jason@binarycoder.net]
> > On 3/13/2014 9:08 AM, Branko Čibej wrote:
> > > I understand that; it simply means that when we install a file from
> > > .svn/tmp into the working copy proper, we have to change that
> > > attribute; either before or after the file is moved into place.
> >
> > When this came up on ACLs, there seemed to be a lot of resistance to
> changing
> > the file after it was in place. Kind of switching the subject, but if you're
> > okay with changing files after they are in place, consider also propagating
> > the ACL at that time too.
>
> What about first setting the ACL and attributes while the file still is in the
> temp folder, and then moving the file to the final place?
To what would you want to set the ACL?
The code snippet posted makes the file inherit the ACL of its parent directory, which in this case would be the temp directory.
Which is exactly the current behavior.
And updating these attributes one at a time during checkout would be huge (not a small) performance killer during checkout, as both operations would be updating the MFT.
We should try to perform as much operations as possible while having the files open anyway, as each file open (which at the kernel level includes operations as setting properties) will involve virus scanners, etc.
With the current trunk code the best solution might be to just remove the setting of the 'don't index' property, as the current code already moves the files in place while still open and 100% locked from indexers.
(Those indexers really slowed us down on Windows XP and 7 around the time that we introduced this flag as the indexer opened the file between us creating and moving the file)
Bert
AW: Subversion checked-out files not indexed in Windows search
Posted by Markus Schaber <m....@codesys.com>.
Hi,
Von: Jason Kresowaty [mailto:jason@binarycoder.net]
> On 3/13/2014 9:08 AM, Branko Čibej wrote:
> > I understand that; it simply means that when we install a file from
> > .svn/tmp into the working copy proper, we have to change that
> > attribute; either before or after the file is moved into place.
>
> When this came up on ACLs, there seemed to be a lot of resistance to changing
> the file after it was in place. Kind of switching the subject, but if you're
> okay with changing files after they are in place, consider also propagating
> the ACL at that time too.
What about first setting the ACL and attributes while the file still is in the
temp folder, and then moving the file to the final place?
[....]
Best regards
Markus Schaber
CODESYS® a trademark of 3S-Smart Software Solutions GmbH
Inspiring Automation Solutions
3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50
E-Mail: m.schaber@codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com
Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
Re: Subversion checked-out files not indexed in Windows search
Posted by Jason Kresowaty <ja...@binarycoder.net>.
On 3/13/2014 9:08 AM, Branko Čibej wrote:
> I understand that; it simply means that when we install a file from
> .svn/tmp into the working copy proper, we have to change that
> attribute; either before or after the file is moved into place.
When this came up on ACLs, there seemed to be a lot of resistance to
changing the file after it was in place. Kind of switching the subject,
but if you're okay with changing files after they are in place, consider
also propagating the ACL at that time too. As has been pointed out,
below is not the best possible solution as the file momentarily has the
undesired (but current-behavior) ACL and so there could be a problem on
abort. But I think it is an improvement over the current behavior, and
it is simple to implement.
This seems to give the desired effect:
DWORD ret;
ACL emptyAcl;
if (!InitializeAcl(&emptyAcl, sizeof(ACL), ACL_REVISION)) {
return 1;
}
// UNPROTECTED_SACL_SECURITY_INFORMATION ensures to enable inheritance.
// An empty DACL must be specified, not a NULL pointer.
ret = SetNamedSecurityInfo(fileName, SE_FILE_OBJECT,
DACL_SECURITY_INFORMATION | UNPROTECTED_SACL_SECURITY_INFORMATION,
NULL, NULL, &emptyAcl, NULL);
if (ret != ERROR_SUCCESS) {
return 1;
}
I think also a check that the file supports ACLs (e.g., because FAT/etc.
systems do not).
Thanks, I'll try not to harp on this subject again in the future. It's
just the fact that SVN does not do this (even as an option that can be
turned on) happens to be my #1 issue with svn.
Re: Subversion checked-out files not indexed in Windows search
Posted by Branko Čibej <br...@wandisco.com>.
On 13.03.2014 13:55, Ivan Zhakov wrote:
> On 13 March 2014 16:46, Branko Čibej <br...@wandisco.com> wrote:
>> On 13.03.2014 13:37, Ivan Zhakov wrote:
>>
>> On 12 March 2014 18:17, Gareth McCaughan
>> <Ga...@lightblueoptics.com> wrote:
>>
>> Ivan Zhakov wrote:
>>
>> It looks like serious issue. Could you please file issue in Subversion
>> issue tracker: http://subversion.tigris.org/issues
>>
>> Done. Issue #4478.
>>
>> Gareth, thanks a lot!
>>
>> It seems we have second reason to create temporary files in target WC
>> directory, instead of .svn/tmp. Another problem we had before is
>> inherited ACL on Windows discussed year ago:
>> http://mail-archives.apache.org/mod_mbox/subversion-dev/201309.mbox/%3C20130928110059.d1bb8d007dfe7b26cbcb4f719cb77fd6.be88925bf6.wbe%40email16.secureserver.net%3E
>>
>> I've prepared stupid patch to create temporary files in WC, instead of
>> .svn/tmp and it seems working fine.
>>
>>
>> -1. This leaves us with no reliable way to clean up the working copy in case
>> of an aborted operation.
>>
>> There should be a better way for managing ACLs on windows that does not
>> require us to mess up the working copy. I'm perfectly happy with just
>> documenting that we don't support different ACLs for .svn and the rest of
>> the WC, at least for now.
>>
> Please note that problem reported is not about inherited ACL: now
> users got NOT_INDEXED attribute on all WC files, because .svn marked
> to by not indexed by Windows Search and tools like that.
I understand that; it simply means that when we install a file from
.svn/tmp into the working copy proper, we have to change that attribute;
either before or after the file is moved into place.
See, e.g., SetFileInformationByHandle with FILE_BASIC_INFO, or
SetFileAttributes.
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com
Re: Subversion checked-out files not indexed in Windows search
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 13 March 2014 16:46, Branko Čibej <br...@wandisco.com> wrote:
> On 13.03.2014 13:37, Ivan Zhakov wrote:
>
> On 12 March 2014 18:17, Gareth McCaughan
> <Ga...@lightblueoptics.com> wrote:
>
> Ivan Zhakov wrote:
>
> It looks like serious issue. Could you please file issue in Subversion
> issue tracker: http://subversion.tigris.org/issues
>
> Done. Issue #4478.
>
> Gareth, thanks a lot!
>
> It seems we have second reason to create temporary files in target WC
> directory, instead of .svn/tmp. Another problem we had before is
> inherited ACL on Windows discussed year ago:
> http://mail-archives.apache.org/mod_mbox/subversion-dev/201309.mbox/%3C20130928110059.d1bb8d007dfe7b26cbcb4f719cb77fd6.be88925bf6.wbe%40email16.secureserver.net%3E
>
> I've prepared stupid patch to create temporary files in WC, instead of
> .svn/tmp and it seems working fine.
>
>
> -1. This leaves us with no reliable way to clean up the working copy in case
> of an aborted operation.
>
> There should be a better way for managing ACLs on windows that does not
> require us to mess up the working copy. I'm perfectly happy with just
> documenting that we don't support different ACLs for .svn and the rest of
> the WC, at least for now.
>
Please note that problem reported is not about inherited ACL: now
users got NOT_INDEXED attribute on all WC files, because .svn marked
to by not indexed by Windows Search and tools like that.
--
Ivan Zhakov
CTO | VisualSVN | http://www.visualsvn.com
Re: Subversion checked-out files not indexed in Windows search
Posted by Branko Čibej <br...@wandisco.com>.
On 13.03.2014 13:37, Ivan Zhakov wrote:
> On 12 March 2014 18:17, Gareth McCaughan
> <Ga...@lightblueoptics.com> wrote:
>> Ivan Zhakov wrote:
>>
>>> It looks like serious issue. Could you please file issue in Subversion
>>> issue tracker: http://subversion.tigris.org/issues
>> Done. Issue #4478.
>>
> Gareth, thanks a lot!
>
> It seems we have second reason to create temporary files in target WC
> directory, instead of .svn/tmp. Another problem we had before is
> inherited ACL on Windows discussed year ago:
> http://mail-archives.apache.org/mod_mbox/subversion-dev/201309.mbox/%3C20130928110059.d1bb8d007dfe7b26cbcb4f719cb77fd6.be88925bf6.wbe%40email16.secureserver.net%3E
>
> I've prepared stupid patch to create temporary files in WC, instead of
> .svn/tmp and it seems working fine.
-1. This leaves us with no reliable way to clean up the working copy in
case of an aborted operation.
There should be a better way for managing ACLs on windows that does not
require us to mess up the working copy. I'm perfectly happy with just
documenting that we don't support different ACLs for .svn and the rest
of the WC, at least for now.
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com
Re: Subversion checked-out files not indexed in Windows search
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On 12 March 2014 18:17, Gareth McCaughan
<Ga...@lightblueoptics.com> wrote:
> Ivan Zhakov wrote:
>
>> It looks like serious issue. Could you please file issue in Subversion
>> issue tracker: http://subversion.tigris.org/issues
>
> Done. Issue #4478.
>
Gareth, thanks a lot!
It seems we have second reason to create temporary files in target WC
directory, instead of .svn/tmp. Another problem we had before is
inherited ACL on Windows discussed year ago:
http://mail-archives.apache.org/mod_mbox/subversion-dev/201309.mbox/%3C20130928110059.d1bb8d007dfe7b26cbcb4f719cb77fd6.be88925bf6.wbe%40email16.secureserver.net%3E
I've prepared stupid patch to create temporary files in WC, instead of
.svn/tmp and it seems working fine.
--
Ivan Zhakov
RE: Subversion checked-out files not indexed in Windows search
Posted by Gareth McCaughan <Ga...@lightblueoptics.com>.
Ivan Zhakov wrote:
> It looks like serious issue. Could you please file issue in Subversion
> issue tracker: http://subversion.tigris.org/issues
Done. Issue #4478.
--
g
Re: Subversion checked-out files not indexed in Windows search
Posted by Branko Čibej <br...@wandisco.com>.
On 17.03.2014 17:29, Pete wrote:
> Ivan Zhakov <ivan <at> visualsvn.com> writes:
>
>> On 10 March 2014 18:25, Gareth McCaughan
>> <Gareth.McCaughan <at> lightblueoptics.com> wrote:
>>> It appears that files checked out using Subversion (1.7 onwards)
>>> are marked as not indexable by Windows search. The way in which
>>> this happens appears to be as follows:
> Wouldn't it be the easiest way to copy the temporary file (in such a case
> it gets the default attributes and ACLs of the destination folder) and
> after that erase the original temporary file?
Easiest, yes. Correct, no. Copying a file is not atomic; renaming is,
and we depend on the atomic behaviour of rename to keep working copies
consistent. That's ignoring the fact that a rename usually takes the
same amount of time regardless of the size of the file, whereas copy
does not.
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com
Re: Subversion checked-out files not indexed in Windows search
Posted by Pete <pe...@volja.net>.
Ivan Zhakov <ivan <at> visualsvn.com> writes:
>
> On 10 March 2014 18:25, Gareth McCaughan
> <Gareth.McCaughan <at> lightblueoptics.com> wrote:
> > It appears that files checked out using Subversion (1.7 onwards)
> > are marked as not indexable by Windows search. The way in which
> > this happens appears to be as follows:
Wouldn't it be the easiest way to copy the temporary file (in such a case
it gets the default attributes and ACLs of the destination folder) and
after that erase the original temporary file?
RE: Subversion checked-out files not indexed in Windows search
Posted by Gareth McCaughan <Ga...@lightblueoptics.com>.
Ivan Zhakov wrote:
> It looks like serious issue. Could you please file issue in Subversion
> issue tracker: http://subversion.tigris.org/issues
Done. Issue #4478.
--
g