You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Lübbe Onken <l....@rac.de> on 2003/12/12 09:22:18 UTC
Deleting trouble
Hi Folks,
a while ago I encountered the above mentioned problem, when working with
TortoiseSVN under Win2K. Before anybody shouts "This is the Subversion
developer list", yes, I know and that's why I am posting it here. Stefan
and I spent some time investigating the problem and came to the
conclusion that the problem is caused by the subversion libs leaking
file handles.
Try the following recipe under Win2K. You have to have TortoiseSVN
installed.
1) Do a fresh Checkout (using cl client or TortoiseSVN) into c:\foobar
2) modify a file inside c:\foobar, do *not* commit your changes
3) hit <f5> in Windows explorer to refresh the status overlay
4) try to delete c:\foobar using windows explorer
5) you will get an error like "c:\foobar\.svn\tmp is in use by another
process" (roughly translated from german)
We think, that this is caused by the subversion libraries leaking file
handles (especially .\svn\tmp). A quote from Stefan:
> The problem here is that explorer keeps the TSVN shell
> extension in memory, and the subversion command line
> client exits completely after every command. And windows
> automatically closes all still open handles a program created
> after it exits - so you won't notice that problem with the
> cl client...
I tried to lock down the problem using SysInternal's file monitor and
process explorer.
Attachement 1 contains a short log of what File monitor shows, when svn
st -v is run on the directory containing just a single modified file.
Notice, that there are a "TextDatei.pas.tmp" and a
"TextDatei.pas.tmp.tmp" created.
Then it looks like "TextDatei.pas.tmp.tmp" is renamed (to
"TextDatei.pas.tmp" ?!?) and then
"TextDatei.pas.tmp" is closed twice.
I think (cannot prove) that this sequence leaves a handle open.
The same sequence happens (more often) when the windows explorer, using
the TortoiseSVN shell integration, using the subversion libraries tries
to fetch the status. Only in this case the file handle is left open,
because the explorer process doesn't terminate.
A check with Process explorer shows, that explorer.exe has tons of
"*\.svn\tmp" file handles open at any time.
I hope that someone with deeper knowledge of the svn libs can
investigate further and maybe find a solution.
This only happens, when the working copy to delete contains uncommitted
changes, but it's a real blocker.
Cheers & thanks
-Lübbe
Re: Deleting trouble
Posted by Lübbe Onken <l....@rac.de>.
Edmund Horner wrote:
> I went through this on Win 2K using svn.exe at the command prompt.
Do you have TortoiseSVN installed?
There is no problem, when you *only* use the cl client. If you once
fetch the status with explorer and TortoiseSVN it will happen. 100%
reliable on two different Win2K machines. The WC has to contain modified
files.
> I also looked at the FILEMON log, and I can't say I noticed anything
> strange. TextDatei.pas.tmp is closed twice, because it is opened twice
> (CREATE counts as OPEN).
I know, but what about TextDatei.pas.tmp.tmp ?!? It disappears somehow.
Is this double .tmp.tmp intentional or accidental?
> Is it possible the problem is that, behind the scenes, TortoiseSVN is
> still in the foobar/.svn/tmp directory? That sounds like something that
> could be remedied in TortoiseSVN or libsvn_wc.
TortoiseSVN never does anything directly. It fetches all it's
information via the svn libraries, so I think it doesn't 'cd into
/.svn/tmp' anywhere, but I'm not the person to know. That must be Stefan.
Cheers
-Lübbe
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by Edmund Horner <ed...@chrysophylax.cjb.net>.
Lübbe Onken wrote:
> Hi Folks,
>
> a while ago I encountered the above mentioned problem, when working with
> TortoiseSVN under Win2K. Before anybody shouts "This is the Subversion
> developer list", yes, I know and that's why I am posting it here. Stefan
> and I spent some time investigating the problem and came to the
> conclusion that the problem is caused by the subversion libs leaking
> file handles.
>
> Try the following recipe under Win2K. You have to have TortoiseSVN
> installed.
>
> 1) Do a fresh Checkout (using cl client or TortoiseSVN) into c:\foobar
> 2) modify a file inside c:\foobar, do *not* commit your changes
> 3) hit <f5> in Windows explorer to refresh the status overlay
> 4) try to delete c:\foobar using windows explorer
> 5) you will get an error like "c:\foobar\.svn\tmp is in use by another
> process" (roughly translated from german)
I went through this on Win 2K using svn.exe at the command prompt. The
only thing I noticed is that you can't delete foobar if the command
prompt current directory is foobar. That's nothing new.
I also looked at the FILEMON log, and I can't say I noticed anything
strange. TextDatei.pas.tmp is closed twice, because it is opened twice
(CREATE counts as OPEN).
Is it possible the problem is that, behind the scenes, TortoiseSVN is
still in the foobar/.svn/tmp directory? That sounds like something that
could be remedied in TortoiseSVN or libsvn_wc.
Edmund.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by Lübbe Onken <l....@rac.de>.
Lübbe Onken wrote:
> A check with Process explorer shows, that explorer.exe has tons of
> "*\.svn\tmp" file handles open at any time.
PS: I reproduced the problem with a working copy that contains only one
file, which is checked out into c:\foobar\a.
Sysinternal's Process explorer shows the following thing happening:
- When fetching the status inside the WC (c:\foobar\a), explorer.exe has
two *\.svn\tmp file handles open after the first time fetching the
status. Each time I hit F5, the number of open handles increases by two.
- When fetching the status from the parent dir (c:\foobar), explorer.exe
has four *\.svn\tmp file handles open after the first time fetching the
status. Each time I hit F5, the number of open handles increases by four.
It looks like two file handles are leaked for each directory accessed
during status fetches.
Cheers
-Lübbe
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by Lübbe Onken <l....@rac.de>.
John Peacock wrote:
> Any time files are created and destroyed in rapid succession (like the
> temporary files in .svn/tmp), there is a significant risk that the
> following will happen:
The significance is 100% here :-| since all developers have the same
virus scanner installed. It's 100% reproducable with McAfee. Switch the
service off -> everything works. Switch the service on -> modified WCs
get wedged when trying to delete them.
>
> 1) Process A (svn) creates a temporary file
> 2) Process B (virus scanner) opens the file to scan it
> 3) Process A attempts to delete the temporary file
> 4) Process B blocks #3
I wonder, why this only happens, if the WC contains modifications. If
the WC is unmodified, it is never wedged.
> There is already code in svn to retry certain operations under Win32
> because of this. The CVSNT project has also come up against this
> behavior and made similar changes. Ultimately, it is poor design at the
> NTFS level, but all we can do is work around it.
I wonder, if the workaround in subversion only works around the problem,
when the WC is not modified. Maybe there is a small oversight, that in
some cases svn does not retry this operation.
> Depending on the load
> on the system, the only acceptable way to completely prevent the problem
> is to completely uninstall the virus scanner from the server (subversion
Client in my case. Our Servers run under Linux. ;-)
Cheers & thanks
-Lübbe
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by John Peacock <jp...@rowman.com>.
Lübbe Onken wrote:
> Now I wonder, if it's the scanner itself who is leaking the file handles
> or if it's Subversion, because of some interference by the scanner?!? I
> also wonder, why it's always the .svn/tmp directory and never anything
> else.
The way that virus scanners work in a Win32 environment is to hook into the
filesystem at a very low level. I would suspect the filehandle leakage is
actually in the virus scanner, because the way the hook works, the scanner has
to open the file before it can find out where the file was located and then
ignore it if it is not in one of the watched directories.
Any time files are created and destroyed in rapid succession (like the temporary
files in .svn/tmp), there is a significant risk that the following will happen:
1) Process A (svn) creates a temporary file
2) Process B (virus scanner) opens the file to scan it
3) Process A attempts to delete the temporary file
4) Process B blocks #3
There is already code in svn to retry certain operations under Win32 because of
this. The CVSNT project has also come up against this behavior and made similar
changes. Ultimately, it is poor design at the NTFS level, but all we can do is
work around it. Depending on the load on the system, the only acceptable way to
completely prevent the problem is to completely uninstall the virus scanner from
the server (subversion or CVSNT).
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by Lübbe Onken <l....@rac.de>.
Lübbe Onken wrote:
> I attached a log of the sequence:
I *knew* I'd forget this :-)
-Lübbe
Re: Deleting trouble
Posted by Lübbe Onken <l....@rac.de>.
Hi Folks,
some more input: Comparing event logs I found out, that when the virus
scanner is disabled, the log output of "svn st -v" and "f5" in the
explorer is exactly the same.
The WC containins a single modified file.
When the Virus scanner is enabled, the sequence shows, that the
directory .svn/tmp is opened twice, but never closed, hence the two
leaking file handles.
I attached a log of the sequence:
- the lines with numbers correspond to the standard sequence, when the
virus scanner is disabled
- the lines without numbers are extra. Note that there are additional
uppercase/lowercase accesses to the file. Who is causing these? SVN in
or McAfee?
- I marked the directory open actions with '-->'. There's no
corresponding close.
I hope this input helps someone to locate the problem.
Cheers
-Lübbe
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by Lübbe Onken <l....@rac.de>.
D.J. Heap wrote:
> Is TSVN using a long-lived context/pool or is this an even nastier leak
> that is not being freed even on pool cleanup?
I hope Stefan can jump in here to give the answer. I tell you what I
know. TortoiseSVN is an explorer shell extension and it stays in memory
as long as the explorer process lives. So if any handles are leaked they
will still be there. TSVN is using the Subversion libs, so I don't know,
if TSVN has a chance to tell whether handles are leaked.
> Presumably it is not
> reproducible with the commandline client?
No, it's not. I tried logging the "svn st -v" file open/close events
using sysinternals file Monitor and posted them here, because I thougt i
might have found something.
I will now log the corresponding sequence when I hit F5 in windows
explorer and try to pinpoint the action that fails. Maybe there's some
oversight in SVN, since it only fails, if the WC to be deleted is modified.
Cheers
-Lübbe
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by "D.J. Heap" <dj...@shadyvale.net>.
Lübbe Onken wrote:
[snip]
>
> It occurs 100% of the time, when the WC contains uncommitted changes.
> Maybe that's why nobody noticed it before. When the WC is not modified,
> the error doesn't occur.
>
>> The ugly resolution could be the same, of course, but I've never seen
>> this aspect of the problem so far.
>
>
> Try the following recipe under Win2K. You have to have TortoiseSVN
> installed.
> McAfee's background scan has to be enabled.
>
> 1) Do a fresh Checkout (using cl client or TortoiseSVN) into c:\foobar
> 2) modify a file inside c:\foobar, do *not* commit your changes
> 3) hit <f5> in Windows explorer to refresh the status overlay
> 4) try to delete c:\foobar using windows explorer or the dos box.
> 5) you will get an error like "c:\foobar\.svn\tmp is in use by another
> process" (roughly translated from german)
>
> This working copy stays wedged until you *completely* restart explorer.exe.
>
> If you switch the background scan off, everything works as expected.
>
> I reproduced the problem with a working copy that contains only one
> file, which is checked out into c:\foobar\a.
>
> Sysinternal's Process explorer shows the following thing happening:
>
> - When fetching the status inside the WC (c:\foobar\a), explorer.exe has
> two *\.svn\tmp file handles open after the first time fetching the
> status. Each time I hit F5, the number of open handles increases by two.
>
> - When fetching the status from the parent dir (c:\foobar), explorer.exe
> has four *\.svn\tmp file handles open after the first time fetching the
> status. Each time I hit F5, the number of open handles increases by four.
>
> It looks like two file handles are leaked for each directory accessed
> during status fetches.
>
> Cheers & thanks
> -Lübbe
Yes, if file handles are leaked then attempting to remove the directory
will fail -- no amount of retrying will help. The leak has to be fixed.
In the meantime, perhaps TSVN could be changed to clear and re-create
the client context for each operation if just clearing the main
pool/context gets rid of this problem.
I'm not sure the libraries were designed to expect long lived client
contexts and pools. I could be wrong, but I'm pretty sure there are
conditions (usually error conditions) where cleanup is just left to when
the pool gets cleared. Perhaps a core developer could shed some light
on the expected lifetime of the client's context/pool?
Is TSVN using a long-lived context/pool or is this an even nastier leak
that is not being freed even on pool cleanup? Presumably it is not
reproducible with the commandline client?
DJ
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by Lübbe Onken <l....@rac.de>.
D.J. Heap wrote:
> Can you post a reproduction recipe?
Yes, summing up my previous postings at the bottom:
> Do you have a realtime virus scanner enabled?
Yes, McAfee VirusScan profesisonal.
> Even with an intrusive AV system this never occurred
> 100% of the time for me, so it sounds like it might be a different
> problem.
It occurs 100% of the time, when the WC contains uncommitted changes.
Maybe that's why nobody noticed it before. When the WC is not modified,
the error doesn't occur.
> The ugly resolution could be the same, of course, but I've
> never seen this aspect of the problem so far.
Try the following recipe under Win2K. You have to have TortoiseSVN
installed.
McAfee's background scan has to be enabled.
1) Do a fresh Checkout (using cl client or TortoiseSVN) into c:\foobar
2) modify a file inside c:\foobar, do *not* commit your changes
3) hit <f5> in Windows explorer to refresh the status overlay
4) try to delete c:\foobar using windows explorer or the dos box.
5) you will get an error like "c:\foobar\.svn\tmp is in use by another
process" (roughly translated from german)
This working copy stays wedged until you *completely* restart explorer.exe.
If you switch the background scan off, everything works as expected.
I reproduced the problem with a working copy that contains only one
file, which is checked out into c:\foobar\a.
Sysinternal's Process explorer shows the following thing happening:
- When fetching the status inside the WC (c:\foobar\a), explorer.exe has
two *\.svn\tmp file handles open after the first time fetching the
status. Each time I hit F5, the number of open handles increases by two.
- When fetching the status from the parent dir (c:\foobar), explorer.exe
has four *\.svn\tmp file handles open after the first time fetching the
status. Each time I hit F5, the number of open handles increases by four.
It looks like two file handles are leaked for each directory accessed
during status fetches.
Cheers & thanks
-Lübbe
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by "D.J. Heap" <dj...@shadyvale.net>.
Lübbe Onken wrote:
[snip]
> Quoting my other posting:
> >> I wonder, if the workaround in subversion only works around the
> >> problem, when the WC is not modified.
> >> Maybe there is a small oversight, that in some cases svn
> >> does not retry this operation.
> The access denied error occurs in 100% of the cases, if the WC contains
> uncommitted modifications. If there's nothing to commit, there's no
> trouble deleting the WC.
Can you post a reproduction recipe? Do you have a realtime virus
scanner enabled? Even with an intrusive AV system this never occurred
100% of the time for me, so it sounds like it might be a different
problem. The ugly resolution could be the same, of course, but I've
never seen this aspect of the problem so far.
DJ
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by Lübbe Onken <l....@rac.de>.
Hi Folks,
I just spent a minute looking into subversion\libsvn_subr\io.c. Since
the error occurs on my machine, when the windows explorer is trying to
delete the directory .svn/tmp, I'd expect svn_io_remove_dir to contain
the same sleep loop as svn_io_remove_file. No deeper analysis, just an idea.
Near the end of the function we have:
---- SNIP ---
status = apr_dir_remove (path_apr, subpool);
if (status)
return svn_error_createf (status, NULL, err_msg_fmt, path);
---- SNAP
I'd insert the "#ifdef SVN_WIN32 sleep" block in between these two
lines, but I'm currently not able to build SVN & TSVN.
Maybe someone can check whether this would solve the problem?!?
Cheers
-Lübbe
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by Lübbe Onken <l....@rac.de>.
mark benedetto king wrote:
> http://subversion.tigris.org/project_faq.html#windows-access-denied
Ugly thing to do. It may reduce the frequency of occurence, make it even
more random and harder to locate, but lacking a better idea it's not up
to me to hurl a stone.
> Hmm. It looks like the patch has already been applied in the trunk.
Hmm. It looks like that patch doesn't solve the problem. I just switched
my sources to the 0.34 tag to check it and subversion/libsvn_subr/io.c
already contains that patch.
I'm running Subversion 0.34 and TortoiseSVN 0.23 which is built against
the 0.34 libraries, so according to the patch it "should not" happen.
Quoting my other posting:
>> I wonder, if the workaround in subversion only works around the
>> problem, when the WC is not modified.
>> Maybe there is a small oversight, that in some cases svn
>> does not retry this operation.
The access denied error occurs in 100% of the cases, if the WC contains
uncommitted modifications. If there's nothing to commit, there's no
trouble deleting the WC.
> Time to update the FAQ!
;-)
Cheers & thanks for your answer
-Lübbe
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by mark benedetto king <mb...@lowlatency.com>.
On Fri, Dec 12, 2003 at 01:50:12PM +0100, Lübbe Onken wrote:
> Hi Folks,
>
> I think I've found one possible culprit. It's McAfee's VirusScan. When
> the virusscanner is disabled, everything works as expected. When the
> background virusscanner is running, the explorer starts leaking file
> handles. It does not matter whether die directory in question is on the
> virusscanner's exclude list or not.
>
> Now I wonder, if it's the scanner itself who is leaking the file handles
> or if it's Subversion, because of some interference by the scanner?!? I
> also wonder, why it's always the .svn/tmp directory and never anything else.
>
http://subversion.tigris.org/project_faq.html#windows-access-denied
Hmm. It looks like the patch has already been applied in the trunk.
Time to update the FAQ!
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Deleting trouble
Posted by Lübbe Onken <l....@rac.de>.
Hi Folks,
I think I've found one possible culprit. It's McAfee's VirusScan. When
the virusscanner is disabled, everything works as expected. When the
background virusscanner is running, the explorer starts leaking file
handles. It does not matter whether die directory in question is on the
virusscanner's exclude list or not.
Now I wonder, if it's the scanner itself who is leaking the file handles
or if it's Subversion, because of some interference by the scanner?!? I
also wonder, why it's always the .svn/tmp directory and never anything else.
Cheers
-Lübbe
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org